You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Magnus Mølgaard Lund 098daa829b Update 'segment.py' 1 year ago
IO_wrapper Noise reduction 1 year ago
classification Upload 1 year ago
data_acquisition Upload 1 year ago
datastructure Upload 1 year ago
schema Upload 1 year ago
text_extraction Upload 1 year ago
utils Upload 1 year ago
.gitignore Noise reduction 1 year ago
IO_handler.py Upload 1 year ago
LICENSE Noise reduction 1 year ago
PDFminerLayoutExtractor.py Upload 1 year ago
PDFminerLineStreamer.py Upload 1 year ago
README.md Update 'README.md' 1 year ago
SegmentedPDF.py Upload 1 year ago
coordinates_calculator.py Upload 1 year ago
ignore_coordinates.py Upload 1 year ago
miner.py Removed comments 1 year ago
requirements.txt Upload 1 year ago
segment.py Update 'segment.py' 1 year ago
text_analyzer.py Upload 1 year ago

README.md

Grundfos preprocessing

This project is part of the Knox multiproject and is located in layer one. The goal of this module is to extract and segment information from PDF-documents provided by Grundfos.

The module is able to extract text, titles, images and tables from PDF-files and produce a folder containing the extracted information. The module contains three components in addition to a few utilities:

  • The Text segmenter component recursively scans the document to find and segment text into their correct sections or subsections.
  • The Miner component analyzes the lines in the document to find tables and figures.
  • The Inference component uses computer vision to find tables and images (as well as text, lists and titles if desired).

Installation

  1. Clone the repository using git clone https://git.its.aau.dk/Knox/grundfos-preprocessing.git
  2. Create a new virtual environment using your preferred tool for example Conda. For Conda: conda create -n grundfos-preprocessing python=3.8 pip
  3. Activate the virtual environment. For Conda: conda activate grundfos-preprocessing
  4. Locate the root folder of the repository cd grundfos-preprocessing
  5. Install required dependencies with pip: pip install -r requirements.txt
  6. To use the machine-intelligence component download the model from https://drive.google.com/file/d/1Jx2m_2I1d9PYzFRQ4gl82xQa-G7Vsnsl/view?usp=sharing and place it in the classification folder.

Usage

To segment a document run the segment.py file in the root folder of the repository using the following command: python segment.py [FLAGS] INPUT_FOLDER OUTPUT_FOLDER

The INPUT_FOLDER must contain all of the PDF files to include in the segmention. Subfolders are omitted. The OUTPUT_FOLDER must exist as it is not created by the program. Optional arguments are written before specifying the INPUT_FOLDER and *OUTPUT_FOLDER, indicated by [FLAGS].

Flags

The flags available for segment.py are:

  • -h, --help Provides an overview of the flags and arguments available.
  • -a A, --accuraccy A Minimum threshold for the prediction accuracy used by machine intelligence module. Value between 0 and 1. Defult is 0.7.
  • -m, --machine Enable the machine intelligence module when running the program.
  • -t, -temporay Keep the temporary files created while running the program.
  • -c, --clean Clear the output folder before running the program.
  • -s, SCHEMA, --schema SCHEMA Path to the JSON schema. Default is schema/manuals_v1.2.schema.json.
  • -d, --download Download the Grundfos data set before running the program.

Acknowledgement

The wrapper module uses another module created by the Nordjyske preprocessing group. The module and its documentation can be found at https://git.its.aau.dk/Knox/source-data-io.