Dataset Container

Interface

class deepparse.dataset_container.DatasetContainer(is_training_container: bool = True)[source]

Interface for the dataset. This interface defines most of the methods that the dataset needs to define. If you define another dataset container, the init must define the attribute data.

We also recommend using the validate_dataset method in your init to validate some characteristics of your dataset.

For a training container, it validates the following:

  • all addresses are not None value,

  • all addresses are not empty,

  • all addresses are not whitespace string,

  • all tags are not empty, if data is a list of tuple ([('an address', ['a_tag', 'another_tag']), ...]), and

  • if the addresses (whitespace-split) are the same length as their respective tags list.

While for a predict container (unknown prediction tag), it validates the following:

  • all addresses are not None,

  • all addresses are not empty, and

  • all addresses are not whitespace string.

Parameters:

is_training_container (bool) – Either or not, the dataset container is a training container. This will determine the dataset validation test we apply to the dataset. That is, a predict dataset doesn’t include tags. The default value is true.

Implementations

class deepparse.dataset_container.PickleDatasetContainer(data_path: str, is_training_container: bool = True)[source]

Pickle dataset container that imports a list of addresses in pickle format and does some validation on it.

The dataset needs to be a list of tuples where the first element of each tuple is the address (a string), and the second is a list of the expected tag to predict (e.g. [('an address', ['a_tag', 'another_tag']), ...]). The len of the tags needs to be the same as the len of the address when whitespace split.

For a training container, the validation tests applied on the dataset are the following:

  • all addresses are not None value,

  • all addresses are not empty,

  • all addresses are not whitespace string,

  • all tags are not empty, if data is a list of tuple ([('an address', ['a_tag', 'another_tag']), ...]), and

  • if the addresses (whitespace-split) are the same length as their respective tags list.

While for a predict container (unknown prediction tag), the validation tests applied on the dataset are the following:

  • all addresses are not None value,

  • all addresses are not empty, and

  • all addresses are not whitespace string.

Parameters:

  • data_path (str) – The path to the pickle dataset file.

  • is_training_container (bool) – Either or not, the dataset container is a training container. This will determine the dataset validation test we apply to the dataset. That is, a predict dataset doesn’t include tags. The default value is true.

class deepparse.dataset_container.CSVDatasetContainer(data_path: str, column_names: List | str, is_training_container: bool = True, separator: str = '\t', tag_seperator_reformat_fn: None | Callable = None, csv_reader_kwargs: None | Dict = None)[source]

CSV dataset container that imports a CSV of addresses. If the dataset is a predict one, it needs to have at least one column with some addresses. If the dataset is a training one (with prediction tags), it needs to have at least two columns, one with some addresses and another with a list of tags for each address.

After loading the CSV dataset, some tests will be applied depending on its type.

For a training container, the validation tests applied on the dataset are the following:

  • all addresses are not None value,

  • all addresses are not empty,

  • all addresses are not whitespace string, and

  • if the addresses (whitespace-split) are the same length as their respective tags list.

While for a predict container (unknown prediction tag), the validation tests applied on the dataset are the following:

  • all addresses are not None value,

  • all addresses are not empty, and

  • all addresses are not whitespace string.

Parameters:

  • data_path (str) – The path to the CSV dataset file.

  • column_names (list) – A column name list to extract the dataset element. If the dataset container is a predict one, the list must be of exactly one element (i.e. the address column). On the other hand, if the dataset container is a training one, the list must be of exactly two elements: addresses and tags.

  • is_training_container (bool) – Either or not, the dataset container is a training container. This will determine the dataset validation test we apply to the dataset. That is, a predict dataset doesn’t include tags. The default value is true.

  • separator (str) – The CSV columns separator to use. By default, "\t".

  • tag_seperator_reformat_fn (Callable, optional) – A function to parse a tags string and return a list of address tags. For example, if the tag column is a former python list saved with pandas, the characters ] , ] and ' will be included as the tags’ element. Thus, a parsing function will take a string as is parameter and output a python list. The default function process it as a former python list. That is, it removes the [], characters and splits the sequence at each comma (",").

  • csv_reader_kwargs (dict, optional) – Keyword arguments to pass to pandas read_csv use internally. By default, the data_path is passed along with our default sep value ( "\t") and the "utf-8" encoding format. However, this can be overridden by using this argument again.

class deepparse.dataset_container.ListDatasetContainer(data: List, is_training_container: bool = True)[source]

List dataset container that loads a list dataset into a DatasetContainer class. It also validates the dataset.

Parameters:

  • data (list) – The dataset in a list format. The list format (if a train or test container) is identical as the PickleDatasetContainer.

  • is_training_container (bool) – Either or not, the dataset container is a training container. This will determine the dataset validation test we apply to the dataset. That is, a predict dataset doesn’t include tags. The default value is true.

Dataset Validation Steps

We also applied data validations to all data containers using the following three functions.

deepparse.data_validation.data_validation.validate_if_any_empty(string_elements: List) → bool[source]

Return true if one of the string element is an empty one.

Parameters:

string_elements (list) – A list of string to validate.

deepparse.data_validation.data_validation.validate_if_any_whitespace_only(string_elements: List) → bool[source]

Return true if one of the string element is only whitespace.

Parameters:

string_elements (list) – A list of string to validate.

deepparse.data_validation.data_validation.validate_if_any_none(string_elements: List) → bool[source]

Return true if one of the string element is a None value.

Parameters:

string_elements (list) – A list of string to validate.