9

I have a dataframe, which has two columns (review and sentiment). I am using pytorch and torchtext library for preprocessing data. Is it possible to use dataframe as source to read data from, in torchtext? I am looking for something similar to, but not

data.TabularDataset.splits(path='./data')

I have performed some operation (clean, change to required format) on data and final data is in a dataframe.

If not torchtext, what other package would you suggest that would help in preprocessing text data present in a datarame. I could not find anything online. Any help would be great.

2 Answers 2

15

Adapting the Dataset and Example classes from torchtext.data

    from torchtext.data import Field, Dataset, Example
    import pandas as pd

     class DataFrameDataset(Dataset):
         """Class for using pandas DataFrames as a datasource"""
         def __init__(self, examples, fields, filter_pred=None):
             """
             Create a dataset from a pandas dataframe of examples and Fields
             Arguments:
                 examples pd.DataFrame: DataFrame of examples
                 fields {str: Field}: The Fields to use in this tuple. The
                     string is a field name, and the Field is the associated field.
                 filter_pred (callable or None): use only exanples for which
                     filter_pred(example) is true, or use all examples if None.
                     Default is None
             """
             self.examples = examples.apply(SeriesExample.fromSeries, args=(fields,), axis=1).tolist()
             if filter_pred is not None:
                 self.examples = filter(filter_pred, self.examples)
             self.fields = dict(fields)
             # Unpack field tuples
             for n, f in list(self.fields.items()):
                 if isinstance(n, tuple):
                     self.fields.update(zip(n, f))
                     del self.fields[n]

     class SeriesExample(Example):
         """Class to convert a pandas Series to an Example"""
        
         @classmethod
         def fromSeries(cls, data, fields):
             return cls.fromdict(data.to_dict(), fields)

         @classmethod
         def fromdict(cls, data, fields):
             ex = cls()
             
             for key, field in fields.items():
                 if key not in data:
                     raise ValueError("Specified key {} was not found in "
                     "the input data".format(key))
                 if field is not None:
                     setattr(ex, key, field.preprocess(data[key]))
                 else:
                     setattr(ex, key, data[key])
             return ex

Then, first define fields using torchtext.data fields. For example:

    TEXT = data.Field(tokenize='spacy')
    LABEL = data.LabelField(dtype=torch.float)
    TEXT.build_vocab(train, max_size=25000, vectors="glove.6B.100d") 
    LABEL.build_vocab(train)
    fields = { 'sentiment' : LABEL, 'review' : TEXT }

before simply loading the dataframes:

    train_ds = DataFrameDataset(train_df, fields)
    valid_ds = DataFrameDataset(valid_df, fields)
9
  • I have tried implementing this, but it is not clear what "fields" should consist of or how it is constructed. In the questions case with two "Keys" in the dataframe: review and sentiment. Any further elaboration would highly appreciated
    – NicolaiF
    Commented Jan 15, 2019 at 12:34
  • 3
    Figured it out, it should be in the format of a dictionary where each key is series name and each value is what to do them: fields = { 'sentiment' : LABEL, 'review' : TEXT } where label and text are torchtext data fields such as: TEXT = data.Field(tokenize='spacy') LABEL = data.LabelField(dtype=torch.float) TEXT.build_vocab(train, max_size=25000, vectors="glove.6B.100d") LABEL.build_vocab(train)
    – NicolaiF
    Commented Jan 16, 2019 at 9:42
  • +1 because this implementation follows the original implementation logic and style pytorch.org/text/_modules/torchtext/data/… Commented Jun 21, 2020 at 19:02
  • 3
    @NicolaiF : What does variable 'train' refer to in the line: TEXT.build_vocab(train, max_size=25000, vectors="glove.6B.100d") LABEL.build_vocab(train) ? Commented Oct 10, 2020 at 1:48
  • 1
    @stackoverflowuser2010 I think it is: the Example is returned once all the fields are processed. Commented Oct 10, 2020 at 8:32
0

Thanks Geoffrey.

From looking at the source code for torchtext.data.field

https://pytorch.org/text/_modules/torchtext/data/field.html

It looks like the 'train' parameter needs to be either a Dataset already, or some iterable source of text data. But given we haven't created a dataset at this point I am guessing you have passed in just the column of text from the dataframe.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.