3

So what I am trying to do is to crawl data on S3 bucket with AWS Glue. Data stored as nested json and path looks like this:

s3://my-bucket/some_id/some_subfolder/datetime.json

When running default crawler (no custom classifiers) it does partition it based on path and deserializes json as expected, however, I would like to get a timestamp from the file name as well in a separate field. For now Crawler omits it.

For example if I run crawler on:

s3://my-bucket/10001/fromage/2017-10-10.json

I get table schema like this:

  • Partition 1: 10001
  • Partition 2: fromage
  • Array: JSON data

I did try to add custom classifier based on Grok pattern:

%{INT:id}/%{WORD:source}/%{TIMESTAMP_ISO8601:timestamp}

enter image description here

However, whenever I re-run crawler it skips custom classifier and uses default JSON one. As a solution obviously I could append file name to the JSON itself before running a crawler, but was wondering if I can avoid this step?

1 Answer 1

3

Classifiers only analyze the data within the file, not the filename itself. What you want to do is not possible today. If you can change the path where the files land, you could add the date as another partition:

s3://my-bucket/id=10001/source=fromage/timestamp=2017-10-10/data-file-2017-10-10.json
4
  • Classifiers not only analyze data as they add partitions based on the relative path. Thus was just hoping it would be possible to crawl it rather then doing this task via custom Map Reduce Dec 13, 2017 at 16:26
  • I can't find any reference suggesting that the crawlers apply classifiers to the relative path of the files found. The AWS docs indicates that Crawlers (not classifiers) partition the data based on the prefix of the files.
    – hoaxz
    Dec 13, 2017 at 17:09
  • On slide 23 from here slideshare.net/AmazonWebServices/… Dec 13, 2017 at 17:31
  • On that slide it shows how it accounts the partitions as a part of the algorithm to determine if data falls into the same schema. But it doesn't suggest that the classifiers are applied to the file path.
    – hoaxz
    Dec 13, 2017 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.