3

I want to generate the TPC-DS data (1 TB and 10 TB) directly in AWS S3 without transferring from local machine to s3. What is the easiest way to do that?

1 Answer 1

1

I did similar work several month ago, hive-testbench can be an option. Check the README.md about how to make it happen.

You need to configure fs.defaultFS in $HADOOP_HOME/etc/hadoop/core-site.xml to your AWS S3 bucket, the data will be generated in AWS directly.
Pass data scale parameter to ./tpcds-setup.sh to generate date with different scale.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.