0

Can we configure flume to copy data in a near time manner from a table of any RDBMS e.g. Oracle / Teradata to HDFS.

I know sqoop is the standard method to use to copy data back-n-forth from RDBMS to HDFS, but that normally works in batch mode and I want the copy to happen in near real time mode.

If yes, then can I get a small working example of it.

1 Answer 1

0

There is no Sources for RDBMS databases. But you can take advantage of exec type in Apache Flume. See an example here.

4
  • Thanks, the example explains streaming of Oracle log file; but here I am interested in streaming of Oracle table's data. Will it be possible by using corresponding tablespace etc. Feb 25, 2016 at 16:23
  • 1
    Yes It is possible. You can mix it with this example as it uses MySQL Table Data to Flume : toadworld.com/platforms/oracle/w/wiki/… Feb 25, 2016 at 16:48
  • Thanks for this. Could you please also let me know what ever the data pulled can this be loaded in any hive table straight a way? Actually I haven't the environment available with me, so can't verify it. Feb 25, 2016 at 18:28
  • Yes of course. There is HiveSink that you can load your Oracle table data into it. Here is an example for HiveSink in Flume: henning.kropponline.de/2015/05/19/hivesink-for-flume Feb 25, 2016 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.