Skip to main content
Filter by
Sorted by
Tagged with
22 votes
7 answers
33k views

What's the difference between Flume and Sqoop?

Both Flume and Sqoop are meant for data movement, then what is the difference between them? Under what condition should I use Flume or Sqoop?
Cacheing's user avatar
  • 3,461
20 votes
1 answer
25k views

flume vs kafka vs others [closed]

May be this question has been asked before but I think it is good to consider it again today given that these technologies have matured. We're looking to use one of flume, kafka, scribe, or others to ...
pranavsharma's user avatar
  • 1,105
19 votes
5 answers
39k views

Rebalancing issue while reading messages in Kafka

I am trying to read messages on Kafka topic, but I am unable to read it. The process gets killed after sometime, without reading any messages. Here is the rebalancing error which I get: [2014-03-21 ...
divinedragon's user avatar
  • 5,326
18 votes
3 answers
4k views

What is the most mature library for building a Data Analytics Pipeline in Java/Scala for Hadoop?

I found many options recently, and interesting in their comparisons primarely by maturity and stability. Crunch - https://github.com/cloudera/crunch Scrunch - https://github.com/cloudera/crunch/tree/...
yura's user avatar
  • 14.6k
14 votes
7 answers
13k views

failing to load log4j2 while running fatjar

i am working on a project where i utilize log4j2 logging. while developing in intellij, all works fine and the logging is done as expected. the log4j2.xml is linked through java property passed to ...
atarno's user avatar
  • 355
14 votes
2 answers
2k views

how to efficiently move data from Kafka to an Impala table?

Here are the steps to the current process: Flafka writes logs to a 'landing zone' on HDFS. A job, scheduled by Oozie, copies complete files from the landing zone to a staging area. The staging data ...
Alex Woolford's user avatar
12 votes
7 answers
13k views

JMeter - Could not find the TestPlan class

I have a simple flume setup with a HTTP souce and a sink that writes the POST request payload to a file. (This complete setup is on a Linux machine). After that my task is to do a performance test on ...
Himanshu's user avatar
  • 1,443
12 votes
3 answers
16k views

How to setup a HTTP Source for testing Flume setup?

I am a newbie to Flume and Hadoop. We are developing a BI module where we can store all the logs from different servers in HDFS. For this I am using Flume. I just started trying it out. Succesfully ...
Himanshu's user avatar
  • 1,443
12 votes
1 answer
1k views

How to configure Flume to listen a web api http petitions

I have built an api web application, which is published on IIS Server, I am trying to configure Apache Flume to listen that web api and to save the response of http petitions in HDFS, this is the post ...
MelgoV's user avatar
  • 656
11 votes
2 answers
6k views

Apache Flume vs Apache Flink difference

I need to read a stream of data from some source (in my case it's UDP stream, but it shouldn't matter), transform the each record and write it to the HDFS. Is there any difference between using Flume ...
Kateryna  Khotkevych's user avatar
11 votes
0 answers
4k views

Scribe, Flume and Chukwa - what are alternatives? [closed]

I would like to learn about alternatives to those projects, especially designed to aggregate data from logs from multiple nodes (>500) and designed for low memory/cpu usage. I'm familiar with scribe, ...
wlk's user avatar
  • 5,765
9 votes
2 answers
4k views

Transferring files from remote node to HDFS with Flume

I have a bunch of binary files compressed into *gz format. These are generated on a remote node and must be transferred to HDFS located one of the datacenter's server. I'm exploring the option of ...
Mister Arduino's user avatar
9 votes
5 answers
22k views

How to install and configure apache flume? [closed]

Am new in the Apache Flume. I need to install the flume on top of the HDFS cluster environment. I did Google it, all are saying using the cloudera distribution but I need to install and configure from ...
venkat's user avatar
  • 513
9 votes
3 answers
11k views

real time log processing using apache spark streaming

I want to create a system where I can read logs in real time, and use apache spark to process it. I am confused if I should use something like kafka or flume to pass the logs to spark stream or should ...
Y0gesh Gupta's user avatar
  • 2,204
8 votes
2 answers
7k views

Save flume output to hive table with Hive Sink

I am trying to configure flume with Hive to save flume output to hive table with Hive Sink type. I have single node cluster. I use mapr hadoop distribution. Here is my flume.conf agent1.sources = ...
Andrey Braslavskiy's user avatar

15 30 50 per page
1
2 3 4 5
76