1,135
questions
22
votes
7
answers
33k
views
What's the difference between Flume and Sqoop?
Both Flume and Sqoop are meant for data movement, then what is the difference between them? Under what condition should I use Flume or Sqoop?
20
votes
1
answer
25k
views
flume vs kafka vs others [closed]
May be this question has been asked before but I think it is good to consider it again today given that these technologies have matured. We're looking to use one of flume, kafka, scribe, or others to ...
19
votes
5
answers
39k
views
Rebalancing issue while reading messages in Kafka
I am trying to read messages on Kafka topic, but I am unable to read it. The process gets killed after sometime, without reading any messages.
Here is the rebalancing error which I get:
[2014-03-21 ...
18
votes
3
answers
4k
views
What is the most mature library for building a Data Analytics Pipeline in Java/Scala for Hadoop?
I found many options recently, and interesting in their comparisons primarely by maturity and stability.
Crunch - https://github.com/cloudera/crunch
Scrunch - https://github.com/cloudera/crunch/tree/...
14
votes
7
answers
13k
views
failing to load log4j2 while running fatjar
i am working on a project where i utilize log4j2 logging.
while developing in intellij, all works fine and the logging is done as expected. the log4j2.xml is linked through java property passed to ...
14
votes
2
answers
2k
views
how to efficiently move data from Kafka to an Impala table?
Here are the steps to the current process:
Flafka writes logs to a 'landing zone' on HDFS.
A job, scheduled by Oozie, copies complete files from the landing zone to a staging area.
The staging data ...
12
votes
7
answers
13k
views
JMeter - Could not find the TestPlan class
I have a simple flume setup with a HTTP souce and a sink that writes the POST request payload to a file. (This complete setup is on a Linux machine).
After that my task is to do a performance test on ...
12
votes
3
answers
16k
views
How to setup a HTTP Source for testing Flume setup?
I am a newbie to Flume and Hadoop. We are developing a BI module where we can store all the logs from different servers in HDFS.
For this I am using Flume. I just started trying it out. Succesfully ...
12
votes
1
answer
1k
views
How to configure Flume to listen a web api http petitions
I have built an api web application, which is published on IIS Server, I am trying to configure Apache Flume to listen that web api and to save the response of http petitions in HDFS, this is the post ...
11
votes
2
answers
6k
views
Apache Flume vs Apache Flink difference
I need to read a stream of data from some source (in my case it's UDP stream, but it shouldn't matter), transform the each record and write it to the HDFS.
Is there any difference between using Flume ...
11
votes
0
answers
4k
views
Scribe, Flume and Chukwa - what are alternatives? [closed]
I would like to learn about alternatives to those projects, especially designed to aggregate data from logs from multiple nodes (>500) and designed for low memory/cpu usage.
I'm familiar with scribe, ...
9
votes
2
answers
4k
views
Transferring files from remote node to HDFS with Flume
I have a bunch of binary files compressed into *gz format. These are generated on a remote node and must be transferred to HDFS located one of the datacenter's server.
I'm exploring the option of ...
9
votes
5
answers
22k
views
How to install and configure apache flume? [closed]
Am new in the Apache Flume. I need to install the flume on top of the HDFS cluster environment. I did Google it, all are saying using the cloudera distribution but I need to install and configure from ...
9
votes
3
answers
11k
views
real time log processing using apache spark streaming
I want to create a system where I can read logs in real time, and use apache spark to process it. I am confused if I should use something like kafka or flume to pass the logs to spark stream or should ...
8
votes
2
answers
7k
views
Save flume output to hive table with Hive Sink
I am trying to configure flume with Hive to save flume output to hive table with Hive Sink type. I have single node cluster. I use mapr hadoop distribution.
Here is my flume.conf
agent1.sources = ...