Questions tagged [scalding]
Scalding is a scala DSL for Cascading, running on Hadoop.
181
questions
0
votes
1
answer
24
views
Scalding Unit Test - How to Write A Local File?
I work at a place where scalding writes are augmented with a specific API to track dataset meta data. When converting from normal writes to these special writes, there are some intricacies with ...
0
votes
1
answer
21
views
having trouble installing scalding
I'm installing Scalding and sbt on my system but running command sbt assembly gives the following error:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting ...
0
votes
1
answer
889
views
How to concatenate list of strings with map reduce in scala
I have a list of list of strings and I want to concatenate all the unique strings into a single (delimited space) string, something that flatMap allows one to do. However I am confused about the ...
0
votes
1
answer
244
views
Is there a way to specify the number of mappers in Scalding?
I am new to scalding world. My scalding job will have multiple stages, and I need to tune each stage individually.
I have found that we might be able to change the number of reducers by using ...
4
votes
0
answers
163
views
How can I sort elements of a TypedPipe in Scalding?
I have not been able to find a way to sort elements of a TypedPipe in Scalding (when not performing a group operation). Here are the relevant parts of my program (replacing irrelevant parts with ...
1
vote
1
answer
90
views
How to mock a TextLine for Scalding using the type safe API?
I am trying to mock a TextLine for a Scalding job, but the offset appears to be getting mixed in with the line, whether I express the offset explicitly or implicitly.
Here is my job:
package changed
...
5
votes
3
answers
695
views
Recommended way to access HBase using Scala
Now that SpyGlass is no longer being maintained, what is the recommended way to access HBase using Scala/Scalding? A similar question was asked in 2013, but most of the suggested links are either dead ...
1
vote
0
answers
248
views
Scalding Execution Monad - What is it & how to use it
I am working on Big Data technologies using MR based on Java. But recently my company has moved to Scalding framework. I am not able get my head around the Scalding Execution Monad. What it is and how ...
2
votes
0
answers
346
views
oozie wofklow intermittently fails on java action for scalding
I am using CDH (Cloudera Hadoop) version 5.12.0 (which uses: Hadoop 2.6.0 Oozie 4.1.0) and Scalding 2.11
I am using a shaded jar with my dependencies built in.
I can run all my jobs properly without ...
1
vote
0
answers
110
views
scald.rb results in error (could not find or load main class)
I am trying to run the tutorial files from https://github.com/twitter/scalding/tree/develop/tutorial.
I cloned the 0.17.x branch and current develop branch and haven't had much success with either.
I ...
0
votes
0
answers
43
views
Use Kerberos for Authentication with Scalding/Cascading
I have some jobs that use Scalding to ingest Avro files into Accumulo. The current system uses username/password for authentication. The new system uses Kerberos for authentication.
Currently the ...
4
votes
2
answers
404
views
How to visualize steps of a scalding job
My scalding job is translated into 9 map reduce jobs (m/r jobs). It's not easy for me to understand which part of code each m/r job represents. Is there anything that could help me understand my job ...
0
votes
0
answers
281
views
Scalding operate on group after groupBy
I am writing a scalding job.
Here's what I want to do: first groupBy a Key. That should give me a bunch of (Key, Iterator[Value]) pairs for each Key (correct me if I'm wrong here). Then for each Ley,...
4
votes
0
answers
136
views
Scalding NPE only when assigning pipe to val
I'm new to Scala and Scalding, and in working on my first Job I'm encountering a NullPointerException when assigning a pipe to a val. The exact same job that just chains to a .write() without the ...
0
votes
2
answers
182
views
NoSuchMethodError when running scalding job
I am trying to upgrade a scalding job running on CDH 4.5 to CDH 5.5.1.
The job uses json4s to parse through json data. I am getting the below error when I run the job on the cluster. It works well on ...
0
votes
1
answer
269
views
What is MonoidAggregator in Algebird
I don't find any documention about MonoidAggregator.
What is it for ?
An example of its use:
MultiAggregator(
...,
Aggregator.forall[T](_.use)).andThenPresent(...)
...,
)
forAll return a ...
0
votes
0
answers
120
views
Java 7 to migration compatibility issue
After migrating the jobs from java 7 to 8 getting following error,even though compilation is successful.
Same version of java i.e java 8 was used in compilation and the same version is being used ...
0
votes
1
answer
201
views
Use AWS Java SDK inside EMR Job
I have a Scalding job packed in fatjar and running on EMR Hadoop cluster. Recently I added new feature requiring DynamoDB connection inside map. But as soon as mapper reaches DynamoDB initialization ...
0
votes
0
answers
662
views
Scalding read Timestamp from parquet
Hi I am trying to read a timestamp field (datetime) from A parquet file using a case class (in a typedpipe).
The parquet file is from an impala table.
the parquet-tools lib gives the type as
...
0
votes
1
answer
160
views
How to emit tuples instead of a list of tuples
I have a scalding job that looks like this:
import com.twitter.scalding.{Args, Csv, Job, TextLine}
class DataJob(args: Args) extends Job(args) {
val input = args("input")
val output = Csv(args("...
0
votes
0
answers
110
views
How to process CSV files with unknown token length
I am writing a Scalding job to transform data in the following format:
Id,Name,Param1,Val1,Param2,Val2....ParamxValx
1,Cat,Hair,White,Eye,Blue...
Into:
Id,Name,Param,Val
1,Cat,Hair,White
1,Cat,Eye,...
0
votes
1
answer
75
views
ClassNotFoundException with Scalding on Zeppelin managed on YARN
I'm trying to get Scalding working on Zeppelin while using YARN. I followed the steps in the docs here to build the interpreter and set up the classpath override. When I run in local mode, code ...
3
votes
1
answer
540
views
Read multiple files using scalding and output a SINGLE file
I experience an issue these days, i am trying to read from multiple files using scalding and create an output with a single file. My code is this:
def getFilesSource (paths: Seq[String]) = {
new ...
1
vote
1
answer
270
views
How do I use HyperLogLogMonoid from Algebird to carry out arbitrary intersections and unions
I'd like to aggregate a bunch of values that belong to a particular category into an HLL data structure so I can carry out intersections and unions later and count resulting cardinality of such ...
1
vote
2
answers
1k
views
How to override setup and cleanup methods in spark map function
Suppose there is following map reduce job
Mapper:
setup() initializes some state
map() add data to state, no output
cleanup() ouput state to context
Reducer:
aggregare all states into one ...
0
votes
1
answer
160
views
Scalding 'multiple map()' optimization
Are the following two code blocks equivalent in terms of performances?
val input: TypedPipe[Person] = ....
input
.map(_.getName)
.map(_.split(" "))
and...
val input: TypedPipe[Person] = ....
...
0
votes
1
answer
45
views
scala TypedPipe switch columns
I have a val in the format of TypedPipe[(Long, Long)], how do I switch left and right columns around? More clearly, how to create a new val with left column as the value in the right column, the right ...
2
votes
1
answer
208
views
Store algebird Bloom Filter with Storehaus
I have a Spark job whose final output is an Algebird bloom filter, and I'd need to reuse this bloom filter in another Spark job.
Is there a way to store this bloom filter in a kv store (eg: redis) ...
0
votes
1
answer
124
views
Scalding: Cannot reduce with partial function
I am using scalding to do a simple word count type of things. I get an error when using partial function to expand on the tuple. The exact error message is:
Error:(15, 14) missing parameter type for ...
1
vote
0
answers
73
views
Hortonworks Sandbox 2.1 | Split class cascading.tap.hadoop.io.MultiInputSplit not found
I’m executing a scalding job in Hortonworks distribution (HDP 2.1) and it throws this following issue:
I tried to locate the cascading jar in Hortonworks but I couldn’t find it. What am I doing wrong ...
0
votes
1
answer
239
views
What version of Scala does Hortonwork HDP 2.1 support?
I'm building Scalding job using Scala 2.10.4. Its successfully creating the job. But when I run the job in my Hortonworks it throwing the following exception.
According to some answers, this issue ...
0
votes
1
answer
251
views
Vertical Partitioning in Scalding
I have a TypedTipe[(String, String, Long)] where the first String can assume only a limited (~10) number of values. I'd like to partition my output so that a folder is created for each type (I.E. 10 ...
1
vote
2
answers
346
views
Scalding flatMap tuple containing list
I have the following input tuple that I'd like to flatMap: (String, List[String])
E.G. Input:
("a", ["1", "2"])
("b", ["3", "4"])
Needed output:
("a", "1")
("a", "2")
("b", "3")
("b", "4")
Is ...
1
vote
1
answer
484
views
Convert Seq to Pipe in Scalding
Context: I'm reading in a file where multiple fields are a list of IDs. I need to convert these fields into a Pipe to join them with other Pipes.
What I have tried:
val otherPipe = pipe
....
0
votes
1
answer
71
views
Is it possible to access the underlying org.apache.hadoop.mapreduce.Job from a Scalding job?
In my Scalding job, I have code like this:
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
class MyJob(args: Args) extends Job(args) {
FileInputFormat.setInputPathFilter(???, classOf[...
0
votes
1
answer
237
views
How can I make my Scalding job operate recursively on its input bucket?
I have a Scalding job which runs on EMR. It runs on an S3 bucket containing several files. The source looks like this:
MultipleTextLineFiles("s3://path/to/input/").read
/* ... some data processing ....
0
votes
0
answers
72
views
Can a Scalding source select a subset of the files in an S3 bucket to process?
I have a Scalding job which operates on all the files in a particular timestamped S3 bucket. It looks like this:
JsonLine("s3://path/to/timestampedbuckets/2016-02-03/", ('key1, 'key2)).read
I want ...
0
votes
2
answers
69
views
Write output in different directories
Are there any settings that I can use to make the output go in a separate timestamped (give a format) directory every time I run the job?
I use the following Scalding code to write my flow output:
...
0
votes
1
answer
389
views
Error importing scalding in sbt project
I got this error why importing the scalding sbt in my project build.sbt(ref:
How to declare dependency on Scalding in sbt project?). Kindly help me out.
lazy val scaldingCore = ProjectRef(uri("...
0
votes
1
answer
192
views
Find max from Scalding UnsortedGrouped[String, Set[MyStuff]]
I need to do the following:
group my records by some String property
discard groups that are too big
retrieve the max element in each of the remaining groups
Here is what I got so far:
val records:...
0
votes
1
answer
23
views
Scalding TypedType[Set[MyObject]] to TypedType[MyObject]
In my Scalding Flow I have a TypedType[Set[MyObject]]. How do I flatten it to a TypedType[MyObject]?
0
votes
1
answer
62
views
How do I change how keys are serialized in Scalding?
I am grouping by a custom type in my scalding job:
typedPipe
.map(someMapper)
.groupBy(_.nonPrimitiveField)
.sum
.write(sink)
In my output, the keys show up as the toString output, which ...
2
votes
0
answers
78
views
Is there a class to signify "grouped by and reduced"?
consider the following code in Scalding:
Let's say I have the following tuples in a scalding TypedPipe[(Int, Int)]:
(1, 2)
(1, 3)
(2, 1)
(2, 2)
On this pipe I can call groupBy(t => t._1) to ...
0
votes
2
answers
71
views
Generate diff of List[String] in Scalding
I have a records:TypedType[(String, util.List[String])] in my Scalding job where the first value is an id and the second a list of stuff. Imagine the following:
("1", ["a","b","c"])
("1", ["a","b","c"...
2
votes
0
answers
1k
views
What is causing "org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: null"?
I have an Elastic MapReduce job which uses elasticsearch-hadoop via scalding-taps to transfer data from Amazon S3 to Amazon Elasticsearch Service. For a long time this job ran successfully. However, ...
1
vote
1
answer
1k
views
Printing to Console in scalding script
I am trying to display some content on the console in a scalding script. When I run the same logic in the scalding shell I get the desired output and when I run the script I get an error:
scripttest....
0
votes
1
answer
503
views
How to count frequency of columns in row in a typedpipe in scalding?
I'm currently working on a mapreduce job using scalding. I'm trying to threshold based on how many times I see a particular value among the rows in my typedpipe. For example, if I had these rows in my ...
1
vote
1
answer
204
views
mutable.Buffer does not work with Scalding JobTest for Type Safe API
I have almost finished my Scalding project which uses the Type Safe API instead of the Fields API. The last issue that remains for me in overall project set up is the integration tests of the entire ...
0
votes
1
answer
292
views
loop through values in richPipe : scalding
I am trying to solve a issue where I have to loop through all values in pipe.To simulate my problem I am explain through a sample problem
Input file :
number
1
2
3
4
...
3
votes
2
answers
788
views
Scalding TypedPipe API External Operations pattern
I have a copy of Programming MapReduce with Scalding by Antonios Chalkiopoulos. In the book he discusses the External Operations design pattern for Scalding code. You can see an example on his ...