As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [scalding]

Scalding is a scala DSL for Cascading, running on Hadoop.

Filter by
Sorted by
Tagged with
0 votes
1 answer
24 views

Scalding Unit Test - How to Write A Local File?

I work at a place where scalding writes are augmented with a specific API to track dataset meta data. When converting from normal writes to these special writes, there are some intricacies with ...
codeaperature's user avatar
0 votes
1 answer
21 views

having trouble installing scalding

I'm installing Scalding and sbt on my system but running command sbt assembly gives the following error: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting ...
SAKSHI 's user avatar
0 votes
1 answer
889 views

How to concatenate list of strings with map reduce in scala

I have a list of list of strings and I want to concatenate all the unique strings into a single (delimited space) string, something that flatMap allows one to do. However I am confused about the ...
Soumajyoti Sarkar's user avatar
0 votes
1 answer
244 views

Is there a way to specify the number of mappers in Scalding?

I am new to scalding world. My scalding job will have multiple stages, and I need to tune each stage individually. I have found that we might be able to change the number of reducers by using ...
WarfDog's user avatar
  • 11
4 votes
0 answers
163 views

How can I sort elements of a TypedPipe in Scalding?

I have not been able to find a way to sort elements of a TypedPipe in Scalding (when not performing a group operation). Here are the relevant parts of my program (replacing irrelevant parts with ...
Ellen Spertus's user avatar
1 vote
1 answer
90 views

How to mock a TextLine for Scalding using the type safe API?

I am trying to mock a TextLine for a Scalding job, but the offset appears to be getting mixed in with the line, whether I express the offset explicitly or implicitly. Here is my job: package changed ...
Ellen Spertus's user avatar
5 votes
3 answers
695 views

Recommended way to access HBase using Scala

Now that SpyGlass is no longer being maintained, what is the recommended way to access HBase using Scala/Scalding? A similar question was asked in 2013, but most of the suggested links are either dead ...
Ellen Spertus's user avatar
1 vote
0 answers
248 views

Scalding Execution Monad - What is it & how to use it

I am working on Big Data technologies using MR based on Java. But recently my company has moved to Scalding framework. I am not able get my head around the Scalding Execution Monad. What it is and how ...
learner4life's user avatar
2 votes
0 answers
346 views

oozie wofklow intermittently fails on java action for scalding

I am using CDH (Cloudera Hadoop) version 5.12.0 (which uses: Hadoop 2.6.0 Oozie 4.1.0) and Scalding 2.11 I am using a shaded jar with my dependencies built in. I can run all my jobs properly without ...
Murium's user avatar
  • 183
1 vote
0 answers
110 views

scald.rb results in error (could not find or load main class)

I am trying to run the tutorial files from https://github.com/twitter/scalding/tree/develop/tutorial. I cloned the 0.17.x branch and current develop branch and haven't had much success with either. I ...
DBD's user avatar
  • 11
0 votes
0 answers
43 views

Use Kerberos for Authentication with Scalding/Cascading

I have some jobs that use Scalding to ingest Avro files into Accumulo. The current system uses username/password for authentication. The new system uses Kerberos for authentication. Currently the ...
Glen Warholic's user avatar
4 votes
2 answers
404 views

How to visualize steps of a scalding job

My scalding job is translated into 9 map reduce jobs (m/r jobs). It's not easy for me to understand which part of code each m/r job represents. Is there anything that could help me understand my job ...
Oleksii's user avatar
  • 1,101
0 votes
0 answers
281 views

Scalding operate on group after groupBy

I am writing a scalding job. Here's what I want to do: first groupBy a Key. That should give me a bunch of (Key, Iterator[Value]) pairs for each Key (correct me if I'm wrong here). Then for each Ley,...
DVB's user avatar
  • 1
4 votes
0 answers
136 views

Scalding NPE only when assigning pipe to val

I'm new to Scala and Scalding, and in working on my first Job I'm encountering a NullPointerException when assigning a pipe to a val. The exact same job that just chains to a .write() without the ...
jpk's user avatar
  • 281
0 votes
2 answers
182 views

NoSuchMethodError when running scalding job

I am trying to upgrade a scalding job running on CDH 4.5 to CDH 5.5.1. The job uses json4s to parse through json data. I am getting the below error when I run the job on the cluster. It works well on ...
Sruthi's user avatar
  • 187
0 votes
1 answer
269 views

What is MonoidAggregator in Algebird

I don't find any documention about MonoidAggregator. What is it for ? An example of its use: MultiAggregator( ..., Aggregator.forall[T](_.use)).andThenPresent(...) ..., ) forAll return a ...
azerty's user avatar
  • 688
0 votes
0 answers
120 views

Java 7 to migration compatibility issue

After migrating the jobs from java 7 to 8 getting following error,even though compilation is successful. Same version of java i.e java 8 was used in compilation and the same version is being used ...
shaheer01's user avatar
  • 111
0 votes
1 answer
201 views

Use AWS Java SDK inside EMR Job

I have a Scalding job packed in fatjar and running on EMR Hadoop cluster. Recently I added new feature requiring DynamoDB connection inside map. But as soon as mapper reaches DynamoDB initialization ...
chuwy's user avatar
  • 6,218
0 votes
0 answers
662 views

Scalding read Timestamp from parquet

Hi I am trying to read a timestamp field (datetime) from A parquet file using a case class (in a typedpipe). The parquet file is from an impala table. the parquet-tools lib gives the type as ...
dasman's user avatar
  • 227
0 votes
1 answer
160 views

How to emit tuples instead of a list of tuples

I have a scalding job that looks like this: import com.twitter.scalding.{Args, Csv, Job, TextLine} class DataJob(args: Args) extends Job(args) { val input = args("input") val output = Csv(args("...
JMM's user avatar
  • 3,892
0 votes
0 answers
110 views

How to process CSV files with unknown token length

I am writing a Scalding job to transform data in the following format: Id,Name,Param1,Val1,Param2,Val2....ParamxValx 1,Cat,Hair,White,Eye,Blue... Into: Id,Name,Param,Val 1,Cat,Hair,White 1,Cat,Eye,...
JMM's user avatar
  • 3,892
0 votes
1 answer
75 views

ClassNotFoundException with Scalding on Zeppelin managed on YARN

I'm trying to get Scalding working on Zeppelin while using YARN. I followed the steps in the docs here to build the interpreter and set up the classpath override. When I run in local mode, code ...
pwb2103's user avatar
  • 174
3 votes
1 answer
540 views

Read multiple files using scalding and output a SINGLE file

I experience an issue these days, i am trying to read from multiple files using scalding and create an output with a single file. My code is this: def getFilesSource (paths: Seq[String]) = { new ...
George Lica's user avatar
  • 1,788
1 vote
1 answer
270 views

How do I use HyperLogLogMonoid from Algebird to carry out arbitrary intersections and unions

I'd like to aggregate a bunch of values that belong to a particular category into an HLL data structure so I can carry out intersections and unions later and count resulting cardinality of such ...
harshsinghal's user avatar
  • 3,710
1 vote
2 answers
1k views

How to override setup and cleanup methods in spark map function

Suppose there is following map reduce job Mapper: setup() initializes some state map() add data to state, no output cleanup() ouput state to context Reducer: aggregare all states into one ...
Julias's user avatar
  • 5,722
0 votes
1 answer
160 views

Scalding 'multiple map()' optimization

Are the following two code blocks equivalent in terms of performances? val input: TypedPipe[Person] = .... input .map(_.getName) .map(_.split(" ")) and... val input: TypedPipe[Person] = .... ...
Marsellus Wallace's user avatar
0 votes
1 answer
45 views

scala TypedPipe switch columns

I have a val in the format of TypedPipe[(Long, Long)], how do I switch left and right columns around? More clearly, how to create a new val with left column as the value in the right column, the right ...
Gina Sun's user avatar
2 votes
1 answer
208 views

Store algebird Bloom Filter with Storehaus

I have a Spark job whose final output is an Algebird bloom filter, and I'd need to reuse this bloom filter in another Spark job. Is there a way to store this bloom filter in a kv store (eg: redis) ...
arnaud briche's user avatar
0 votes
1 answer
124 views

Scalding: Cannot reduce with partial function

I am using scalding to do a simple word count type of things. I get an error when using partial function to expand on the tuple. The exact error message is: Error:(15, 14) missing parameter type for ...
user_1357's user avatar
  • 7,736
1 vote
0 answers
73 views

Hortonworks Sandbox 2.1 | Split class cascading.tap.hadoop.io.MultiInputSplit not found

I’m executing a scalding job in Hortonworks distribution (HDP 2.1) and it throws this following issue: I tried to locate the cascading jar in Hortonworks but I couldn’t find it. What am I doing wrong ...
Renien's user avatar
  • 551
0 votes
1 answer
239 views

What version of Scala does Hortonwork HDP 2.1 support?

I'm building Scalding job using Scala 2.10.4. Its successfully creating the job. But when I run the job in my Hortonworks it throwing the following exception. According to some answers, this issue ...
Renien's user avatar
  • 551
0 votes
1 answer
251 views

Vertical Partitioning in Scalding

I have a TypedTipe[(String, String, Long)] where the first String can assume only a limited (~10) number of values. I'd like to partition my output so that a folder is created for each type (I.E. 10 ...
Marsellus Wallace's user avatar
1 vote
2 answers
346 views

Scalding flatMap tuple containing list

I have the following input tuple that I'd like to flatMap: (String, List[String]) E.G. Input: ("a", ["1", "2"]) ("b", ["3", "4"]) Needed output: ("a", "1") ("a", "2") ("b", "3") ("b", "4") Is ...
Marsellus Wallace's user avatar
1 vote
1 answer
484 views

Convert Seq to Pipe in Scalding

Context: I'm reading in a file where multiple fields are a list of IDs. I need to convert these fields into a Pipe to join them with other Pipes. What I have tried: val otherPipe = pipe ....
gstvolvr's user avatar
  • 650
0 votes
1 answer
71 views

Is it possible to access the underlying org.apache.hadoop.mapreduce.Job from a Scalding job?

In my Scalding job, I have code like this: import org.apache.hadoop.mapreduce.lib.input.FileInputFormat class MyJob(args: Args) extends Job(args) { FileInputFormat.setInputPathFilter(???, classOf[...
fblundun's user avatar
  • 967
0 votes
1 answer
237 views

How can I make my Scalding job operate recursively on its input bucket?

I have a Scalding job which runs on EMR. It runs on an S3 bucket containing several files. The source looks like this: MultipleTextLineFiles("s3://path/to/input/").read /* ... some data processing ....
fblundun's user avatar
  • 967
0 votes
0 answers
72 views

Can a Scalding source select a subset of the files in an S3 bucket to process?

I have a Scalding job which operates on all the files in a particular timestamped S3 bucket. It looks like this: JsonLine("s3://path/to/timestampedbuckets/2016-02-03/", ('key1, 'key2)).read I want ...
fblundun's user avatar
  • 967
0 votes
2 answers
69 views

Write output in different directories

Are there any settings that I can use to make the output go in a separate timestamped (give a format) directory every time I run the job? I use the following Scalding code to write my flow output: ...
Marsellus Wallace's user avatar
0 votes
1 answer
389 views

Error importing scalding in sbt project

I got this error why importing the scalding sbt in my project build.sbt(ref: How to declare dependency on Scalding in sbt project?). Kindly help me out. lazy val scaldingCore = ProjectRef(uri("...
Taiwotman's user avatar
  • 875
0 votes
1 answer
192 views

Find max from Scalding UnsortedGrouped[String, Set[MyStuff]]

I need to do the following: group my records by some String property discard groups that are too big retrieve the max element in each of the remaining groups Here is what I got so far: val records:...
Marsellus Wallace's user avatar
0 votes
1 answer
23 views

Scalding TypedType[Set[MyObject]] to TypedType[MyObject]

In my Scalding Flow I have a TypedType[Set[MyObject]]. How do I flatten it to a TypedType[MyObject]?
Marsellus Wallace's user avatar
0 votes
1 answer
62 views

How do I change how keys are serialized in Scalding?

I am grouping by a custom type in my scalding job: typedPipe .map(someMapper) .groupBy(_.nonPrimitiveField) .sum .write(sink) In my output, the keys show up as the toString output, which ...
mushroom's user avatar
  • 6,141
2 votes
0 answers
78 views

Is there a class to signify "grouped by and reduced"?

consider the following code in Scalding: Let's say I have the following tuples in a scalding TypedPipe[(Int, Int)]: (1, 2) (1, 3) (2, 1) (2, 2) On this pipe I can call groupBy(t => t._1) to ...
lezebulon's user avatar
  • 7,487
0 votes
2 answers
71 views

Generate diff of List[String] in Scalding

I have a records:TypedType[(String, util.List[String])] in my Scalding job where the first value is an id and the second a list of stuff. Imagine the following: ("1", ["a","b","c"]) ("1", ["a","b","c"...
Marsellus Wallace's user avatar
2 votes
0 answers
1k views

What is causing "org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: null"?

I have an Elastic MapReduce job which uses elasticsearch-hadoop via scalding-taps to transfer data from Amazon S3 to Amazon Elasticsearch Service. For a long time this job ran successfully. However, ...
fblundun's user avatar
  • 967
1 vote
1 answer
1k views

Printing to Console in scalding script

I am trying to display some content on the console in a scalding script. When I run the same logic in the scalding shell I get the desired output and when I run the script I get an error: scripttest....
Rahul Vatsa's user avatar
0 votes
1 answer
503 views

How to count frequency of columns in row in a typedpipe in scalding?

I'm currently working on a mapreduce job using scalding. I'm trying to threshold based on how many times I see a particular value among the rows in my typedpipe. For example, if I had these rows in my ...
Lucas's user avatar
  • 1
1 vote
1 answer
204 views

mutable.Buffer does not work with Scalding JobTest for Type Safe API

I have almost finished my Scalding project which uses the Type Safe API instead of the Fields API. The last issue that remains for me in overall project set up is the integration tests of the entire ...
PhillipAMann's user avatar
0 votes
1 answer
292 views

loop through values in richPipe : scalding

I am trying to solve a issue where I have to loop through all values in pipe.To simulate my problem I am explain through a sample problem Input file : number 1 2 3 4 ...
Ankita's user avatar
  • 1
3 votes
2 answers
788 views

Scalding TypedPipe API External Operations pattern

I have a copy of Programming MapReduce with Scalding by Antonios Chalkiopoulos. In the book he discusses the External Operations design pattern for Scalding code. You can see an example on his ...
PhillipAMann's user avatar