Questions tagged [scala]

Scala is a general-purpose programming language principally targeting the Java Virtual Machine. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles. Its key features are: an advanced static type system with type inference; function types; pattern-matching; implicit parameters and conversions; operator overloading; full interoperability with Java; concurrency

scala
Filter by
Sorted by
Tagged with
1 vote
0 answers
428 views

spark withColumn fails on creating new dataframe when done in a loop

i have a dataframe where i have split it into a "matchedDF" and an "unmatchedDF" based on a columnid that meets a criteria. that works; then i have a collection of columns in an array that I do a ...
user2704580's user avatar
5 votes
4 answers
1k views

scala how to convert future of one type to future of another type

i have a function which retrieve data from DB and return type of Future[ResponseDTO], then I need to convert it to Future of HttpResponse my code: val responseDTO = database.getResponseDto(service) =...
user468587's user avatar
  • 4,969
0 votes
1 answer
62 views

Macro annotation generated companion class's apply method not resolved

I've written a macro annotation processor that generates all the same methods you would get from declaring a case class, but providing hash-consing. It was a bit tricky, but overall I'm very pleased ...
user1063042's user avatar
3 votes
1 answer
99 views

java.lang.RuntimeException: You must run the `stage` task before deploying your app when running `sbt stage deployHeroku`

I am trying to deploy my application to Heroku using sbt-nativepackager and sbt-heroku. My code is available at https://github.com/hhimanshu/sbt101/tree/m5 (branch is m5) When I run sbt stage ...
daydreamer's user avatar
  • 89.4k
4 votes
3 answers
2k views

How can I get random data generated for scala case classes with the ability to "change some values" for unit testing?

I'm working with a piece of code that has a broad/deep case class hierarchy. For unit testing, I'd like to have "random data" populated in the classes with the ability to change the data for the ...
PhD's user avatar
  • 11.2k
3 votes
1 answer
389 views

How to get the COUNT of emails for each id in Scala

I use this query in SQL to get return how many user_id's have more than one email. How would I write this same query against a users DataFrame in Scala? also how would I be able to return to exact ...
KurryF's user avatar
  • 73
2 votes
1 answer
50 views

passing a sequence into varargs method that does not use type ascription

I am working with a third party library and am trying to do some abstractions and am running into an issue with a method that accepts varargs but is not using type ascription. Something like: def ...
adambsg's user avatar
  • 131
0 votes
1 answer
132 views

Cannot deploy sbt project to Heroku

I have an sbt based project https://github.com/hhimanshu/sbt101/tree/m5 (branch is m5). I have heroku toolbelt installed locally on my machine and have run the following commands: 594 heroku login ...
daydreamer's user avatar
  • 89.4k
0 votes
0 answers
428 views

How to parse List of Values from a column of a file into Spark sql Dataframe

I am still new to spark scala, i have a requirement to extract the first partition from every table in hive. I have extracted the table list in a separate text file and created as a sequence, i have ...
Karthik Ramachandran's user avatar
0 votes
1 answer
312 views

How to send access_token which is generated in saveAs("access_token") to be sent to ElFileBody of Gatlin Soap Request

I have this code for gatling where i wish to pass auth_token to soap request body. Soap request body is written in a txt file and that txt file is called in ElFilebody All i need is to send ...
Tk infostrings's user avatar
1 vote
1 answer
551 views

How to properly decompress gz archive in Scala

I a newbie in Scala and I have small task that requires me to decompress *.gz file from resources directory. So I want a proper way to do that to be able to parse file content after. Surely I have ...
FLASh's user avatar
  • 97
2 votes
1 answer
224 views

ScalikeJDBC wrong order of columns in the result string

I am using ScalikeJDBC to get the results of queries. However, the problem is that the order of columns in the output does not conform to the one I define in queries. For me, the order of columns is ...
Cassie's user avatar
  • 3,031
2 votes
0 answers
311 views

How to run a scala application on Spark standalone cluster?

I'm trying to set up a standalone spark cluster on my Windows machine and run a scala application in the form a jar that would simply read a 100MB CSV file and write it into another location. I am ...
Tusharjain93's user avatar
0 votes
3 answers
918 views

Converting a dataframe to an array of struct of column names and values

Suppose I have a dataframe like this val customer = Seq( ("C1", "Jackie Chan", 50, "Dayton", "M"), ("C2", "Harry Smith", 30, "Beavercreek", "M"), ("C3", "Ellen Smith", 28, "Beavercreek", "...
Srinivas's user avatar
  • 2,070
2 votes
2 answers
696 views

Akka HTTP client side websocket closes unexpectedly

I have a websocket endpoint which sends a text message to the client every second. The client never sends any message to the server. Using the below JS code, it works as expected, it keeps logging ...
kurgan's user avatar
  • 83
5 votes
0 answers
527 views

Scala-cats: Memoization support

Scalaz provides Memo for memoization. I hope scala-cats also has a similar solution. In general I have a trait with different methods: trait t { def f1:Int = ??? def f2(i:Int):Int = ??? def f3(...
Alexandr's user avatar
  • 9,371
4 votes
1 answer
185 views

Check how many object properties are defined

I'm quite new in Scala world. Is there a better way to check how many properties are defined in an object, rather than going through all of them with idDefined() and incrementing a value? case class ...
UnguruBulan's user avatar
3 votes
1 answer
643 views

Initialisation order throws null pointer on lazy val access

Expectedly, the following initialisation order without lazy val throws null pointer exception class Foo { Bar.x // NullPointerException } object Bar extends Foo { val x = 42 } object Hello ...
Mario Galic's user avatar
1 vote
0 answers
325 views

Netflix Archaius Scala API and yaml file

I'm using Netflix/Archaius library for configuration management of scala/spark job. currently my property files are in the java properties format like com.comp.appname.solrconfig.collectionname=...
venBigData's user avatar
0 votes
0 answers
308 views

Spark Hbase connector not working in parallel mode?

I am trying to use Hortonworks hbase connector for spark 2.0 to work with hbase (https://github.com/hortonworks-spark/shc/tree/v1.1.0-2.0) With the provided example in the above link, val spark = ...
William R's user avatar
  • 739
-2 votes
1 answer
2k views

Read, transform and write data within each partition in the DataFrame

Language - Scala Spark version - 2.4 I am new to both Scala and Spark. (I am from python background, so the whole JVM ecosystem is quite new to me) I want to write a spark program to parallelize ...
Vijayant's user avatar
  • 682
1 vote
1 answer
3k views

Guice Scala DI, understanding how

Will default constructor be called in scala if I have @Provides annotation in my Module file to return an object but I never inject it anywhere?
Kumar Kavish's user avatar
0 votes
1 answer
634 views

How to serve static HTML file in Scala?

I have an EchoServer that I was provided to run and serve static local HTML files on my computer. It would then put out an error if the files were non-existent. I want to know how do I provide a local ...
KKNg's user avatar
  • 89
2 votes
1 answer
561 views

Sqoop command with --query in Scala using Process("<sqoop command>").! not working

I am trying to run a Sqoop command having --query option on Scala shell using Process("<sqoop command>").! but it is showing an error described below. If I use --table instead of --query, it ...
Sourav Das's user avatar
1 vote
0 answers
137 views

How do we set scalac options for test environment, not just test file tree?

We want to cause scalac warnings to fail our build in the test/CI environment, but not in a dev environment. We tried this in our build.sbt: scalacOptions in Test += "-Xfatal-warnings" However, ...
elijah's user avatar
  • 2,914
2 votes
2 answers
1k views

How to iterate over DataStream

I am a newbie to scala. I have a custom class Analytics.scala which has few variables (var a, var b, var c). I get a DataStream of type Analytics in my test case and I want to set value of var c as '0'...
Akhil Agarwal's user avatar
1 vote
1 answer
121 views

How to cast a Map based on the type of its keys?

I have a function, getMap, that returns Map[Any, Double], to reflect that the type of the keys can vary. However, the keys of each Map returned will all be of the same type, depending ...
gmds's user avatar
  • 19.7k
2 votes
1 answer
966 views

Spark Scala groupBy(cols).agg( 20 sum up functions), how to use map to simplify 20 agg functions?

Let's say A list of Seq("a", "b", "c") and eventDF, eventDF.groupBy("date").agg(sum("a"), sum("b"), sum("c")) works fine. Another case is I have a list with 26 columns val alpha = Seq("a", ... "z")...
seninus's user avatar
  • 256
1 vote
2 answers
92 views

How to make merge or intercalate two maps in scala one by one?

I want to merge two maps in a list of maps as follow: val map1 = {"a" -> 1, "b" -> 2, "c" -> 3} val map2 = {"x" -> 10, "y" -> 20, "z" -> 30} val res = [{"a" ->1, "x" -> 10},{"...
Marcos N. Peralta's user avatar
2 votes
1 answer
727 views

How to change "hostname" while running sbt run?

sbt run defaults binds to 127.0.0.1 but i want to bind it to local ip address of the machine. I have tried to do: sbt run host 192.168.2.123
msbomrel's user avatar
  • 521
2 votes
1 answer
832 views

How to convert java.util.List[java.lang.Long], to java.util.List[Long]

I'm working with the following class and have trouble converting a scala long list to java.util.List[Long]: import scala.collection.JavaConverters._ import org.apache.flink.streaming.connectors....
orestisf's user avatar
  • 353
13 votes
4 answers
7k views

How to fix the Error: "org.jetbrains.jps.incremental.scala.remote.ServerException java.lang.StackOverflowError"

I'm trying to run the main class in the scala-sbt project. Running the class is resulting in stackoverflow Error, StackTrace pasted below. I am sure this is not a code Issue because for the same ...
Pavan_Obj's user avatar
  • 1,121
0 votes
1 answer
80 views

GraphStage with shape of 2-in 2-out

I need to write a custom GraphStage that has two input ports, and two output ports. This GraphStage will allow two otherwise independent flows to affect each other. What shape could I use for that? ...
ig-dev's user avatar
  • 529
0 votes
1 answer
122 views

not found: value fromRegistries

I am trying to use https://doc.akka.io/docs/alpakka/current/mongodb.html as the following: import akka.actor.ActorSystem import akka.stream.ActorMaterializer import cats.data.Chain import com.mongodb....
softshipper's user avatar
  • 33.3k
0 votes
2 answers
1k views

Logging using Logback on Spark StandAlone

We are using Spark StandAlone 2.3.2 and logback-core/logback-classic with 1.2.3 Have very simple Logback configuration file which allows us to log the data to a specific directory and on local I can ...
skjagini's user avatar
  • 3,177
3 votes
0 answers
690 views

What settings are needed to use GraalVM with sbt in Intellij?

I'm want to use graalvm in Intellij with my Scala project and sbt. In my build.sbt I set this: scalacOptions += "-target:1.8.0" I have my graalvm in my path, and see this when I run "java -version": ...
Greg's user avatar
  • 11k
1 vote
1 answer
3k views

How to set the timestamp format when reading CSV in Spark +2.4

I have a Databricks 5.3 cluster on Azure which runs Apache Spark 2.4.0 and Scala 2.11. I'm trying to parse a CSV file with a custom timestamp format but I don't know which datetime pattern format ...
empz's user avatar
  • 12k
1 vote
1 answer
37 views

Scala: Generic function set -> mutable map

I have an issue with the scala type hinting w.r.t generic types. I would like to write a function as follows: def set2mutable_map[A,B](set:Set[A],default:B):collection.mutable.Map[A,B] = { ...
Chris's user avatar
  • 30.1k
3 votes
1 answer
272 views

Wordspec "should" "when" "in"

I'm initializing a companion object for one of my scala test suites. One of the fields in this companion object is lazy evaluated and uses some of the fields in the test suite for initialization. ...
Jackson Kelley's user avatar
3 votes
3 answers
5k views

Spark filter based on the max date record

I am using Spark/Scala to process a Hive table which contains transaction data for each member. I need to get the max record for each member. I did this task using the below code and it works ...
khaled's user avatar
  • 45
2 votes
1 answer
547 views

Do we have Suppressed Exception in Scala?

In Scala, can we add exception throw in a finally block back to the original exception as an suppressed exception? In Java, if we hit an exception in the try block and then another exception in the ...
William Wong's user avatar
3 votes
2 answers
142 views

Scala pattern match with varargs

I've a web server to handle incoming requests. Based on the http method and endpoint I process requests differently. Currently the code handles it: def routes: HttpRequest => Future[HttpResponse] =...
user468587's user avatar
  • 4,969
2 votes
1 answer
794 views

How to append to a list variable in session

I'm new to Gatling and struggling with a basic task. I'm setting up two scenarios. First scenario: I have a list of values, I want to make a request for each value, append the response of the request ...
Madhuri R's user avatar
1 vote
3 answers
169 views

Sum of int elements in list and vector using single function in Scala

How to make this code work? sealed abstract class Addable[A] { def sum(el: Seq[A]): A } class MyAddable[A]() extends Addable[A] { override def sum(el: Seq[A]): A = { el.sum } } val myvec ...
techkuz's user avatar
  • 3,786
1 vote
0 answers
464 views

How to append content of Dstream or RDD to an existing output file - Spark Streaming

I am working on simple SparkStreaming wordcount example to count the number of words in text data received from a data server listening on a TCP socket. I would like to save the content of every ...
A Alnafessah's user avatar
2 votes
1 answer
333 views

Improve the efficiency of Spark SQL in repeated calls to groupBy/count. Pivot the outcome

I have a Spark DataFrame consisting of columns of integers. I want to tabulate each column and pivot the outcome by the column names. In the following toy example, I start with this DataFrame df +--...
Placidia's user avatar
  • 252
-1 votes
1 answer
132 views

Can't do a post scenario with gatling

I'm trying to get familiar with Gatling but i have a problem with the post scenario. I'm using their api, so i tried to add a computer, here is my scenario : var httpProtocol = http.baseUrl("http://...
OthK's user avatar
  • 55
0 votes
1 answer
238 views

How to know if a map is a sub-map of another in Scala?

I've two maps, and i need to know if all the elements in map1 are in map2 also. I think that using sets is a good idea but this doesn't work. map1 = Map("provider" -> pepe, "consumer" -> pipo) ...
Teo Cicciari's user avatar
1 vote
1 answer
116 views

PlayFramework: transform json to json

i have a json like this { "value.first" : "one", "value.second" : "two", "value.third" : "three" } how do I transform it like this in Scala/Play? : { "value": { "first": "one", "...
soumayaKoch's user avatar
0 votes
1 answer
758 views

Unable to resolve unapply

object micro extends App { sealed trait FuncExpr sealed trait FuncSpecialize sealed case class Func(i:Int) extends FuncExpr sealed case class Cube(b:Boolean) extends FuncSpecialize ...
Tilman Zuckmantel's user avatar

1
496 497
498
499 500
2249