Questions tagged [scala]

Scala is a general-purpose programming language principally targeting the Java Virtual Machine. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles. Its key features are: an advanced static type system with type inference; function types; pattern-matching; implicit parameters and conversions; operator overloading; full interoperability with Java; concurrency

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

sbt scoverage plugin error: java.lang.NoSuchMethodError: sbt.package$.singleFileJsonFormatter()Lsjsonnew/JsonFormat;

i am trying to use scoverage for code coverage in my plugins.sbt i have added addSbtPlugin("org.scoverage" % "sbt-scoverage" % "2.0.9") here is my build.sbt ThisBuild / ...
swaheed's user avatar
  • 3,761
0 votes
0 answers
8 views

How can we implement Requirements Traceability Matrix (RTM) in Scala automation scripts for API testing? [closed]

How to automatically implement Requirements Traceability Matrix (RTM) in Scala automation scripts? I am looking for a way to automatically implement RTM in my Scala automation scripts. Is there any ...
freak7's user avatar
  • 1
0 votes
0 answers
19 views

Spark fails to reconfigure log4j2 when using plug-ins bundled in fat jars

After upgrading Spark code base from 3.2 to 3.3+, porting Log4j1 to Log4j2 code, we find that Spark doesn't let us reconfigure Log4j programmatically like we were able to do before. I use Maven shade ...
jeanfrancis's user avatar
1 vote
1 answer
29 views

Double type with custom comparation in Scala

I'm migrating some old Java code to Scala which uses a "precision" threshold for comparing real values. A fragment of the already translated to Scala code would be: object CompareDouble { ...
xandor19's user avatar
1 vote
2 answers
49 views

Consuming io.Source iterator conditionally

I'd like to read a file with scala.io.Source in chunks conditionally, so after I processed the chunk, I can continue reading later from the same place. Suppose I have a file with 10 lines with numbers ...
Sergey Bushmanov's user avatar
0 votes
0 answers
32 views

Calculating list of all files in S3 directory resulting in compilation error

I am trying to calculate size of all files in a given S3 folder using Scala. The following functions gives a compilation error. def getFolderSize(filePath: String): Long = { val objects = s3client....
vvazza's user avatar
  • 361
0 votes
0 answers
23 views

Classes code generated from protobuf file not found in Scalapb

I'm new to Scalapb and currently I'm trying to compile a proto file to Scala classes as below build.sbt (made by George Leung / phiSgr - run normally on his demo project): // the enterprisePackage ...
Nguyễn Phúc Trí's user avatar
0 votes
0 answers
39 views

Converting Java Optional to Scala Option in Interfaces

I am working with a Java interface that is implemented in Scala. The Java interface returns an optional: public interface MyJavaInterface { Optional<Kitten> fetchKittenIfAny(String name)); } ...
PistachioGirl08's user avatar
0 votes
0 answers
23 views

Unable to read file from s3 using spark-shell

I am trying to read a file from s3 from spark-shell. I am initiating spark-shell using the below command including the required packages. spark-shell --packages io.delta:delta-core_2.12:2.0.0,org....
Robinhood's user avatar
  • 100
2 votes
2 answers
68 views

Circe encoder for generic ADT with context bound

I have the following setup: import io.circe.generic.extras.Configuration import io.circe.generic.extras.auto._ import io.circe.syntax._ implicit val Conf: Configuration = Configuration ....
sanyi14ka's user avatar
  • 829
0 votes
0 answers
25 views

NoSuchMethodError while trying to save parquet files on s3 bucket

I am trying to save data into s3 bucket using Scala code. I am always getting following error. How can I resolve this error? After changing jar version of hadoop-aws from 3.3.0 to 3.3.1 version, the ...
Omkar  Gaikwad's user avatar
0 votes
2 answers
31 views

Azure Databricks Spark (Scala) - Reading a MultiLine json with schema returns null rows

I have a multiline JSON file. When I read the JSON file without a schema but multiLine option set to true, I get records well. However, reading the file with both schema and multiLine option returns ...
Kenneth Chirchir's user avatar
0 votes
0 answers
23 views

slick 3.3.3, oracle 11g, scala

I want to use slick.jdbc.OracleProfile to support the autoincremention of primary key with oracle 11g. I use slick version 3.3.3. I've noticed that in slick.jdbc.OracleProfile there is method that ...
Im89's user avatar
  • 139
1 vote
1 answer
36 views

Scala Spark Iceberg writeStream. How to set bucket?

I'm trying to write data to Iceberg table in Spark streaming (written in Scala). Writer code: val streamResult = joined.writeStream .format("iceberg") .partitionBy("...
Netrunner's user avatar
0 votes
0 answers
44 views

Symbol 'type io.gatling.core.check.FindCheckBuilder' is missing from the classpath

I tried to implement gatling with Grpc as the following script, by phiSgr: package load import com.github.phisgr.gatling.grpc.Predef._ import example.myapp.helloworld.grpc.hello_world.{...
Nguyễn Phúc Trí's user avatar
0 votes
0 answers
44 views

Generation of source code based on specification files

I am starting a new project in Scala (my first one in this language) and I'm trying to automate modelling my main data class. I want to have a specification file from which I can just generate the ...
user3617992's user avatar
1 vote
0 answers
25 views

Spark Hive Integration issue with eclipse

i am using cloudera-quickstart-vm-5.12.0-0-virtualbox-disk1 for my BigData practise. i was trying to integrate spark and hive using a scala code. this scala code is being written in windows eclipse. ...
Rajarshi Maity's user avatar
1 vote
0 answers
26 views

How to convert a sequence into a streaming DataFrame?

Having a sequence of strings Seq[String], how can I convert it into a streaming DataFrame? The functions SparkSession.createDataFrame(...) create static DataFrames only. And the class MemoryStream ...
pgrandjean's user avatar
0 votes
0 answers
37 views

import scala.Serializable not found in scala 2.13

I've come here from questions like: java: cannot access scala.Serializable class file for scala.Serializable not found And other similar questions. Recently, I've taken the initiative to move my from ...
RMSD's user avatar
  • 476
0 votes
0 answers
112 views

Scala: java.lang.NoClassDefFoundError: Could not initialize class sbt.internal.parser.SbtParser

I'm unable to use sbt or run tests from within IntelliJ on an existing Scala project. My setup seems basic. % brew install coursier/formulas/coursier && cs setup % cs setup Checking if a JVM ...
Mac's user avatar
  • 1
0 votes
0 answers
17 views

Convert Json Dataset to normal Dataframe in spark using scala with custom schema

I followed this post's comment from @stefanobaghino (Efficient way to generate large randomized data in Spark) to generate dataframe with random values. I then stored that df in json using - df.select(...
Glarixon's user avatar
0 votes
1 answer
40 views

Akka Typed: Replying to messages (tell) sent from different actors

With Akka classic, one can easily implement a service (as an actor: ServiceActor) that receives requests from different actors (i.e., actors of different types: RequestorA, RequestorB, etc.), ...
david's user avatar
  • 311
0 votes
0 answers
30 views

ScalaPb annotations on sealed trait

I am trying to put an annotation on top of a generated sealed trait using scalaPb this way: message SealedTrait { option (scalapb.message).annotations ="@zio.json.jsonDiscriminator("...
Matroska's user avatar
  • 6,895
1 vote
1 answer
70 views

Aggregate Data based on date difference range in Spark

I have an scenario where I need to aggregate data based on the Date difference of within two days range and aggregated date should have the oldest date as show in output. Input Dataframe InputDate ...
manmur's user avatar
  • 19
0 votes
1 answer
49 views

Gatling with GRPC setup, guide and more?

hope you have a good day today. I am a new user to both Gatling and GRPC, trying to set up a demo for investigating. Firstly, this is my set up for akka-grpc build.sbt file: name := "akka-grpc-...
Nguyễn Phúc Trí's user avatar
0 votes
0 answers
28 views

Mocking a Post request with apache.http.client.HttpClient in Scala (2.11)

I'm trying to implement some unit tests for my HttpClient (in Scala 2.11); Here's my client: import org.apache.http.client.methods.HttpPost import org.apache.http.impl.client.HttpClientBuilder import ...
fricadelle's user avatar
0 votes
1 answer
84 views

How to write a function defined for a subset of all enumeration values?

Suppose I've got enumeration A : object A extends Enumeration { type A = Int; val A1 = 1; val A2 = 2; val A3 = 3 } Also, I have a function defined only for A1 or A2 but not for A3. def foo(a: A): Int ...
Michael's user avatar
  • 41.1k
1 vote
0 answers
29 views

What does the warn message "failed to create Hotspot JVM interface, using NilJvm instead" mean?

When migrating from Jre11 to Jre17 our backend applications started to log "failed to create Hotspot JVM interface, using NilJvm instead" when starting up. We use scala with finagle and ...
ChillyPro's user avatar
  • 182
0 votes
1 answer
38 views

Spark Delta Merge runs too slow in Test

We have a Scalatest + Spark based Test. The Data transformation is fairly complex with lots of transformation and the final step is Delta merge. The tests have minimal input data (around 20 KB) and ...
Ganesha's user avatar
  • 145
0 votes
1 answer
31 views

A proper way to identify a genericly-typed contents on an Any collection in Scala

first of all, I know many resources that explain the issue with type erasure, how to use ClassTags, TypeTags, etc... Yet, in this case I would kindly ask to focus on an exact problem, and suggest the ...
Uko's user avatar
  • 13.1k
0 votes
0 answers
17 views

Is it possible to set scala imports layout in editorconfig using IntelliJ

I want to set a rule for imports using .editorconfig for my scala project the same as Java rule : ij_java_imports_layout = *, |, javax.**, java.**, |, $* Idea says property not supported: ...
Niko's user avatar
  • 402
0 votes
1 answer
33 views

Scala nested match pattern matching

I'm writing a sort of rules engine that traverses a tree and establishes a context for each sub-tree and then uses match conditions to validate the tree and infer information about the nodes. I had ...
David Regan's user avatar
0 votes
0 answers
45 views

Issue with bigtable mutateRow() call to update row. The update operation is not working as expected without any errors

I am trying to update an existing row with key "2023-09-25" in my bigtable table. The mutateRow() call succeeds, but I don't see update happening. The row key "2023-09-25" still ...
Krishna Shetty's user avatar
0 votes
1 answer
14 views

passing the Spark ParserInterface in the udf

I have a data frame that has user queries as one of the columns. I have a function that will be deriving the important columns in these queries and returning the columns as a set of string. This set ...
Data Geek's user avatar
0 votes
1 answer
13 views

Play Framework Unit Testing Controllers With Compile Time DI

I have the following Controller in Play Framework: final class ProvisioningController(bindings: Bindings[IO], val controllerComponents: ControllerComponents) extends ControllerBase { ...... ........
joesan's user avatar
  • 14.1k
-1 votes
0 answers
19 views

filter condition for one to many [closed]

I've tables, like below primary locationId, locationTypeName, cityId, cityTypeName, countryId, countryTypeName 121, PXB, 200, CBP, 900, OPT 122, PXB, 210, CBP, 980, OPT secondaryPXB TypeNames,...
Maria's user avatar
  • 492
0 votes
1 answer
23 views

Using readstream against kafka in databricks, how can I tell the stream to gracefully self-terminate after a certain time?

I'm using Azure Databricks to run scala notebooks that read from several kafka topics. Right now they run continuously, but I'd like them to wrap up after 10 minutes and allow the scheduler to ...
Troy Terry's user avatar
0 votes
1 answer
49 views

Type inference issue using Pekko (Akka) AskPattern

I am using the Ask pattern to implement simple RPC. import org.apache.pekko.actor.typed.{ActorRef, Scheduler} import org.apache.pekko.actor.typed.scaladsl.AskPattern.Askable import org.apache.pekko....
Tim's user avatar
  • 26.8k
0 votes
1 answer
18 views

class RestHighLevelClient in package client is deprecated

I am upgrading elastic search from 7.9.2 to 7.17.6 and then to 8.4.2, in the first step i am upgrading the es cluster to the 7.17.6 version , and compiling all other ES clients, the compilation is ...
itIsNaz's user avatar
  • 631
0 votes
0 answers
12 views

Spark streaming sourceArchiveDir does not move file to archive directory

I'm using scala and spark with the version 3.4.1. While streaming using Spark, I give the cleanSource option as "archive" and the sourceArchiveDir option as "archived" as in the ...
Y. Emre's user avatar
-1 votes
1 answer
18 views

What cause a "Spike Load" in Load Testing?

I am a new user to Gatling and Scala, I have tried to set up a script and send to reqres.in, however when I check the report, there seems to be some spike load as below What/Why these spikes appear ...
Nguyễn Phúc Trí's user avatar
0 votes
0 answers
34 views

How to read JSON Messages from Kafka Topic using Apache Flink in Scala

I am trying to read JSON Messages from Kafka topic using Apache Flink and Scala. This is an example of one massege: {"brand": "apple", "category": "phone", &...
Yaser's user avatar
  • 29
0 votes
1 answer
35 views

Write in Hbase with Spark 2.3 and assign timestamp

I am writing in Hbase using HBase-Spark connector, but I want to assign a Timestamp to each row based in a column in the DS. It is possible to do such thing? Here is an example of what I a want to do ...
jmjO's user avatar
  • 1
0 votes
1 answer
37 views

Spark UDF throws NullPointerException

I have spark UDF, which I call from the SQL statement. This is how I have defined and register UDF function: UDF: import org.apache.spark.sql.functions.udf def udf_temp_emp (_type: String, _name: ...
ConfusedDeveloper's user avatar
1 vote
0 answers
59 views

Connecting to Test Container with Postgres Fails

For one of my project, I'm using the Test Containers set up. What I do here is basically create a Postgres image like this: # To build this Dockerfile run the folloing command # docker build -t custom-...
joesan's user avatar
  • 14.1k
0 votes
1 answer
36 views

Scala Generic Class invoke class method

I am currently learning generic class in Scala. I have created below codes, which it has a class to store a list of items that extends from ItemInterface; and it can use printAllId as a way to access ...
Tobias To's user avatar
1 vote
0 answers
38 views

Running jooq from testcontainers with a flyway migration via SBT

I've seen some guides on how to achieve this in java (gradle/maven) but I can't seem to get it to work with an SBT project. The goal is for my current setup to not require an available postgres ...
H4uZ's user avatar
  • 137
2 votes
1 answer
28 views

last function with Window orderBy expression not working as expected

Apache Spark: 3.3.2, Google Dataproc 2.1 scala> val orderDF = Seq(("1234","created","2023-09-17 00:00:00"),("1234","updated","2023-09-17 01:00:...
Vikas's user avatar
  • 182
0 votes
0 answers
32 views

Ambiguous Implicit Resolution Error With Doobie

I have this function that inserts a One To Many relationship representation between 3 tables: def insertRelations(cs: TableA): Fragment = { sql"""INSERT INTO my_schema.table_a ( ...
joesan's user avatar
  • 14.1k
0 votes
1 answer
56 views

sbt test does not work and all the tests fail

I have installed rocket-chip and it needs chisel to compile i have also downloaded chisel , but when i run sbt test all the tests fail, am i doing something wrong . is there an alternative for sbt ...
saras's user avatar
  • 3

1
2 3 4 5
2237