112,681
questions
0
votes
1
answer
14
views
Read CSV with "§" as delimiter using Databricks autoloader
I'm very new to spark streaming and autoloader and had a query on how we might be able to get autoloader to read a text file with "§" as the delimiter. Below I tried reading the file as a ...
-1
votes
0
answers
11
views
Scala Spark JDBC format
I have a scenario with Scala spark JDBC connecting to Snowflake using a private key.
When I give a privatekey (of type java.security.PrivateKey) in the
option(attached screenshot), it throws an error ...
1
vote
0
answers
34
views
Combining new Scala typeclass syntax with GADT types
I am writing a library in Scala, and decided to jump on the updated (experimental) syntax introduced in version 3.5.0 for working with type classes. So my code has:
trait HasBoundary:
type Self
...
1
vote
3
answers
83
views
Problems with Java Vector API to sum a list of doubles
I implemented three naive sum methods (in scala) which act on an Array[Double] (or double[])
inline def sum2 =
var sum: Double = 0.0
var i: Int = 0
inline val species = ...
1
vote
1
answer
51
views
Is there a functional approach to extensible variant types in Scala like in Ocaml?
In Ocaml you can do the following:
type attr = ..
type attr += Str of string
type attr +=
| Int of int
| Float of float
Is there a version of this in Scala?
I know there is a way to ...
0
votes
0
answers
33
views
inconsistent behaviour of spark min max window function on boolean column
why am i experiencing inconsistent results when using the min and max aggregate functions over BooleanType columns in Spark SQL? Specifically, when applying these functions within window operations on ...
0
votes
1
answer
53
views
Save a file in Databricks Workspace using Scala/Java
My goal is to run a Spark job using Databricks, and my challenge is that I can't store files in the local filesystem since the file is saved in the driver, but when my executors tried to access the ...
0
votes
0
answers
55
views
+50
how to mock tracing in scala using open telemetry
object Main {
private val tracerInjector = Guice.createInjector(new TracingModule)
implicit val tracer: Tracer = tracerInjector.getInstance(classOf[Tracer])
private val testInjector: Injector = ...
0
votes
1
answer
44
views
Scala fold function to detect occurrence of a value in Seq[String]
I'm trying to write a scala function to detect whether a site id (site_id) exists within a sequence of targeted site ids (targetedSiteIds).
If the site id exists in the Seq, the function ...
2
votes
0
answers
62
views
Scala Seq.reduceLeft does not update the accumulator value?
I have the following code snippet to combine a Seq of (DataFrame, BloomFilter) tuples. The DFs are in chronological order, and the goal is to union and deduplicate all the DFs while always selecting ...
0
votes
1
answer
39
views
unpacking nested tuples after Spark RDD join
The resources for this are scarce and I'm not sure that there's a solution to this issue.
Suppose you have 3 simple RDD's. Or more specifically 3 PairRDD's.
val rdd1: RDD[(Int, Int)] = sc.parallelize(...
1
vote
0
answers
21
views
Mismatch found for java and native libraries java build version 6.1.0.20180926230239.GA, native build version 6.1.0.20171109191718.GA
I am trying to read a CSV file for using Spark Scala API as part of prepping data.
While running the same file I am getting below issue.
2024-09-12 11:26:46,0484 ERROR JniCommon fc/jni_MapRClient.cc:...
1
vote
0
answers
49
views
+50
OAuth 1.0 and X API v2 with Scala and sttp
I try to create an App to send automagically chess tournament announcements to X. Everything works as expected except the OAuth-1.0 authorization. I followed Kevin Williams. HMAC-SHA1 by Aravind_G
...
3
votes
0
answers
64
views
+50
spark - job cancelled while doing json schema inference
I have a spark job which first is supposed to infer the schema then do the real "job". To infer the schema we use
sqlContext.sparkSession.read.json(df.select($"columns").as[String])...
0
votes
1
answer
35
views
scala tapir test IO endpoint
i cant find anywere how to make a stub of IO tapir endpoint.
In the docs https://tapir.softwaremill.com/en/latest/testing.html
i can see only example for scala Future.
Basically i need to make a stub ...
0
votes
1
answer
40
views
How to use aggregation function in groupby
I am having a hard time in transforming the ABC class to CustomClass.
Logic is, I want the count in CustomClass to be the total count of the groupyBy of a & b where date matches 1 year filter, ...
1
vote
0
answers
24
views
Null Pointer Exception while reading Hbase Dataframe
**When I load an Hbase table I get this exception: **
java.lang.NullPointerException
at org.apache.hadoop.hbase.spark.HBaseRelation.<init>(DefaultSource.scala:138)
at org.apache.hadoop....
0
votes
2
answers
58
views
Refer to Scala class parameters using a string variable
Say I have this scala class:
case class Something(
x: Int,
y: String
)
val someClass = new Something(1, "someString")
Is there a way I can refer to the ...
0
votes
1
answer
88
views
Scala - List("a","b").map(fn) versus map(fn, List("a","b")
From Scala's site, there is an example of reimplementing method map for List, and the usage of it is as below
Case 1
def strlen(s: String): Int = s.length
map(strlen, List("a", "bb"...
0
votes
1
answer
25
views
Scala spark subtract 2 rdd’s by a common field
I am new to scale and spark
need help in subtracting one rdd from another by ignoring a field
rdd1.take(2)
/*
[{'uuid': 'uuid1',
'Id ': 1,
'Name': ‘jack’},
{'uuid': 'uuid2',
'Id': 0,
'Name':...
1
vote
2
answers
46
views
How to generate Verilog rather than SystemVerilog from Chisel?
Here is my GCD.scala
package gcd
import chisel3._
class GCD extends Module {
val io = IO(new Bundle {
val value1 = Input(UInt(16.W))
val value2 = Input(UInt(16.W))
val ...
1
vote
0
answers
51
views
Scala 3.5.0 unreachable case which was working with 3.4.2
I'm using Scala 3.5.0 with these compiler options:
-language:strictEquality
-Yexplicit-nulls
-Wsafe-init
-explain
-indent
-new-syntax
-Xfatal-warnings
-Wunused:all
-...
0
votes
0
answers
47
views
ClassCastException in Spark 3.4.1 during Dataset reduce operation
I'm encountering a runtime error related to type casting after upgrading Apache Spark from version 3.2.1 to 3.4.1. This error didn't occur in the previous version. Specifically, a java.lang....
0
votes
1
answer
46
views
How to replace a block of consecutive values if the values immediately before and after the block are the same in spark?
I am new to Spark and am facing a problem that I have been struggling to solve for several days. I have not been able to find a solution yet. Answers in Spark Java or Scala would be appreciated. My ...
1
vote
1
answer
66
views
new syntax (indentation) doesn't allow if statement
Here is my code. The idea is that the grades go from 0-5 and it cannot go higher, thus the if command.
Highlighted in the code is the part that apparently is not allowed anymore (grade > 5)- I get ...
-1
votes
0
answers
44
views
Replacing incorrect quotes in scala/java
I've a dataframe which has column detail , the data for it looks like following, which is a invalid json and is a mix of single and double quotes ,
{'Pages':['Stackoverflow',"you'll send ...
0
votes
0
answers
30
views
Global KieSession for Sliding Window Rules Becomes Null in Spark Streaming with Drools
I'm working on a Scala project using Spark Streaming and Drools, and I have an issue with the global KieSession that I initialized for sliding window rules. I fetch sliding window rules from a ...
0
votes
0
answers
48
views
Added directive for wrapping route
I have added a directive
object CustomDirectives extends Directives {
def wrapWithTracing(name: String)(innerRoute: => Route)(implicit req: RequestContext): Route = {
//start span
// ...
0
votes
1
answer
48
views
regarding scala extension and Givens
Here I have some simple scala program with added extension method to type like string or int, I want to understand what I am missing here so that it works if I can do 1===1 instead of 1.toEqops=...
1
vote
0
answers
30
views
SecretManagerClient.create() raise NoSuchMethodError io.grpc.MethodDescriptor$Marshaller with Scala on Dataproc Batch
I'm trying to read from google secretmanager. I deploy my scala/spark as Fat JAR on dataproc with SBT.
When I call :
val client: SecretManagerServiceClient = SecretManagerServiceClient.create()
I ...
1
vote
1
answer
37
views
FunctionK transformation between two ReaderT instances providing the environment to the result
I have the following definition of effect types that I use in my service:
type Traced[F[_], A] = ReaderT[F, TracingCtx, A]
type TracedErrorHandling[F[_], E, A] = Traced[EitherT[F, ...
2
votes
1
answer
31
views
How to customize serialization of a key of a Map with jsoniter
I have a claas that jsoniter is able to create automatically a custom codec, except for the key of a map.
How to create a custom codec just for the key?
case class IconReference(area: AreaId, source: ...
1
vote
1
answer
37
views
Spark on Dataproc: Slow Data Insertion into BigQuery for Large Datasets (~30M Records)
I have a Scala Spark job running on Google Cloud Dataproc that sources and writes data to Google BigQuery (BQ) tables. The code works fine for smaller datasets, but when processing larger volumes (e.g....
0
votes
1
answer
39
views
What is Spark default pattern of the to_date function
I inherited a spark project at my work which convert string column to date one using the to_date function, sometimes with explicit date pattern (i.e. to_date(dateCol, "yyyy-MM-dd")) and ...
0
votes
0
answers
44
views
Tomcat NIO(non-blocking) Connector Threads Getting Blocked
I am using Jersey web framework with Scala language for one of their services.
This service is using NIO(non-blocking IO) connector with Tomcat to handle requests asynchronously. There are 10 ...
0
votes
0
answers
35
views
Support of multiple main functions in a Scala CLI script
I am trying to migrate my Ammonite Scripts to Scala CLI.
I am using lihaoyi's mainargs,like
import mainargs._
...
@main(doc = "> update ...'.")
def update(): Unit = ???
@main(doc = "...
0
votes
1
answer
41
views
RowEncoder.apply(schema).resolveAndBind() and Row/InternalRow serializer/deserializer equivalent in Spark 3.5
We use RowEncoder.apply(schema).resolveAndBind() in Spark 3.1 to create ExpressionEncoder[row] and create a serializer and deserializer from it.
val encoder = Encoders.row(schema).resolveAndBind()
val ...
0
votes
1
answer
33
views
Per EntityType Active Entity Limit in Cluster Sharding Passivation
I have a clustered system that has a number of different entity types with different memory and computation complexity characteristics and would like to use active-entity-limit in my Passivation ...
0
votes
2
answers
57
views
pekko: Using throttle with foreachAsync
I want to rate limit something that requires asynchronous processing.
However, when I try to use throttle with a Sink.foreachAsync it doesn't seem to wait for the futures to complete before allowing ...
1
vote
0
answers
57
views
How to have a sbt module for standardized test utility that uses util module
In a fairly large project (~50 submodules), I want to create a "myspec" module that will be the standard way to create test in this project.
To be successful, this module need to help with ...
1
vote
0
answers
31
views
Unable to ge SubGroup Id in KeyCloak using Admin REST API
I'm working with Key Cloak Admin REST API to get the SubGroup ID in scala.
Firstly I've generated Access Token.
Then I've created group using that Access Token in authorization, group created ...
0
votes
0
answers
28
views
How to setup test resources correctly to read them in a Scala-Cli project
I use munit and scala-cli in a small Scala 3 project
In a test I want to read a resource. Under the project root the resource is at the path:
src/test/resources/testResource.txt
The Scala file of the ...
0
votes
0
answers
34
views
Spark On-Prem EWS - CDATA JDBC read returning a null body and no attachment list
I am trying to receive the body and attachments IDs of an email from an EWS On-Prem server using the CDATA JDBC driver, however, it is always coming back null. The Subject line, ItemId, dates, and ...
2
votes
1
answer
144
views
How to convert String with format "2024-08-19 00:20:04.0 +2:00" to datetime format in Scala
I have string and I wanted to converted to toLocalDateTime with the same format. But when I try the same format "yyyy-MM-dd HH:mm:ss.S Z" it failed to convert because it did not parse the ...
0
votes
1
answer
38
views
Scala Test squashing nested suites future asserts
I'm running nested test suites that call futures, any exceptions or assertions from inside a transform are being ignored by the test handler.
class TestManager extends AsyncFlatSpec {
val reporter: ...
0
votes
0
answers
49
views
Scala doesn't detect MANIFEST.MD leading: Caused by: java.lang.ClassNotFoundException:
I am very aware that this covers the same error and output Caused by: java.lang.ClassNotFoundException.
The distinction here is i'm not using Java, I'm using Scala. I have not seen a question that ...
2
votes
0
answers
56
views
scala compiler complains about given CanEqual being unused but it is
In our project, we are using strict equality (-language:strictEquality), so when I have a type alias like the following I also have to define a CanEqual explicitly
object Foo {
type A = Double | ...
1
vote
1
answer
48
views
How to serialize a Map[String, Any] with jsoniter
I'd like to adopt jsoniter, a wonderful JSON serailizer lib for Scala. I'm using Scala 3.
The only stopper is that I need to serialize this:
Map[String, Any]
Where Any is a class where a given codec ...
1
vote
1
answer
40
views
Impossible to load latest Akka from SBT with Scala
I load some Akka libraries here, in fact the only dependencies in my project. It is Scala based and very simple and here's the snipped of my build.sbt:
scalaVersion := "3.4.2"
...
0
votes
0
answers
30
views
I am getting an error while running my spark job in apache airflow
I am running this job in apache airflow
my code:
package ScalaWIthAirflow
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.col
object Airflow extends App {
val spark =...