Skip to main content
Filter by
Sorted by
Tagged with
0 votes
1 answer
14 views

Read CSV with "§" as delimiter using Databricks autoloader

I'm very new to spark streaming and autoloader and had a query on how we might be able to get autoloader to read a text file with "§" as the delimiter. Below I tried reading the file as a ...
beingmanny's user avatar
-1 votes
0 answers
11 views

Scala Spark JDBC format

I have a scenario with Scala spark JDBC connecting to Snowflake using a private key. When I give a privatekey (of type java.security.PrivateKey) in the option(attached screenshot), it throws an error ...
Ganesh Dogiparthi's user avatar
1 vote
0 answers
34 views

Combining new Scala typeclass syntax with GADT types

I am writing a library in Scala, and decided to jump on the updated (experimental) syntax introduced in version 3.5.0 for working with type classes. So my code has: trait HasBoundary: type Self ...
Mikael Vejdemo-Johansson's user avatar
1 vote
3 answers
83 views

Problems with Java Vector API to sum a list of doubles

I implemented three naive sum methods (in scala) which act on an Array[Double] (or double[]) inline def sum2 = var sum: Double = 0.0 var i: Int = 0 inline val species = ...
Simon's user avatar
  • 431
1 vote
1 answer
51 views

Is there a functional approach to extensible variant types in Scala like in Ocaml?

In Ocaml you can do the following: type attr = .. type attr += Str of string type attr += | Int of int | Float of float Is there a version of this in Scala? I know there is a way to ...
Andrew Fox's user avatar
0 votes
0 answers
33 views

inconsistent behaviour of spark min max window function on boolean column

why am i experiencing inconsistent results when using the min and max aggregate functions over BooleanType columns in Spark SQL? Specifically, when applying these functions within window operations on ...
jaydeemourg's user avatar
0 votes
1 answer
53 views

Save a file in Databricks Workspace using Scala/Java

My goal is to run a Spark job using Databricks, and my challenge is that I can't store files in the local filesystem since the file is saved in the driver, but when my executors tried to access the ...
John Doe's user avatar
  • 423
0 votes
0 answers
55 views
+50

how to mock tracing in scala using open telemetry

object Main { private val tracerInjector = Guice.createInjector(new TracingModule) implicit val tracer: Tracer = tracerInjector.getInstance(classOf[Tracer]) private val testInjector: Injector = ...
sarah w's user avatar
  • 3,415
0 votes
1 answer
44 views

Scala fold function to detect occurrence of a value in Seq[String]

I'm trying to write a scala function to detect whether a site id (site_id) exists within a sequence of targeted site ids (targetedSiteIds). If the site id exists in the Seq, the function ...
Vakindu's user avatar
  • 739
2 votes
0 answers
62 views

Scala Seq.reduceLeft does not update the accumulator value?

I have the following code snippet to combine a Seq of (DataFrame, BloomFilter) tuples. The DFs are in chronological order, and the goal is to union and deduplicate all the DFs while always selecting ...
Alex Guguchev's user avatar
0 votes
1 answer
39 views

unpacking nested tuples after Spark RDD join

The resources for this are scarce and I'm not sure that there's a solution to this issue. Suppose you have 3 simple RDD's. Or more specifically 3 PairRDD's. val rdd1: RDD[(Int, Int)] = sc.parallelize(...
Nizar's user avatar
  • 763
1 vote
0 answers
21 views

Mismatch found for java and native libraries java build version 6.1.0.20180926230239.GA, native build version 6.1.0.20171109191718.GA

I am trying to read a CSV file for using Spark Scala API as part of prepping data. While running the same file I am getting below issue. 2024-09-12 11:26:46,0484 ERROR JniCommon fc/jni_MapRClient.cc:...
Adidev-b's user avatar
1 vote
0 answers
49 views
+50

OAuth 1.0 and X API v2 with Scala and sttp

I try to create an App to send automagically chess tournament announcements to X. Everything works as expected except the OAuth-1.0 authorization. I followed Kevin Williams. HMAC-SHA1 by Aravind_G ...
Christian H. Kuhn's user avatar
3 votes
0 answers
64 views
+50

spark - job cancelled while doing json schema inference

I have a spark job which first is supposed to infer the schema then do the real "job". To infer the schema we use sqlContext.sparkSession.read.json(df.select($"columns").as[String])...
jeekiii's user avatar
  • 261
0 votes
1 answer
35 views

scala tapir test IO endpoint

i cant find anywere how to make a stub of IO tapir endpoint. In the docs https://tapir.softwaremill.com/en/latest/testing.html i can see only example for scala Future. Basically i need to make a stub ...
Дима Шестаев's user avatar
0 votes
1 answer
40 views

How to use aggregation function in groupby

I am having a hard time in transforming the ABC class to CustomClass. Logic is, I want the count in CustomClass to be the total count of the groupyBy of a & b where date matches 1 year filter, ...
Saad's user avatar
  • 967
1 vote
0 answers
24 views

Null Pointer Exception while reading Hbase Dataframe

**When I load an Hbase table I get this exception: ** java.lang.NullPointerException at org.apache.hadoop.hbase.spark.HBaseRelation.<init>(DefaultSource.scala:138) at org.apache.hadoop....
pgb314's user avatar
  • 13
0 votes
2 answers
58 views

Refer to Scala class parameters using a string variable

Say I have this scala class: case class Something( x: Int, y: String ) val someClass = new Something(1, "someString") Is there a way I can refer to the ...
stackTracer's user avatar
0 votes
1 answer
88 views

Scala - List("a","b").map(fn) versus map(fn, List("a","b")

From Scala's site, there is an example of reimplementing method map for List, and the usage of it is as below Case 1 def strlen(s: String): Int = s.length map(strlen, List("a", "bb"...
datnt's user avatar
  • 374
0 votes
1 answer
25 views

Scala spark subtract 2 rdd’s by a common field

I am new to scale and spark need help in subtracting one rdd from another by ignoring a field rdd1.take(2) /* [{'uuid': 'uuid1', 'Id ': 1, 'Name': ‘jack’}, {'uuid': 'uuid2', 'Id': 0, 'Name':...
Java123's user avatar
1 vote
2 answers
46 views

How to generate Verilog rather than SystemVerilog from Chisel?

Here is my GCD.scala package gcd import chisel3._ class GCD extends Module { val io = IO(new Bundle { val value1 = Input(UInt(16.W)) val value2 = Input(UInt(16.W)) val ...
boat's user avatar
  • 13
1 vote
0 answers
51 views

Scala 3.5.0 unreachable case which was working with 3.4.2

I'm using Scala 3.5.0 with these compiler options: -language:strictEquality -Yexplicit-nulls -Wsafe-init -explain -indent -new-syntax -Xfatal-warnings -Wunused:all -...
Karl S.'s user avatar
  • 355
0 votes
0 answers
47 views

ClassCastException in Spark 3.4.1 during Dataset reduce operation

I'm encountering a runtime error related to type casting after upgrading Apache Spark from version 3.2.1 to 3.4.1. This error didn't occur in the previous version. Specifically, a java.lang....
kk jj's user avatar
  • 17
0 votes
1 answer
46 views

How to replace a block of consecutive values if the values immediately before and after the block are the same in spark?

I am new to Spark and am facing a problem that I have been struggling to solve for several days. I have not been able to find a solution yet. Answers in Spark Java or Scala would be appreciated. My ...
Shubham Ingle's user avatar
1 vote
1 answer
66 views

new syntax (indentation) doesn't allow if statement

Here is my code. The idea is that the grades go from 0-5 and it cannot go higher, thus the if command. Highlighted in the code is the part that apparently is not allowed anymore (grade > 5)- I get ...
Patxi Tolonen's user avatar
-1 votes
0 answers
44 views

Replacing incorrect quotes in scala/java

I've a dataframe which has column detail , the data for it looks like following, which is a invalid json and is a mix of single and double quotes , {'Pages':['Stackoverflow',"you'll send ...
gaurav mathur's user avatar
0 votes
0 answers
30 views

Global KieSession for Sliding Window Rules Becomes Null in Spark Streaming with Drools

I'm working on a Scala project using Spark Streaming and Drools, and I have an issue with the global KieSession that I initialized for sliding window rules. I fetch sliding window rules from a ...
Basil Saju's user avatar
0 votes
0 answers
48 views

Added directive for wrapping route

I have added a directive object CustomDirectives extends Directives { def wrapWithTracing(name: String)(innerRoute: => Route)(implicit req: RequestContext): Route = { //start span // ...
Nirmal's user avatar
  • 37
0 votes
1 answer
48 views

regarding scala extension and Givens

Here I have some simple scala program with added extension method to type like string or int, I want to understand what I am missing here so that it works if I can do 1===1 instead of 1.toEqops=...
Depa Reddy's user avatar
1 vote
0 answers
30 views

SecretManagerClient.create() raise NoSuchMethodError io.grpc.MethodDescriptor$Marshaller with Scala on Dataproc Batch

I'm trying to read from google secretmanager. I deploy my scala/spark as Fat JAR on dataproc with SBT. When I call : val client: SecretManagerServiceClient = SecretManagerServiceClient.create() I ...
Arno's user avatar
  • 101
1 vote
1 answer
37 views

FunctionK transformation between two ReaderT instances providing the environment to the result

I have the following definition of effect types that I use in my service: type Traced[F[_], A] = ReaderT[F, TracingCtx, A] type TracedErrorHandling[F[_], E, A] = Traced[EitherT[F, ...
smur89's user avatar
  • 338
2 votes
1 answer
31 views

How to customize serialization of a key of a Map with jsoniter

I have a claas that jsoniter is able to create automatically a custom codec, except for the key of a map. How to create a custom codec just for the key? case class IconReference(area: AreaId, source: ...
david.perez's user avatar
  • 6,762
1 vote
1 answer
37 views

Spark on Dataproc: Slow Data Insertion into BigQuery for Large Datasets (~30M Records)

I have a Scala Spark job running on Google Cloud Dataproc that sources and writes data to Google BigQuery (BQ) tables. The code works fine for smaller datasets, but when processing larger volumes (e.g....
Sekar Ramu's user avatar
0 votes
1 answer
39 views

What is Spark default pattern of the to_date function

I inherited a spark project at my work which convert string column to date one using the to_date function, sometimes with explicit date pattern (i.e. to_date(dateCol, "yyyy-MM-dd")) and ...
Juh_'s user avatar
  • 15.3k
0 votes
0 answers
44 views

Tomcat NIO(non-blocking) Connector Threads Getting Blocked

I am using Jersey web framework with Scala language for one of their services. This service is using NIO(non-blocking IO) connector with Tomcat to handle requests asynchronously. There are 10 ...
Aamir's user avatar
  • 16.9k
0 votes
0 answers
35 views

Support of multiple main functions in a Scala CLI script

I am trying to migrate my Ammonite Scripts to Scala CLI. I am using lihaoyi's mainargs,like import mainargs._ ... @main(doc = "> update ...'.") def update(): Unit = ??? @main(doc = "...
pme's user avatar
  • 14.7k
0 votes
1 answer
41 views

RowEncoder.apply(schema).resolveAndBind() and Row/InternalRow serializer/deserializer equivalent in Spark 3.5

We use RowEncoder.apply(schema).resolveAndBind() in Spark 3.1 to create ExpressionEncoder[row] and create a serializer and deserializer from it. val encoder = Encoders.row(schema).resolveAndBind() val ...
Joy Cheng's user avatar
0 votes
1 answer
33 views

Per EntityType Active Entity Limit in Cluster Sharding Passivation

I have a clustered system that has a number of different entity types with different memory and computation complexity characteristics and would like to use active-entity-limit in my Passivation ...
Arne Claassen's user avatar
0 votes
2 answers
57 views

pekko: Using throttle with foreachAsync

I want to rate limit something that requires asynchronous processing. However, when I try to use throttle with a Sink.foreachAsync it doesn't seem to wait for the futures to complete before allowing ...
Thayne's user avatar
  • 6,933
1 vote
0 answers
57 views

How to have a sbt module for standardized test utility that uses util module

In a fairly large project (~50 submodules), I want to create a "myspec" module that will be the standard way to create test in this project. To be successful, this module need to help with ...
Isammoc's user avatar
  • 893
1 vote
0 answers
31 views

Unable to ge SubGroup Id in KeyCloak using Admin REST API

I'm working with Key Cloak Admin REST API to get the SubGroup ID in scala. Firstly I've generated Access Token. Then I've created group using that Access Token in authorization, group created ...
Ifkaarian's user avatar
0 votes
0 answers
28 views

How to setup test resources correctly to read them in a Scala-Cli project

I use munit and scala-cli in a small Scala 3 project In a test I want to read a resource. Under the project root the resource is at the path: src/test/resources/testResource.txt The Scala file of the ...
halloleo's user avatar
  • 10.1k
0 votes
0 answers
34 views

Spark On-Prem EWS - CDATA JDBC read returning a null body and no attachment list

I am trying to receive the body and attachments IDs of an email from an EWS On-Prem server using the CDATA JDBC driver, however, it is always coming back null. The Subject line, ItemId, dates, and ...
JMore's user avatar
  • 1
2 votes
1 answer
144 views

How to convert String with format "2024-08-19 00:20:04.0 +2:00" to datetime format in Scala

I have string and I wanted to converted to toLocalDateTime with the same format. But when I try the same format "yyyy-MM-dd HH:mm:ss.S Z" it failed to convert because it did not parse the ...
Ayah sa's user avatar
  • 39
0 votes
1 answer
38 views

Scala Test squashing nested suites future asserts

I'm running nested test suites that call futures, any exceptions or assertions from inside a transform are being ignored by the test handler. class TestManager extends AsyncFlatSpec { val reporter: ...
rjmxtt's user avatar
  • 23
0 votes
0 answers
49 views

Scala doesn't detect MANIFEST.MD leading: Caused by: java.lang.ClassNotFoundException:

I am very aware that this covers the same error and output Caused by: java.lang.ClassNotFoundException. The distinction here is i'm not using Java, I'm using Scala. I have not seen a question that ...
Michael Schmidt's user avatar
2 votes
0 answers
56 views

scala compiler complains about given CanEqual being unused but it is

In our project, we are using strict equality (-language:strictEquality), so when I have a type alias like the following I also have to define a CanEqual explicitly object Foo { type A = Double | ...
marcopiii's user avatar
  • 878
1 vote
1 answer
48 views

How to serialize a Map[String, Any] with jsoniter

I'd like to adopt jsoniter, a wonderful JSON serailizer lib for Scala. I'm using Scala 3. The only stopper is that I need to serialize this: Map[String, Any] Where Any is a class where a given codec ...
david.perez's user avatar
  • 6,762
1 vote
1 answer
40 views

Impossible to load latest Akka from SBT with Scala

I load some Akka libraries here, in fact the only dependencies in my project. It is Scala based and very simple and here's the snipped of my build.sbt: scalaVersion := "3.4.2" ...
Ma Sok's user avatar
  • 11
0 votes
0 answers
30 views

I am getting an error while running my spark job in apache airflow

I am running this job in apache airflow my code: package ScalaWIthAirflow import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions.col object Airflow extends App { val spark =...
omar sobhy's user avatar

1
2 3 4 5
2254