I am working on writing a process which will write to SQL Server from Spark- Scala application. Its generating multiple INSERT BULK per partition of data (as expected), batchsize 100K records.
As I am monitoring the transaction log, I can see its filling up, and I was hoping with INSERT BULK it will not.
Can you please suggest, how I can achieve commits per batch.