1

I want to use Spark Structured Streaming to filter an Event Hub stream to separate objects of different type (JSON objects with different schemas) and write each object type to it's own file store.

Am I right if I assume the only supported way to do this is to create a ForeachWriter, cache the microbatch and do the filtering and writing in the process method?

I don't want to create one read stream per filtered write stream, that would severely hamper the egress capacity from the Event Hub.

1

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.