How do You organize the Spark development workflow?
My way:
- Local hadoop/yarn service.
- Local spark service.
- Intellij on one screen
- Terminal with running sbt console
- After I change Spark app code, I switch to terminal and run "package" to compile to jar and "submitSpark" which is stb task that runs spark-submit
- Wait for exception in sbt console :)
I also tried to work with spark-shell:
- Run shell and load previously written app.
- Write line in shell
- Evaluate it
- If it's fine copy to IDE
- After few 2,3,4, paste code to IDE, compile spark app and start again
Is there any way to develop Spark apps faster?