driver image does not crashes on Java exceptions

[dberardo]

Thanks for the helpful material.

I have manged to setup a test spark cluster from the stack file suggested and everything worked. I had to created custom docker images to update the python and pyspark versions , which gave me some headache, but is done now (can make a PR in github if needed).

I have now one issue with the spark driver and wanted to check if there is a known solution. I am working with spark structured streaming and thus need to deploy some persistent spark job. What i have done here is building a custom spark-dirver docker image which starts a python code and deploys the job in "cluster" mode. This works, however if any Java exception is triggered, the job dies, but the container does not ... which should be the intended behavior.

The command CMD i am using for starting the python script is:

CMD /spark/bin/spark-submit \
--packages org.apache.spark:spark-streaming-kafka-0-10_2.12:${SPARK_VERSION},org.apache.spark:spark-sql-kafka-0-10_2.12:${SPARK_VERSION} \
--executor-memory $SPARK_EXECUTOR_MEMORY \
--executor-cores $SPARK_EXECUTOR_CORES \
# --supervise \
script.py

and i have tried both with and without --supervise but without success.

any hint why the container does not fail with exit code 1 when java exceptions occur?