Revisit SparkKubernetesOperator and SparkKubernetesSensor

[kori73]

It seems currently the overall structure of SparkKubernetesOperator and SparkKubernetesSensor have diverged from KubernetesPodOperator. I believe revisiting SparkKubernetes related modules would be beneficial. Here are some improvements I think will make Spark Kubernetes a more mature functionality.

1. Merge SparkKubernetesOperator and SparkKubernetesSensor into a single operator. This will enable us to benefit from auto-retry, for example when there is some random network error.

2. KubernetesPodOperator has an await_pod_start method but SparkKubernetesSensor does not. This results in the tasks keeping forever in Running state in some cases. Examples are: cluster does not have nodes with enough CPU/memory for the driver, nodes have untolerated taints, volume mounting errors etc.

3. Overall Spark Kubernetes does not utilize the helper functions/utilities provided and implements similar functionality. For example, KubernetesPodOperator uses PodManager to read logs, events and await pod start, but SparkKubernetes does not.

There might be other functionality that would improve SparkKubernetes that I'm missing right now, but I would love to hear them. Would you be interested in these changes considering some of them will be potentially backward incompatible? If yes, I would love to implement these ideas and others in the comments.

[Taragolis]

Here are some improvements I think will make Spark Kubernetes a more mature functionality.

Feel free make a PR.

You don't have to wait for someone to tell you "Awesome idea. lets implement it", it would be enough that personally you think that this might be awesome idea