8

What I would like to do is to run some backup scripts on each of Kubernetes nodes periodically. I want it to run inside Kubernetes cluster in contrast to just adding script to each node's crontab. This is because I will store backup on the volume mounted to the node by Kubernetes. It differs from the configuration but it could be CIFS filesystem mounted by Flex plugin or awsElasticBlockStore.

It would be perfect if CronJob will be able to template DaemonSet (instead of fixing it as jobTemplate) and there will be possibility to set DaemonSet restart policy to OnFailure.

I would like to avoid defining n different CronJobs for each of n nodes and then associate them together by defining nodeSelectors since this will be not so convenient to maintain in environment where nodes count changes dynamically.

What I can see problem was discussed here without any clear conclusion: https://github.com/kubernetes/kubernetes/issues/36601

Maybe do you have any hacks or tricks to achieve this?

2 Answers 2

3

You can use DaemonSet with the following bash script:

  while :; do
   currenttime=$(date +%H:%M)
   if [[ "$currenttime" > "23:00" ]] && [[ "$currenttime" < "23:05" ]]; then
     do_something
   else
     sleep 60
   fi
   test "$?" -gt 0 && notify_failed_job
  done
3
  • This is the most hacky way I can imagine ;) Anyway, it will work. Do you think that there is no any other more Kubernetes native solution? Apr 30, 2019 at 7:22
  • 2
    Actually there are some issues (e.g. github.com/kubernetes/kubernetes/issues/36601) on k8s GitHub about the problem and there is no native support. At the moment your solution is the most reasonably way to do it so I marked it as accepted. Aug 6, 2019 at 9:04
  • 2
    on line 3 the || should be an && otherwise do_something is always called Aug 4, 2020 at 23:48
1

i know i am late to party,

First option :

Using the parallelism to run multiple Job PODs, with topologySpreadConstraints to spread/schedule the PODs on all the nodes.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: mycronjob
  labels:
    jobgroup: parallel
spec:
  schedule: "*/5 * * * *"
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 0
  jobTemplate:
    spec:
      template:
        metadata:
          name: kubejob
          labels:
            jobgroup: parallel
        spec:
          topologySpreadConstraints:
            - maxSkew: 2
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: DoNotSchedule
              labelSelector:
                matchLabels:
                  jobgroup: parallel
          containers:
          - name: mycron-container
            image: alpine
            imagePullPolicy: IfNotPresent
            command: ['sh', '-c', 'echo Job Pod is Running ; sleep 10']
          restartPolicy: OnFailure
          terminationGracePeriodSeconds: 0
      parallelism: 5
  concurrencyPolicy: Allow

Option two :

Using cronjob you can apply the YAML template of daemon set and delete it after a certain duration which will work as job ideally on all nodes. Also if a custom docker image runs inside the deamon set it could also be complete once done with execution.

Extra:

You can use the CRD option checkout my Article which will work like Daemonjob : https://medium.com/google-cloud/daemon-job-crd-in-k8s-with-compositecontroller-f263b7f25cb9

Schedule job on each available node.

Read more about the CRD : https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/

You can write your own custom resource and add it to the kuberetes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.