A job in Kubernetes is responsible for creating and managing Pods and perform tasks until its successful termination where as normal Pods restarts continuously regardless of the exit code. If a job fails before the successful termination, the job controller will create a new pod and depending on the nature of the system there’s a chance that our system might end up with duplicated pods.
Let’s consider a one-shot job that fails to successfully terminate. If we check the status of of our pod, we would notice that our pod has been restarted multiple times and Kubernetes will end up in CrashLoopBackOff status.
kubectl get pod -a -l job-name=demojob
———————————————-
NAME READY STATUS RESTARTS AGE
demojob-3ddk0 0/1 CrashLoopBackOff 4 3m
If we change the restartPolicy from OnFailure to Never, we could avoid this kind of CrashLoops. For example,
kubectl get pod -a -l job-name=demojob
———————————————-
NAME READY STATUS RESTARTS AGE
demojob-0wm49 0/1 Error 0 1m
demojob-6h9s2 0/1 Error 0 39s
demojob-hkzw0 1/1 Running 0 6s
demojob-k5swz 0/1 Error 0 28s
demojob-m1rdw 0/1 Error 0 19s
demojob-x157b 0/1 Error 0 57s
What we are seeing here is a set of duplicate pods that are in Error state. With restart policy set to Never we inform kubelet to not to restart pods on failure. But the job object notices a job in Error state and crates a new pod for that in this situation and we will be ended up with multiple duplicate pods. Since it is uncommon to have a pod failure on start up, we could configure the restart policy to OnFailure and avoid duplicate pods.
Last week I had the pleasure of reading “Chaos Engineering; Building Confidence in System Behavior through Experiments“. A book written by few engineers involved in Chaos Engineering at Netflix. Being a software engineer who is involved in a similar set of work in enterprise software context, I would say that I am much thankful for the authors of this book for the content that helped me on thinking of formulating a strategy to ensure the reliability of our product.
In chaos theory, the butterfly effect is the sensitive dependence on initial conditions in which a small change in one state of a deterministic nonlinear system can result in large differences in a later state.
In this blog post I will summarize a simple process of adapting Chaos Engineering.
Under the light of Agile development methodologies, we are following different practices to ensure that our applications are doing what it is suppose to do. This could start at unit level testing with a Test Driven Development framework or could be scale up to component level integration testing. What we are doing in all these cases is testing what we know about our application or what we are expecting from our application. Our support teams have to battle with a different set of problems once we deploy our applications to production.
Chaos engineering is a discipline that allow us to get an understanding of our systems behavior in production environment by simulating faults in it. “Principles of Chaos Engineering” define it as,
Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.
Here are some steps involve in designing experiments,
Create a hypothesis
Define the scope of experiments
Identify the metrics
Notify the organization
Run the experiments
Analyze the results
Increase the scrope
Automate
Create a hypothesis
Failures can happen due to various reasons. For example, hardware failures, functional bugs, network latency or communication barriers, inconsistent state transmissions, etc. What is important at this stage is to select an impactful event that can change the system. Lets say that we have observed traffic of one region of our APIs is increasing, we could test out our load balancing functionality.
Define the scope of the experiments
It is great if we can do experiments on our hypothesis in production, but at first we could choose a less impactful environment and gradually move towards production as confidence on our experiments grows over time.
Identify the metrics
Once the hypothesis and scope is defined we could decided what metrics we are going to use to evaluate our outcome. Equal distribution of traffic across multiple servers or time taken to reach a response to client can be used in load balancing scenario.
Notify organization
It is necessary to keep all stakeholders informed about the experiments and taking their input on how the experiments should designed in order to get maximum insights.
Run the experiments
Lights, Camera, Action! Now we can run the experiments, but at this point it is necessary to keep an on metrics. If the experiments are causing harm to system it is necessary to abort experiments and a mechanism for that should be placed in.
Analyze the result
Once the results are available we could validate the correctness of hypothesis and communicate the results with relevant teams. If the problem is with load balance, maybe the network infrastructure team have to work a bit more on load balancing across the system.
Increase the scope
Once we grow our confidence on experimenting on smaller scale problems we could start extending the scope of experiments. Increasing the scope can reveal a different set of systemic problems. For example, failures in load balancing can cause time outs and inconsistent states in different services that could cause our system to fall apart in peak times.
Automate
Don’t repeat it yourselves as you gain confidence on your experiments. Start automating what you have already experimented and look forward for other areas to build confidence.
Finally, a problem that comes to mind naturally is how good the decision of shutting down or playing around with your system to take it down in production? Well, Chaos Engineering is certainly not playing around with your system. It is based on the same empirical process that uses to test new drugs, therefore whatever the work we are doing in here is for the betterment of our own products.
