MicroK8s Self-Cordon: Automate Node Shutdowns

Introduction

Hey guys! Today, we're diving deep into an interesting challenge: automating the process of self-cordoning nodes in a MicroK8s homelab before they shut down or reboot. This is especially crucial for unattended upgrades where you want to ensure minimal disruption to your Kubernetes workloads. Imagine your cluster happily humming along, and then suddenly, a node decides to take a nap without telling anyone. Not cool, right? That’s where self-cordoning comes in – it's like telling Kubernetes, “Hey, I’m gonna be offline for a bit, so please don’t schedule anything new on me.”

The goal here is to make our MicroK8s nodes smart enough to cordon themselves before the shutdown or reboot actually happens. This ensures that no pods are stranded and that your applications remain highly available. We’ll explore how to achieve this using ExecStop in systemd service files, along with some scripting magic. We’ll tackle the common pitfalls, like scripts executing too quickly, and figure out the best way to reliably cordon our nodes. So, grab your favorite beverage, and let’s get started!

Understanding the Problem: ExecStop and Race Conditions

So, what's the deal with ExecStop and why isn’t it working as expected right out of the box? The main culprit is the timing. When you define an ExecStop command in a systemd service, you expect it to run before the service actually stops. However, in practice, the script might execute almost instantly, which can be too early for our needs. Think of it like trying to catch a train that’s already leaving the station – you’re there, but you’re not quite on board.

The issue is that the ExecStop command might run before the Kubernetes kubelet has a chance to react and cordon the node. This can lead to a race condition where the node shuts down before it’s properly cordoned, resulting in pods being abruptly terminated and potential service disruptions. We need to ensure that our script waits long enough for the kubelet to do its job. This is where our scripting finesse comes into play. We'll need to craft a script that not only initiates the cordon but also verifies that the cordon operation has been successfully completed before allowing the shutdown to proceed.

To make this work reliably, we’ll explore different strategies, such as adding delays, implementing retry mechanisms, and checking the node’s status in Kubernetes. The key is to make the script robust enough to handle various scenarios and ensure that the node is always cordoned before it goes offline. We don’t want any Kubernetes surprises!

Crafting the Self-Cordoning Script

Alright, let’s get our hands dirty and start crafting the script that will handle the self-cordoning magic. This script is the heart of our operation, so we need to make it robust and reliable. We'll use kubectl, the Kubernetes command-line tool, to interact with our cluster and cordon the node. But, as we discussed earlier, we can't just blindly run the kubectl cordon command and hope for the best. We need to add some extra sauce to ensure it works flawlessly.

First, we’ll add a delay to give the kubelet time to process the cordon request. Think of it as giving the kubelet a head start in the race. We can use the sleep command in our script to introduce this delay. However, a simple delay might not be enough in all situations. What if the Kubernetes API is temporarily unavailable, or the kubelet is under heavy load? That’s where a retry mechanism comes in handy. We’ll implement a loop that attempts to cordon the node multiple times, with a short delay between each attempt.

Inside the loop, we’ll use kubectl cordon to mark the node as unschedulable. We’ll also check the exit code of the kubectl command to see if it was successful. If it fails, we’ll wait a bit and try again. If it succeeds, we’ll move on to the next step: verifying that the node is indeed cordoned. To do this, we’ll use kubectl get node and check the node’s status. We’re looking for the SchedulingDisabled condition, which indicates that the node is cordoned. If we find it, we know we’ve successfully cordoned the node, and we can exit the script. If not, we’ll continue to retry until we reach a maximum number of attempts.

This script needs to be idempotent, meaning that running it multiple times should have the same effect as running it once. This is important because the ExecStop command might be executed more than once in some situations. Our script should handle this gracefully and avoid any unintended side effects.

Here’s a basic example of what the script might look like:

#!/bin/bash

NODE_NAME=$(hostname)
MAX_RETRIES=5
RETRY_DELAY=5

for i in $(seq 1 $MAX_RETRIES); do
  echo "Attempting to cordon node $NODE_NAME (attempt $i/$MAX_RETRIES)"
  kubectl cordon "$NODE_NAME" && break
  echo "Cordon failed, retrying in $RETRY_DELAY seconds..."
  sleep $RETRY_DELAY
done

if ! kubectl get node "$NODE_NAME" -o jsonpath='{.spec.unschedulable}' | grep -q 'true'; then
  echo "Failed to cordon node $NODE_NAME after $MAX_RETRIES attempts"
  exit 1
fi

echo "Node $NODE_NAME successfully cordoned"
exit 0

This is just a starting point, and you might need to adjust it based on your specific environment and requirements. For example, you might want to add logging, error handling, or more sophisticated retry logic. But the basic idea is the same: cordon the node, verify that it’s cordoned, and handle failures gracefully.

Integrating the Script with Systemd

Now that we have our self-cordoning script, the next step is to integrate it with systemd. Systemd is the system and service manager used by most modern Linux distributions, including the ones typically used for MicroK8s. We’ll create a systemd service file that will run our script when the node is shutting down or rebooting. This is where the ExecStop directive comes into play.

To create a systemd service, we need to create a file with a .service extension in the /etc/systemd/system/ directory. Let’s call our service self-cordon.service. Inside this file, we’ll define the service’s behavior, including the commands to run before and after the service stops.

The key part for us is the [Service] section, where we’ll use the ExecStop directive to specify our self-cordoning script. We’ll also use the RemainAfterExit=yes directive to ensure that the service stays in an active state even after the ExecStop command has finished. This is important because we want the script to run even if the main service (in this case, the kubelet) has already stopped.

Here’s an example of what the self-cordon.service file might look like:

[Unit]
Description=Self-cordon node before shutdown/reboot
Before=kubelet.service
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/path/to/your/self-cordon-script.sh

[Install]
WantedBy=multi-user.target

Let’s break this down:

• [Unit] section: This section contains metadata about the service. We’ve added a description and specified that this service should run before the kubelet.service. The DefaultDependencies=no directive tells systemd not to start this service automatically with other services.
• [Service] section: This is where we define the service’s behavior. Type=oneshot means that this service runs a single command and then exits. RemainAfterExit=yes ensures that the service stays active even after the command has finished. ExecStop specifies the path to our self-cordoning script.
• [Install] section: This section specifies how the service should be enabled and started. WantedBy=multi-user.target means that this service should be started when the system enters the multi-user target, which is the normal operating mode.

Once we’ve created the service file, we need to enable it and start it. We can do this using the systemctl command:

sudo systemctl enable self-cordon.service
sudo systemctl start self-cordon.service

Now, whenever the system shuts down or reboots, our self-cordon.service will run, executing our script and cordoning the node before the kubelet stops. This helps ensure a smoother shutdown process and minimizes disruption to our Kubernetes workloads.

Testing and Troubleshooting

Okay, we've got our script, we've integrated it with systemd – now comes the crucial part: testing! We need to make sure our self-cordoning mechanism actually works as expected. After all, the best code is code that's been thoroughly tested. Think of it like this: you wouldn't launch a rocket without testing its engines, right? Same goes for our self-cordoning system.

The easiest way to test this is to simply reboot your node. Before you do, keep a close eye on your Kubernetes cluster. You can use kubectl get nodes to monitor the status of your nodes. After the reboot, you should see the node marked as SchedulingDisabled, which means it has been successfully cordoned.

But what if things don't go as planned? What if the node doesn't cordon itself? That's where troubleshooting comes in. The first place to look is the systemd logs. You can use the journalctl command to view the logs for our self-cordon.service:

sudo journalctl -u self-cordon.service

This will show you any errors or warnings that occurred during the execution of the service. Pay close attention to the output of our script. Did it fail to cordon the node? Did it time out? The logs will give you valuable clues about what went wrong.

Another common issue is that the script might not have the necessary permissions to run kubectl. Make sure that the script is executable and that the user running the script (usually root) has the appropriate Kubernetes credentials. You might need to configure kubectl to use a specific kubeconfig file or service account.

If the script is failing intermittently, it could be due to network issues or temporary unavailability of the Kubernetes API. In this case, you might need to adjust the retry logic in your script. Increase the number of retries or the delay between retries. You could even add some logging to the script to track when and why it’s retrying.

Testing and troubleshooting are an iterative process. You might need to make several adjustments to your script and systemd service file before you get everything working perfectly. But don't get discouraged! With a little patience and perseverance, you'll have a robust self-cordoning system that will keep your Kubernetes workloads running smoothly.

Enhancements and Considerations

We've built a solid foundation for self-cordoning our MicroK8s nodes, but there's always room for improvement! Let's explore some enhancements and considerations to make our system even more robust and adaptable.

First, let's talk about error handling. Our current script has some basic error checking, but we can make it even more comprehensive. For example, we could add checks to ensure that kubectl is installed and configured correctly. We could also add more specific error messages to the logs, making it easier to diagnose issues. Think of it as adding extra sensors to our rocket to detect any potential problems before they become critical.

Another enhancement is to add support for uncordoning the node when it comes back online. Currently, our script only cordons the node, but it doesn't automatically uncordon it when the node is ready to accept new workloads. We could add another systemd service that runs on startup and uncordons the node. This would make the process fully automated and require less manual intervention.

Security is another important consideration. Our script currently runs as root, which is necessary to cordon the node. However, running scripts as root should be done with caution. We could explore ways to minimize the script's privileges, such as using capabilities or running the script as a non-root user with specific Kubernetes RBAC permissions. This would reduce the potential impact of a security vulnerability in the script.

We should also think about how our self-cordoning system interacts with other parts of our infrastructure. For example, if we're using a load balancer, we might want to ensure that the node is removed from the load balancer's pool before it's cordoned. This would prevent traffic from being sent to the node while it's shutting down. We could add commands to our script to interact with the load balancer's API.

Finally, let's consider the scalability of our system. If we have a large cluster with many nodes, running the self-cordoning script on each node could put a strain on the Kubernetes API. We might want to explore ways to optimize the script's performance or distribute the load across multiple nodes. For example, we could use a message queue or a distributed lock to coordinate the self-cordoning process.

By considering these enhancements and considerations, we can make our self-cordoning system even more robust, secure, and scalable. It's all about continuously improving our infrastructure to meet our evolving needs.

Conclusion

Alright guys, we've reached the end of our journey into automating MicroK8s node self-cordoning! We've covered a lot of ground, from understanding the challenges of using ExecStop to crafting a robust self-cordoning script and integrating it with systemd. We've also explored testing strategies, troubleshooting tips, and potential enhancements.

Self-cordoning is a crucial step in ensuring the high availability of our Kubernetes applications, especially in environments with unattended upgrades or other automated maintenance tasks. By automating this process, we can minimize disruptions and keep our clusters running smoothly. It’s like having a well-trained pit crew for our Kubernetes nodes, ensuring they’re always ready for the race.

We’ve learned that timing is everything when it comes to self-cordoning. We need to ensure that our script runs before the node shuts down, and that it waits long enough for the Kubernetes kubelet to react. We’ve also seen the importance of error handling and retry mechanisms in making our script resilient to failures. And remember, thorough testing is key to ensuring that our system works as expected.

But the journey doesn’t end here! Kubernetes is a constantly evolving ecosystem, and there are always new challenges and opportunities to explore. I encourage you to continue experimenting, learning, and sharing your knowledge with the community. Whether it’s optimizing your self-cordoning script, exploring new Kubernetes features, or contributing to open-source projects, there’s always something more to discover.

So, go forth and automate your MicroK8s nodes! May your cordons be swift, your shutdowns be smooth, and your Kubernetes clusters always be highly available. Thanks for joining me on this adventure, and I’ll catch you in the next one!