AKS Cluster Autoscaler Bug With Scale-down Mode: Explained

Introduction

Hey guys! Today, we're diving deep into a tricky bug that can pop up when you're using the Cluster Autoscaler with scale-down mode deallocate in Azure Kubernetes Service (AKS). This issue can cause some headaches, so let's break it down, figure out what's going on, and explore some potential solutions. If you're leveraging AKS and autoscaling, this is a must-read to ensure your cluster operates smoothly and efficiently. This article will guide you through the intricacies of the bug, its impact, and how to address it effectively. Understanding the nuances of this interaction between Cluster Autoscaler and scale-down mode deallocate is crucial for maintaining a robust and scalable Kubernetes environment. Let's get started and make sure your AKS cluster stays in top shape!

Describe the Bug

The core of the issue lies in how the Cluster Autoscaler interacts with nodes that have been deallocated when using scale-down mode deallocate. When you're running your AKS cluster with scale-down mode deallocate, nodes that are scaled down aren't completely removed from the cluster. Instead, they're deallocated, which means they remain in the cluster in a Not Ready state. This is where the problem begins. The Cluster Autoscaler has built-in settings, specifically ok-total-unready-count and max-total-unready-percentage, which are designed to prevent it from scaling down too aggressively when nodes are temporarily unavailable. By default, these settings are configured to stop scaling a node pool if there are 3 unready nodes, and to halt completely if 45% of the nodes are in a Not Ready state.

Now, when nodes are deallocated, they're marked as Not Ready, but they aren't actually failed nodes. They're simply in a deallocated state, waiting to be brought back online if needed. The Cluster Autoscaler, however, doesn't differentiate between these deallocated nodes and genuinely failed nodes. As a result, if you scale down your cluster using deallocate mode, you can quickly hit the threshold set by ok-total-unready-count or max-total-unready-percentage. Once the number of deallocated nodes in a node pool reaches the limit (e.g., 3 nodes), the autoscaler will stop considering that node pool for scaling. If the overall percentage of Not Ready nodes in the entire cluster hits 45%, the autoscaler will halt completely, preventing any further scaling operations. This can lead to a situation where your cluster can't scale up to meet demand, even though you have resources available, because the autoscaler is incorrectly interpreting deallocated nodes as a critical issue. This misinterpretation can severely impact the responsiveness and efficiency of your applications running on AKS. It's crucial to address this conflict to ensure your cluster can adapt dynamically to changing workloads.

To Reproduce

Okay, let’s talk about how to actually see this bug in action. If you want to reproduce this issue in your own AKS environment, here’s what you need to do. First, you'll need a cluster with multiple node pools. This is important because the issue becomes more apparent when you have several pools and scale-down operations can affect a larger number of nodes. Make sure you have scale-down mode deallocate enabled on your cluster. This is the key setting that triggers the behavior we’re investigating. Once your cluster is set up, scale your nodes to a relatively high number – say, around 10 nodes across your node pools. This will give you enough headroom to perform scale-down operations and observe the effects.

Next, perform a scale-down operation to reduce the number of active nodes significantly. For example, you could scale down to just 1 node. This will result in the remaining 9 nodes being deallocated, which means they'll enter the Not Ready state. Now, here’s where the bug manifests. Try scheduling some workload on your cluster that requires more resources than your single active node can provide. You would expect the Cluster Autoscaler to kick in and scale up the node pool to accommodate the new workload. However, because of the deallocated nodes, the autoscaler is likely to disregard the node pool. You might see logs indicating that the autoscaler has found the deallocated instances and is setting the target size to 0, effectively preventing any scaling. For instance, you might see a log message like: Found: 9 instances in deallocated state, returning target size: 0 for scaleSet scaleSetName. This message confirms that the autoscaler is recognizing the deallocated nodes but is not treating them appropriately, leading to a standstill in scaling operations. By following these steps, you can reliably reproduce the bug and gain a firsthand understanding of its impact on your cluster's ability to scale. This hands-on experience is invaluable for troubleshooting and implementing effective solutions.

Expected Behavior

Ideally, the Cluster Autoscaler and scale-down mode deallocate should play nicely together. We want our cluster to be able to scale down nodes to save costs when they're not needed, but we also want the autoscaler to be able to scale up quickly when demand increases. The current behavior, where deallocated nodes effectively cripple the autoscaler, isn't cutting it. So, what would the right behavior look like? There are a few ways we could approach this, and each has its own trade-offs. Let's explore some potential solutions.

One straightforward solution would be to inform users about the interaction between scale-down mode deallocate and the autoscaler settings. This means making it clear that when using deallocate mode, the ok-total-unready-count and max-total-unready-percentage settings need to be adjusted or carefully considered. While this is the easiest solution to implement from a technical perspective, it essentially shifts the burden to the user. It also more or less disables the autoscaler settings, which exist for a reason – to prevent over-scaling in the face of temporary node issues. So, while it's a quick fix, it might not be the most user-friendly or robust in the long run. A better approach might be to change how deallocated nodes are handled within the cluster. Instead of marking deallocated nodes as Not Ready, we could remove them from the cluster entirely. This would prevent them from interfering with the autoscaler's calculations, as only genuinely failed nodes would contribute to the Not Ready count. However, this approach has implications for how quickly deallocated nodes can be brought back online, as they would need to be fully reprovisioned. A more nuanced solution would be to make the autoscaler aware of the difference between failed nodes and deallocated nodes. The autoscaler could then be configured to ignore deallocated nodes when calculating the Not Ready count, only taking action if nodes have genuinely failed. This would allow the autoscaler to continue functioning correctly even with deallocated nodes present in the cluster. This approach offers the best of both worlds, as it preserves the functionality of both scale-down mode deallocate and the autoscaler's safety mechanisms. Ultimately, the goal is to have a system where scaling is intelligent and responsive, without being hampered by the specific state of deallocated nodes. Each of these solutions aims to achieve that, but they differ in complexity and impact on the overall cluster behavior.

Potential Solutions

To make this situation better, there are a few paths we can take. Each has its own pros and cons, so let's break them down:

Inform the User

The simplest solution is to educate users about this behavior. We could provide documentation or warnings that explain how scale-down mode deallocate interacts with the autoscaler settings (ok-total-unready-count and max-total-unready-percentage). Users would then need to adjust these settings accordingly to avoid the issue. Pros: Easy to implement. Cons: Puts the onus on the user to understand and manage these settings, effectively disabling the autoscaler's built-in safeguards against over-scaling.

Remove Deallocated Nodes

Another option is to change how deallocated nodes are handled. Instead of leaving them in the cluster in a Not Ready state, we could remove them entirely. This would prevent them from affecting the autoscaler's calculations. Pros: Prevents deallocated nodes from interfering with the autoscaler. Cons: Nodes would need to be fully reprovisioned when scaled up, which can take longer than bringing a deallocated node back online.

Make Autoscaler Aware of Deallocated Nodes

The most sophisticated solution is to make the autoscaler smarter. We could modify it to recognize the difference between failed nodes and deallocated nodes. The autoscaler would then ignore deallocated nodes when calculating the Not Ready count, only taking action if nodes have truly failed. Pros: Allows both scale-down mode deallocate and the autoscaler's safety mechanisms to function correctly. Cons: Requires more complex implementation.

Environment

This bug has been observed in Kubernetes version 1.31.10. It's important to note the Kubernetes version because the behavior of the Cluster Autoscaler and the scale-down mode deallocate can vary between versions. Knowing the specific version helps in accurately diagnosing and addressing the issue. Kubernetes is constantly evolving, with new features, bug fixes, and changes to existing functionalities being introduced regularly. Therefore, an issue that exists in one version might be resolved in a later version, or the workaround might differ. Providing the Kubernetes version is a critical step in reporting bugs and seeking assistance, as it ensures that the information and solutions provided are relevant to your specific environment. This detail allows developers and support teams to replicate the issue, identify the root cause, and implement the appropriate fix. Furthermore, it helps other users experiencing similar problems to determine if they are encountering the same bug and to apply the correct solutions or workarounds. Always include your Kubernetes version when discussing or reporting issues related to cluster behavior, as it forms a crucial part of the context needed for effective troubleshooting and resolution.

Conclusion

So, there you have it, guys! We've walked through a tricky bug in AKS that can cause the Cluster Autoscaler to clash with scale-down mode deallocate. Understanding this issue is crucial for maintaining a healthy and responsive Kubernetes cluster. By being aware of the potential problems and the available solutions, you can ensure that your cluster scales efficiently and effectively, even when using deallocate mode. Whether you choose to inform your users, modify how deallocated nodes are handled, or enhance the autoscaler's awareness, the key is to take proactive steps to address this interaction. Remember, a well-managed cluster is a happy cluster, and a happy cluster means happy applications and users! Keep exploring, keep learning, and keep your clusters scaling smoothly!