Kubernetes is a powerful container orchestration platform that allows users to deploy and manage containerized applications efficiently. However, the health of the nodes in a Kubernetes cluster is crucial for the overall stability and reliability of the applications running on it. Node problems, such as hardware failures, kernel issues, or container runtime problems, can impact the availability of pods and disrupt the entire cluster.
To address this, Kubernetes offers a tool called node-problem-detector, which aims to detect and report various node problems to the cluster management stack. In this blog, we will explore node-problem-detector, its features, how to deploy it in a Kubernetes cluster, and real-world use-cases with code snippets.
Node problems in a Kubernetes cluster can lead to application disruptions and impact user experience. Issues like hardware failures, kernel panics, or unresponsive container runtimes are challenging to detect early and remediate. The node-problem-detector tool aims to address this problem by making various node problems visible to the upstream layers in the cluster management stack.
node-problem-detector uses two mechanisms to report problems to the Kubernetes API server: Event and NodeCondition. Permanent problems that make the node unavailable for pods are reported as NodeConditions, while temporary problems that have limited impact on pods but are informative are reported as Events.
node-problem-detector consists of multiple problem daemons, each responsible for monitoring specific kinds of node problems. The supported problem daemon types include System Log Monitor, System Stats Monitor, Custom Plugin Monitor, and Health Checker.
The System Log Monitor is a crucial component of node-problem-detector that monitors system logs and reports problems and metrics according to predefined rules. It collects log data from various sources, including kernel logs, system logs, and container runtime logs.
The System Stats Monitor collects various health-related system stats as metrics to provide insights into the node’s health status. Although it is not fully supported yet, it’s a promising feature for future releases.
The Custom Plugin Monitor allows users to define and check various node problems using custom check scripts. This flexibility enables users to address node problems specific to their use-cases.
The Health Checker verifies the health of essential components in the node, such as the kubelet and container runtime. It ensures these components are functioning correctly and reports any issues detected.
The Exporter is responsible for reporting node problems and metrics to certain backends. Supported exporters include the Kubernetes exporter, Prometheus exporter, and Stackdriver exporter.
helm repo add deliveryhero https://charts.deliveryhero.io/ helm install --generate-name deliveryhero/node-problem-detector
- Edit node-problem-detector.yaml to fit your environment. Set log volume to your system log directory (used by SystemLogMonitor). You can use a ConfigMap to overwrite the config directory inside the pod.
- Edit node-problem-detector-config.yaml to configure node-problem-detector.
- Edit rbac.yaml to fit your environment.
Create the ServiceAccount and ClusterRoleBinding with:
kubectl create -f rbac.yaml
- Create the ConfigMap with:
kubectl create -f node-problem-detector-config.yaml
- Create the DaemonSet with:
kubectl create -f node-problem-detector.yaml
- Apply required manifests
kubectl create -f node-problem-detector-config.yaml kubectl create -f rbac.yaml kubectl create -f node-problem-detector.yaml
node-problem-detector --hostname-override=my-node --enable-k8s-exporter
System Stats Monitor is still under development, but it will allow you to collect various health-related system stats as metrics.
Node-problem-detector is a valuable tool for monitoring node health in Kubernetes clusters. By making node problems visible to the cluster management stack, it enables administrators to detect and address issues before they impact applications. In this blog, we explored the features of node-problem-detector, how to deploy it, and real-world use-cases. Armed with this knowledge, you can enhance the reliability and stability of your Kubernetes clusters and ensure seamless application deployment.