Docs

    Kubernetes Utilization

    Kubernetes uses pod resource requests and limit to schedule pods on nodes that can handle the resource requirements. Blue Matador monitors actual pod resource usage to help you configure your pod resource requests.

    Resource Requests

    Pod containers can be configured to request a certain amount of CPU or memory. Requesting amounts that are close to the actual usage of the pod helps the Kubernetes scheduling system appropriately schedule pods onto nodes that can meet their resource requirements. It is also possible to not request any CPU or memory, in which case the pod is scheduled as a BestEffort pod that will be among the first to be evicted if a node becomes resource constrained.  

    It is recommended that you tune the resource requests of your pods to closely match usage so that you can accurately scale your cluster with your resource needs, and avoid unnecessary scheduling mistakes that could lead to poor performance in misconfigured pods. For example, a pod requesting only 200Mi of memory, but which gradually builds up memory usage to 1Gi of memory, could get scheduled on a node with only 500Mi, and could cause that node to run out of memory.  On the other hand, a pod requesting 1Gi of memory but only using 200Mi is effectively tying up 800Mi that could be used to schedule other pods.

     Requested CPU

    Blue Matador looks for pod CPU usage that is above the requested amount in the pod spec for any pod that is controlled by a ReplicaSet, DaemonSet, Deployment, StatefulSet, or other pod controller. The following criteria are used to determine if a pod’s requested CPU does not match actual usage:

    • Average CPU usage for pods in a controller is significantly higher than requested and
    • The total amount of CPU that is mis-allocated in the cluster is significantly large

    These two criteria ensure that you are notified of pod resource misconfigurations, but only if the amount is high enough relative to your cluster size. For example, if your entire cluster has capacity for 16,000m of CPU and you have 6 pods in a controller requesting 100m each but actually using 500m, you should reconfigure the pods to request closer to 500m so they can be properly scheduled. In the same scenario, if usage was only 150m, then no change is needed.  Also if in the same scenario you had a very large cluster with a capacity of 480,000m of CPU, then again no change is needed since the difference is likely insignificant in such a large cluster.

    Since CPU requests are often higher than usage to allow guaranteed CPU scheduling for lightweight pods, Blue Matador does not consider low CPU usage to be a configuration issue.

     

    CPU Limits

    Blue Matador monitors the actual CPU usage of pods compared to the CPU limit set in the pod spec. If a pod is within 10m of the limit for prolonged periods of time, Blue Matador will create a warning. Since CPU is not absolutely constrained to the set CPU limit on under-utilized nodes, it may be okay to go over the limit. If your pods are often running at the limit, however, they are in danger of being throttled when actual work is needed. To resolve this, look at increasing both the CPU request and limit for the pod so it can more consistently receive CPU time. For some applications such as Java applications, high CPU can be a side effect of high memory utilization, and the correct solution could be to increase the amount of memory the pod has access to.

     

    Resources