How to Debug Kubernetes OOMKilled Pods
Determine why Kubernetes containers are OOMKilled by comparing memory limits, actual usage, node pressure, application behavior, and restart history.
OOMKilled means the Linux kernel terminated a container after it exceeded an enforced memory boundary or the node encountered severe memory pressure. Raising the limit may hide the symptom; first determine whether usage is expected, leaking, mis-sized, or caused by workload concurrency.
Confirm the termination reason
kubectl describe pod POD -n NAMESPACE
kubectl get pod POD -n NAMESPACE -o jsonpath='{range .status.containerStatuses[*]}{.name}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{" restarts="}{.restartCount}{"\n"}{end}'
OOMKilled commonly appears with exit code 137, but exit code alone does not prove the cause. Use the recorded termination reason and events.
Compare requests, limits, and usage
kubectl get pod POD -n NAMESPACE -o jsonpath='{.spec.containers[*].resources}'
kubectl top pod POD -n NAMESPACE --containers
kubectl top is a recent sample, not the historical peak that caused termination. Use your monitoring system to inspect working-set memory, RSS, cache, and usage leading up to the restart.
Check node memory pressure
kubectl get pod POD -n NAMESPACE -o wide
kubectl describe node NODE
kubectl top node NODE
Look for MemoryPressure, evictions, and other workloads competing on the node. A container limit OOM and a node-wide OOM require different fixes.
Inspect the workload pattern
Read the previous container logs:
kubectl logs POD -n NAMESPACE -c CONTAINER --previous --tail=200
Correlate memory growth with traffic, batch size, concurrency, cache population, large requests, runtime heap settings, and recent deployments. Ensure runtime limits account for native allocations and sidecars, not only the managed-language heap.
Diagnose it with Rumus
Rumus is an AI-native terminal that can use your current Kubernetes context to correlate pods, events, logs, controllers, resources, and nodes. It keeps the investigation read-only until you approve a proposed change.
What Rumus can inspect
- Pods, events, previous logs, and owning controllers
- Resources, probes, scheduling rules, and node state
- Secret-aware output and approval before cluster changes
Diagnose why this Kubernetes container was OOMKilled. Inspect its previous state,
restart history, requests and limits, recent and historical memory evidence,
previous logs, owning controller, node pressure, and rollout history. Redact
secrets and do not restart or edit resources until I approve the plan.
Confirm the fix
Apply the application or resource change to the owning controller and watch the rollout:
kubectl rollout status deployment/NAME -n NAMESPACE
kubectl get pods -n NAMESPACE -w
Monitor memory through the workload’s peak period. The fix is successful only if restarts stop without moving pressure to the node or degrading application behavior.
Open the server in Rumus and let the AI agent inspect context, propose a plan, and ask before it runs changes.