Skip to content
Kien

PM2 cluster mode inside a Kubernetes pod

· 4 min read

2 schedulers · 1 workload

Two schedulers end up managing the same workload and neither knows the other exists. The concurrency nobody chose, a memory limit that belongs to the cgroup, and a crash loop hiding inside a pod that reports itself healthy — and why the fix is to pick one layer and let it do the multiplying.

  • Kubernetes
  • Node.js
  • PM2
  • operations
On this page
  1. Where this comes from
  2. Two multipliers on one axis
  3. The limit belongs to the cgroup; the heap does not know that
  4. The signal the forks flatten
  5. A crash loop inside a healthy pod
  6. Pick one layer to do the multiplying
  7. What I took away

Where this comes from

A Node service that ran for years on a VM under PM2 gets containerised and moved onto Kubernetes. The Dockerfile keeps the line that already worked: start it under PM2, cluster mode, a worker per core. The chart gets what every chart gets — a horizontal autoscaler on the deployment. Neither half is written wrong. Each half is a correct answer to the question the other half is also answering, and the question is how many copies of this process should be running.

Two multipliers on one axis

PM2 in cluster mode forks workers inside the container. The autoscaler scales pod replicas. Effective concurrency is the product of the two, and that product is a number nobody chose. Whoever set the fork count was sizing a machine. Whoever set the replica range was sizing a service. Nobody was sizing the multiplication. Change either side later and real concurrency moves by a factor, while the person changing it believes they are moving it by a step.

The limit belongs to the cgroup; the heap does not know that

The container's memory limit applies to the whole cgroup — every worker together. Each worker, being its own process, sizes its own heap as if it owned the machine. A per-worker ceiling that looks comfortable in isolation, multiplied by the fork count, sits above the pod limit. When it goes over, the kernel takes the container: the pod is OOM-killed rather than the runtime throwing a heap error, so the failure arrives in the platform's vocabulary instead of Node's.

The signal the forks flatten

The autoscaler scales on pod-level metrics, averaged across everything inside the pod. One worker saturated while its siblings idle averages into something that reads as moderate load. Uneven work across forks is the normal case — one long request, one heavy job — so the average is quietest exactly when one worker is in trouble. The forks smooth the signal the autoscaler exists to read.

A crash loop inside a healthy pod

This is the part that cost the setup my trust. PM2 restarts a dead worker in place. The container never exits, so the pod stays Ready and the restart count never moves. A service can be crash-looping continuously while every place a Kubernetes operator looks reports it healthy.

That is not a PM2 bug. It is PM2 doing its job in a place that already has something doing that job. The container contract is a single process, narrow on purpose: signals, exit codes and probe responses are the whole vocabulary the platform has for learning that something is wrong. A supervisor in the middle absorbs all three. SIGTERM lands on the supervisor. The exit code is the supervisor's. The liveness probe is answered by whichever worker happened to take the connection, which says nothing about the others.

Pick one layer to do the multiplying

Inside Kubernetes, that layer is Kubernetes. One process per container. Replicas are the concurrency knob and the only one. Let the process die and the pod die with it, because a pod that dies is one the platform can act on — restart it, reschedule it, alert on it. Everything PM2 provides here, the orchestrator already provides, in a form the rest of the system can see.

Two details survive the move. Readiness has to reflect the thing that actually serves traffic, not a supervisor's opinion of it. And a memory ceiling has to be derived from the cgroup limit divided by the number of processes sharing it, never from the machine's memory — a number the container does not own.

None of this retires PM2; it relocates it. On a plain VM, a small internal service, a box where nothing underneath schedules anything, PM2's process management is the platform, and a good one. The conflict is not with PM2. It is with running two schedulers and telling neither about the other.

One thing I did not do: I never measured the cost of the double layer. No benchmark of forks-in-a-pod against one process per container, nothing I could publish. The case here is structural — about which component is allowed to know things — and I have no figure for the overhead.

What I took away

Two systems that both believe they own scaling do not add up. They cancel out. Each is reasoning about a workload the other is quietly reshaping, and the layer underneath loses, because it is the one that would have to be told and never is.

The symptom is never a line that says scaling is wrong. It is a pod reporting Ready, a dashboard reporting moderate load, a restart count that has not moved since the deploy, and requests queueing behind all of it.