What you'll learn

  1. Quick answer
  2. The problem the scheduler is solving
  3. The two-phase decision: filtering, then scoring
  4. What actually happens during filtering
  5. What actually happens during scoring
  6. The scheduling framework: where custom logic plugs in
  7. A worked example
  8. Debugging a pod stuck in Pending
  9. FAQ

Quick Answer

kube-scheduler assigns each pod to a node in two phases: filtering, which throws out every node that cannot legally run the pod (not enough CPU, an untolerated taint, a failed affinity rule), and scoring, which ranks whatever nodes survive and picks the highest-ranked one. It never does a global optimization across all pods at once — each pod is scheduled independently, one at a time.

The problem the scheduler is solving

Every pod you create starts with spec.nodeName empty. Something has to fill that field in, and that something is kube-scheduler, a control-plane component that watches the API server for pods in this state and assigns each one to a specific node. Once it writes the node name back to the pod object, its job for that pod is done — the kubelet on that node takes over and actually starts the containers.

The reason this needs a dedicated component, instead of "just picking any node with room," is that a real cluster has competing constraints that interact: a pod might request 2 CPU cores and 4GB of memory, need to avoid a node that's tainted for GPU-only workloads, prefer to land in the same availability zone as its database, and want to avoid being packed onto the same node as three other replicas of itself. None of these constraints are checked in isolation — the scheduler has to satisfy the hard ones and optimize for the soft ones, per pod, continuously, as pods and nodes come and go.

This sandbox doesn't have kubectl, minikube, or kind installed, so nothing in this post is a screenshot of a real cluster. Every mechanical claim below — the two-phase split, the specific plugin names, the extension points — is checked directly against the current official Kubernetes documentation (kubernetes.io/docs/concepts/scheduling-eviction/) rather than recited from memory, and quoted where it matters.

The two-phase decision: filtering, then scoring

Kubernetes' own documentation describes scheduling as a two-step operation. Step one is filtering: "the filtering step finds the set of Nodes where it's feasible to schedule the Pod." A node that fails even one hard requirement — say, it doesn't have the requested CPU available — is dropped from consideration entirely. The nodes that make it through are called, in the documentation's own terminology, feasible nodes. If that list comes back empty, the pod isn't schedulable yet and stays in Pending, however many nodes the cluster has.

Step two is scoring: "the scheduler ranks the remaining nodes to choose the most suitable Pod placement. The scheduler assigns a score to each Node that survived filtering, basing this score on the active scoring rules." Notice what scoring does not do — it never reconsiders a node that filtering already rejected. A node that's a perfect fit by every scoring metric is irrelevant if it failed a single filter. This ordering is the single most important thing to internalize: hard constraints are a wall, soft preferences are a ranking, and the wall always goes first.

Finally: "kube-scheduler assigns the Pod to the Node with the highest ranking." If several nodes tie for the top score, the scheduler picks among them at random, which is why identical-looking pods can land on different nodes across separate scheduling runs.

What actually happens during filtering

Filtering runs a set of plugins against every node, and any one of them can veto a node outright. The ones you'll run into constantly:

  • Resource fit — checks whether a node has enough allocatable CPU and memory left to satisfy the pod's requests (not its limits, and critically, not its actual live usage). A node can be sitting at 90% idle CPU and still get rejected if its declared, requested allocation is already full — the scheduler works off the accounting ledger of requests, not real-time metrics.
  • Taints and tolerations — a taint applied to a node repels pods unless the pod explicitly tolerates it. This is how you reserve a pool of GPU nodes for GPU workloads, or keep general workloads off a node that's being drained for maintenance. A toleration doesn't force a pod onto a tainted node, it only removes the automatic rejection — the pod still has to win on other criteria.
  • Node affinity and nodeSelector — nodeSelector is a simple exact-match on node labels. Node affinity is the same idea with more expressiveness: requiredDuringSchedulingIgnoredDuringExecution behaves like a filter (fail it and the node is out), while preferredDuringSchedulingIgnoredDuringExecution is actually a scoring signal, not a filter at all — the "preferred" naming is the tell.

All of this happens per-node and, per the documentation, nodes can be evaluated concurrently — filtering doesn't have to be a slow serial scan of the whole cluster.

What actually happens during scoring

Every node that survives filtering gets run through a separate set of scoring plugins, each of which returns a numeric score in a fixed range. Common scoring signals include how tightly the pod's resource request would pack the node (do you want to fill nodes up, or spread load thin across many?), whether the pod has a preferred affinity match, and topology spread constraints that push replicas of the same workload across different zones or nodes so one failure domain doesn't take out all of them.

The scheduler combines the individual plugin scores using configured weights, then — after a normalization step — sums them into one final number per node. Whichever feasible node ends up with the highest combined score wins. This is why two pods with identical specs can land differently depending on what else is already running: scoring is relative to current cluster state, recomputed fresh for every pod.

The scheduling framework: where custom logic plugs in

Filter and Score aren't the only two hooks. The current Kubernetes documentation describes a full pipeline of named extension points that a scheduling cycle runs through, in order: PreEnqueue, QueueSort, PreFilter, Filter, PostFilter, PreScore, Score, a normalization step, Reserve, Permit, PreBind, Bind, and PostBind.

PreFilter and PreScore let a plugin precompute or cache state once per cycle instead of once per node. PostFilter runs only when filtering leaves zero feasible nodes — this is where pod preemption logic lives, deciding whether evicting a lower-priority pod elsewhere would free up room. Reserve and Permit exist because scheduling a pod and actually binding it aren't atomic; something can still go wrong between "we picked a node" and "the API server confirmed it," and these hooks let plugins hold or release provisional claims on resources during that window. This is also the plumbing that projects like Kubernetes' own descheduler and custom schedulers (for batch workloads, GPU-aware scheduling, and similar) build on, rather than forking the scheduler outright.

A worked example

Say you deploy a pod requesting 2 CPU and 4GB memory, with a toleration for a workload=batch:NoSchedule taint, onto a 4-node cluster:

  • Node A — 8 CPU / 16GB free, no taints. Passes filtering.
  • Node B — only 1 CPU free (already heavily requested by other pods). Fails the resource filter, dropped, never reaches scoring — regardless of how empty it looks in a CPU-usage dashboard.
  • Node C — 6 CPU / 8GB free, tainted workload=batch:NoSchedule. The pod tolerates that exact taint, so it passes filtering.
  • Node D — 4 CPU / 32GB free, no taints. Passes filtering.

Filtering leaves A, C, and D as feasible. Now scoring runs only on those three. If the active scoring plugin favors spreading load thin (leaving more headroom), Node D's large free memory pool likely wins it the highest score even though it has the least free CPU of the three — scoring plugins weigh whatever metric they're configured to weigh, and different clusters can be tuned to prefer packing tightly (bin-packing, to reduce the number of nodes you pay for) or spreading out (to reduce blast radius). Node B never got a chance to compete at all; that's the filtering wall in action.

Debugging a pod stuck in Pending

A pod stays Pending for exactly one structural reason: filtering returned zero feasible nodes, or a feasible node exists but binding hasn't completed yet. kubectl describe pod <name> surfaces this directly in its Events section with a FailedScheduling reason and a human-readable summary like "0/6 nodes are available: 3 Insufficient cpu, 2 node(s) had taint {workload: batch}, that the pod didn't tolerate, 1 node(s) didn't match Pod's node affinity." Reading that line literally, per node, is the fastest way to find the exact constraint that's rejecting every candidate — it's usually one of the three filters covered above, not something exotic.

The other case worth knowing: if the pod defines a priorityClassName, an empty feasible-node list can trigger the PostFilter/preemption path instead of a flat rejection, where the scheduler looks for a lower-priority pod it could evict elsewhere to make room. If that succeeds you'll see the victim pod get terminated moments before your pod schedules — which looks alarming in logs if you don't know to expect it.

Frequently Asked Questions

Does the scheduler look at how busy a node actually is, or just what's been requested? Just what's been requested. Filtering compares a pod's resource requests against a node's allocatable capacity minus the sum of requests already committed on it — real-time CPU or memory usage isn't part of that calculation, which is why a node can look idle in monitoring and still be full from the scheduler's point of view.
What's the practical difference between required and preferred node affinity? Required affinity (requiredDuringSchedulingIgnoredDuringExecution) behaves like a filter: fail it and the node is removed from consideration entirely. Preferred affinity (preferredDuringSchedulingIgnoredDuringExecution) is a scoring input, not a filter — a node that doesn't match is still eligible, it just scores lower.
If two nodes end up with the exact same score, what breaks the tie? The scheduler picks randomly among the tied top-scoring nodes. This is also why two structurally identical pods, scheduled moments apart with an unchanged cluster state, can still land on different nodes.
Can I write and plug in my own scheduling logic without replacing kube-scheduler? Yes — the scheduling framework exposes named extension points (PreFilter, Filter, PreScore, Score, Reserve, Permit, and others) specifically so custom plugins can hook into the existing pipeline instead of forking or reimplementing the whole scheduler.
Does a toleration guarantee a pod will be scheduled onto the tainted node? No. A toleration only cancels the automatic rejection that the taint would otherwise cause during filtering. The pod still has to be evaluated normally against every other filter and still competes in scoring against any other feasible nodes.
Why did a lower-priority pod on another node suddenly get killed right before my pod started running? That's preemption. When filtering finds zero feasible nodes for a pod that defines a priorityClassName, the scheduler's PostFilter step can evict a lower-priority pod elsewhere to free up room, then schedule your pod into the space that frees up.