The Go Scheduler

The M:N model

Most languages map one concurrency unit to one OS thread, so 100,000 of them means 100,000 threads — gigabytes of stack and a thrashing kernel scheduler. Go instead multiplexes many goroutines onto few threads — an M:N scheduler — with three actors:

• G — goroutine. Your concurrent function, its stack, and its scheduling state. Cheap: a g struct plus a ~2 KB growable stack. You can have millions.
• M — machine. An OS thread, the only thing that actually executes instructions. Expensive: each is a real kernel thread.
• P — processor. A scheduling context — the permission-to-run-Go-code token. A P owns a local run queue of runnable goroutines and a slice of caches (the mcache for allocation). To run Go code, an M must hold a P.

The key invariant: you have exactly GOMAXPROCS Ps, so at most GOMAXPROCS goroutines run Go code simultaneously. That is the cap on true parallelism. Threads (Ms) can outnumber Ps — extras exist to absorb blocking syscalls — but only GOMAXPROCS of them hold a P and run Go at any instant.

graph TD
subgraph P1["P0 (local run queue)"]
  G1["G"] --- G2["G"] --- G3["G"]
end
subgraph P2["P1 (local run queue)"]
  G4["G"] --- G5["G"]
end
GQ["global run queue"]
M1["M0 (OS thread)"] --> P1
M2["M1 (OS thread)"] --> P2
P2 -. "steals half" .-> P1
GQ -. "drained when local empties" .-> P2

Think of it as three layers: G is the work, P is a seat with a to-do list, M is a worker who must sit in a seat to do anything. The scheduler’s whole job is keeping a goroutine on every seat.

Run queues: local, global, and runnext

Runnable goroutines live in three places, checked in a deliberate order:

When a P needs work it looks in this order: runnext → local queue → global queue (occasionally, to avoid starving it) → the netpoller → steal from another P. The local queue being lock-free for its owner is why scheduling is cheap: in the common case a P just pops its own ring buffer with no atomics on the fast path. New goroutines from go f() go onto the current P’s runnext/local queue; if the local queue is full, a batch overflows to the global queue so one busy producer can’t monopolize a single P.

Work-stealing keeps cores busy

When a P’s local queue and runnext are empty, it does not go idle. It runs the findrunnable loop: peek at the global queue, poll the netpoller, then steal from a peer. Stealing picks a random victim P and takes half of its local queue.

graph LR
IDLE["P1 local queue empty"] --> GLOBAL{"global queue<br/>has work?"}
GLOBAL -->|yes| TAKE["take a batch"]
GLOBAL -->|no| POLL{"netpoller<br/>ready Gs?"}
POLL -->|yes| RUNNET["run I/O-ready Gs"]
POLL -->|no| STEAL["steal half of a<br/>random P's queue"]
STEAL --> RUN["run stolen Gs"]

Why half and not one? Taking half means the victim isn’t drained on the very next steal, and the stealer gets enough work to stay busy for a while — load balances in O(log P) hops instead of one-at-a-time ping-pong. Why random victims? It avoids every idle P piling onto the same unlucky queue. The result is balanced load with no central scheduler on the hot path — the thing that lets Go scale to many cores. If findrunnable finds nothing anywhere, the P is parked on an idle list and its M may sleep, to be woken when work appears.

Blocking syscalls: handing off the P

A goroutine doing os.Read, a CGO call, or any blocking syscall must block its M in the kernel — there’s no avoiding that. The trick is that the P does not block with it.

sequenceDiagram
participant G as G (syscall)
participant M0 as M0
participant P as P
participant M1 as M1 (idle/new)
G->>M0: enter blocking syscall
M0->>P: detach P (handoff)
P->>M1: M1 acquires P
Note over M1,P: P's other goroutines keep running
G-->>M0: syscall returns
M0->>M0: try to reacquire a P
Note over M0: if none free, park G on a queue and M0 sleeps

On entering a blocking syscall (entersyscall), the runtime marks the P as detachable. The background sysmon thread (or the M itself, on the fast path via entersyscallblock) hands the P to an idle M — or spins up a new one — so the P’s remaining goroutines keep executing. When the syscall returns (exitsyscall), the original M tries to grab a P back; if none is free, it parks its goroutine on the global queue and goes to sleep in the M pool. This is why one slow read() never freezes your whole program — and also why a flood of blocking calls can spawn many Ms (see the gotcha). Non-blocking syscalls that return instantly use a cheaper path that doesn’t hand off the P.

The netpoller: parking on I/O

Network I/O is special. If every blocked socket read tied up an M, a server with 10,000 idle connections would need 10,000 threads. Instead, Go’s network, timer, and (on some platforms) file primitives are built on the network poller — a thin layer over epoll (Linux), kqueue (BSD/macOS), or IOCP (Windows).

When a goroutine reads from a socket with no data ready, the runtime parks the goroutine (removes it from any run queue, no M held) and registers the fd with the poller. The M is now free to run other goroutines. A dedicated check of the poller — inside findrunnable and periodically by sysmon — asks the kernel “which fds are ready?” and un-parks the corresponding goroutines back onto a run queue.

graph TD
G["G: conn.Read() — no data"] --> PARK["park G, register fd with epoll/kqueue"]
PARK --> FREE["M is free to run other Gs"]
POLLER["netpoller checks kernel"] --> READY{"fd ready?"}
READY -->|yes| WAKE["mark G runnable → run queue"]
READY -->|no| POLLER

The payoff: blocking-style code, non-blocking performance. You write the obvious n, err := conn.Read(buf) and the runtime quietly turns it into an event-loop registration — no callbacks, no async/await. Tens of thousands of idle connections cost a handful of threads, not one each. This is the engine under every Go HTTP server.

Preemption: cooperative, then asynchronous

A goroutine must eventually yield its P so others get a turn. Go uses two mechanisms:

• Cooperative preemption (always). At function-call prologues the compiler inserts a tiny stack-growth/preempt check. When sysmon flags a goroutine as running too long (~10 ms), that check trips on the next call and the goroutine yields. Cheap, but it only fires at a call.
• Asynchronous preemption (Go 1.14+). A goroutine in a tight loop with no calls — for { x++ } — never hits a prologue, so before 1.14 it could hog a P indefinitely, starving every other goroutine on that P and stalling the GC (which needs all goroutines to reach a safe point). Go 1.14 fixed this: sysmon sends the M a signal (SIGURG); the signal handler stops the goroutine at a register-safe instruction, parks it, and frees the P.

The runtime also preempts for STW events: when the GC needs every goroutine at a safe point, async preemption is what guarantees even a calls-free loop stops promptly — which is why GC pauses stay sub-millisecond.

GOMAXPROCS and the container default

GOMAXPROCS sets the number of Ps. Read it with runtime.GOMAXPROCS(0) (the 0 means “report, don’t change”); set it with the env var GOMAXPROCS or the same call with a positive argument.

GOMAXPROCS=4 ./server          # pin to 4 Ps regardless of core count
GODEBUG=schedtrace=1000 ./server   # log scheduler state (G/M/P counts) every 1s
GODEBUG=scheddetail=1,schedtrace=1000 ./server  # add per-P detail

// Read (never blindly hard-code) the current value; setting returns the old one.
n := runtime.GOMAXPROCS(0) // report only
old := runtime.GOMAXPROCS(2) // set to 2, returns previous value
_ = old
_ = n

See the runtime for yourself

This prints stable facts about the scheduler. A WaitGroup barrier makes the goroutine count deterministic: we hold all 1000 workers at a gate, observe the live count, then release them.

package main

import (
"fmt"
"runtime"
"sync"
)

func main() {
fmt.Println("NumCPU:       ", runtime.NumCPU() >= 1) // true on any machine
fmt.Println("GOMAXPROCS>=1:", runtime.GOMAXPROCS(0) >= 1)
fmt.Println("goroutines at start:", runtime.NumGoroutine()) // 1 (main)

const N = 1000
var start sync.WaitGroup // gate: workers wait here
var done sync.WaitGroup  // barrier: main waits for finish
start.Add(1)

for i := 0; i < N; i++ {
	done.Add(1)
	go func() {
		defer done.Done()
		start.Wait() // park until main opens the gate
	}()
}

// All N workers are alive and parked on start.Wait(), plus main = N+1.
fmt.Println("goroutines with workers parked:", runtime.NumGoroutine() == N+1)

start.Done() // open the gate — every worker can now finish
done.Wait()
fmt.Println("all workers finished:", runtime.NumGoroutine() == 1) // back to just main
}

The output is deterministic across machines because we print booleans and counts that don’t depend on core count — NumGoroutine() == N+1 is exact while the workers are parked, since none have returned yet.

When this matters in practice

• Don’t fear goroutine count. 100,000 goroutines is normal; the scheduler is built for it. The limit is rarely the scheduler — it’s blocking work spawning Ms, or memory.
• Bound blocking work. Each blocking syscall can park an M; thousands at once means thousands of threads. Cap concurrency with a worker pool or semaphore.
• Never spin-wait. Park on a channel/context so the goroutine yields its P instead of burning a core (the gotcha above).
• Set GOMAXPROCS from your cgroup on pre-1.25 runtimes. Otherwise containers over-subscribe.
• CPU-bound stages scale to GOMAXPROCS, not beyond. More goroutines than Ps on pure CPU work just adds scheduling overhead — see concurrency vs parallelism.

Next: why those goroutine stacks are so cheap and the pauses so short — the garbage collector & stacks.

Related topics