Scheduler Internals

This page assumes the concurrency scheduler page, which introduces the G-M-P model and GOMAXPROCS. Here we go beneath it: the monitor thread, preemption, the netpoller, and stealing.

G, M, P — the quick recap

• G — a goroutine: its stack, instruction pointer, and scheduling state.
• M — a “machine,” i.e. an OS thread. Only an M can execute code.
• P — a logical processor: a scheduling context holding a local run queue and the mcache. An M must hold a P to run Gs. There are GOMAXPROCS Ps, which bounds parallelism.

graph TD
subgraph P0["P (local run queue)"]
  G1["G"] --- G2["G"] --- G3["G"]
end
subgraph P1["P (local run queue)"]
  G4["G"] --- G5["G"]
end
M0["M (OS thread)"] --> P0
M1["M (OS thread)"] --> P1
GQ["global run queue"] -.->|"refilled from / stolen to"| P0
GQ -.-> P1
SM["sysmon (no P)"] -.->|"preempt · retake P · netpoll"| M0

Handoff: why a blocking syscall doesn’t stall everything

The P/M split exists for one big reason. When a goroutine makes a blocking syscall, its M parks in the kernel and can’t run anything. If work lived on the M, the whole local queue would freeze. Instead the work lives on the P: the runtime detaches the P from the blocked M and hands it to another M (waking a parked one or spawning a new one), which immediately resumes the queued goroutines. When the syscall returns, the original M tries to reacquire a P; if none is free, its goroutine goes to the global queue and the M parks.

This handoff is why a handful of blocking calls don’t tank throughput — but it’s also why a program that spawns thousands of simultaneously blocking syscalls can create thousands of Ms.

Work-stealing with handoff

There’s no central dispatcher assigning goroutines to Ps. Instead, when a P’s local run queue empties, its M looks for work in order:

1. its own local queue (empty),
2. the global run queue,
3. the netpoller (goroutines whose I/O just became ready),
4. steal ~half of a randomly chosen other P’s local queue.

Stealing half (not one) means a freshly-stolen P quickly has work to share again, spreading load in a few hops. New goroutines go on the creator’s local queue (fast, no lock); overflow spills to the global queue.

sysmon and preemption

A normal scheduling decision happens at a safe point — typically a function call. But some things can’t wait for the running goroutine to be polite:

• a goroutine in a tight loop with no calls would never yield;
• an M stuck in a long syscall holds resources;
• the network needs polling and the GC needs triggering on time.

So Go runs sysmon, a special M with no P that loops forever (backing off when idle). sysmon:

• retakes Ps from Ms blocked in syscalls for too long,
• marks long-running goroutines for preemption,
• polls the network (the netpoller integrates epoll/kqueue/IOCP so a blocked socket parks its goroutine instead of its thread),
• triggers GC and memory scavenging on timers.

Since Go 1.14, preemption is asynchronous: sysmon sends a signal (SIGURG on Unix) to a thread running a hog goroutine; the runtime handler stops it at a safe point. Before 1.14, a for {} with no calls could pin a P forever and even stall the GC — now it can’t.

Observing the scheduler

You can’t watch the run queues from pure Go, but you can see the moving parts — GOMAXPROCS, thread and goroutine counts — and the effect of yielding:

package main

import (
"fmt"
"runtime"
"sync"
)

func main() {
fmt.Println("GOMAXPROCS (P count):", runtime.GOMAXPROCS(0))
fmt.Println("NumCPU:", runtime.NumCPU())
fmt.Println("goroutines at start:", runtime.NumGoroutine())

var wg sync.WaitGroup
for i := 0; i < 1000; i++ {
	wg.Add(1)
	go func(n int) {
		defer wg.Done()
		// A little CPU work; the scheduler multiplexes 1000 Gs onto GOMAXPROCS Ps.
		sum := 0
		for j := 0; j < 1000; j++ {
			sum += j
		}
		if n == 0 {
			fmt.Println("peak goroutines (incl. this one):", runtime.NumGoroutine())
		}
		runtime.Gosched() // voluntarily yield this G back to the scheduler
	}(i)
}
wg.Wait()
fmt.Println("goroutines after wait:", runtime.NumGoroutine())
}

runtime.Gosched() is the explicit “yield the P now” call. You rarely need it — the scheduler preempts for you — but it makes the cooperative side of scheduling concrete.

To watch the scheduler itself, use the schedtrace debug knob (build-time/runtime env, not in-playground):

# Print scheduler state every 1000ms: run queue sizes, idle Ps/Ms, etc.
GODEBUG=schedtrace=1000 ./myprogram
# SCHED 1003ms: gomaxprocs=8 idleprocs=6 threads=12 spinningthreads=1 runqueue=0 ...

# Add scheddetail=1 for per-P/per-M breakdowns.
GODEBUG=schedtrace=1000,scheddetail=1 ./myprogram

Reference

See also

• the scheduler (concurrency) — the G-M-P model and GOMAXPROCS at the user level.
• runtime introspection — schedtrace, NumGoroutine, the execution tracer.
• goroutines (concurrency) — what’s being scheduled, and leak avoidance.
• the garbage collector — why timely preemption matters for GC.

Next: observing all of this from inside your program — runtime introspection.

Related topics