Metrics with Prometheus

Pull-based metrics

Prometheus is an open-source metrics database and monitoring system (the de-facto standard in cloud-native, usually paired with Grafana for dashboards); see prometheus.io. It is pull-based: your app maintains counters in memory and exposes them on a /metrics HTTP endpoint in a simple text format; Prometheus scrapes that endpoint on an interval and stores the time series.

graph LR
APP["Go app<br/>counters in memory<br/>+ /metrics endpoint"] -->|scrape every 15s| PROM["Prometheus<br/>(stores time series)"]
PROM --> GRAF["Grafana dashboards"]
PROM --> ALERT["Alertmanager<br/>(rate/errors/latency)"]

Three core metric types:

• Counter — monotonically increasing (requests served, errors). Query rate() for per-second rates.
• Gauge — goes up and down (in-flight requests, queue depth, memory).
• Histogram — buckets observations (latency, sizes) so you can compute p95/p99 quantiles — the numbers that describe real user experience.

See it: the /metrics exposition format

This runs here: a tiny in-process registry that counts requests by status and renders the Prometheus text format, served via httptest. The real prometheus/client_golang does this richly, but the shape is exactly this:

package main

import (
"fmt"
"net/http"
"net/http/httptest"
"sort"
"sync"
)

type counter struct {
mu sync.Mutex
v  map[string]int // label "status" -> count
}

func (c *counter) inc(status string) {
c.mu.Lock()
defer c.mu.Unlock()
c.v[status]++
}

func (c *counter) render() string {
c.mu.Lock()
defer c.mu.Unlock()
out := "# TYPE http_requests_total counter\n"
keys := make([]string, 0, len(c.v))
for k := range c.v {
	keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
	out += fmt.Sprintf("http_requests_total{status=%q} %d\n", k, c.v[k])
}
return out
}

func main() {
reqs := &counter{v: map[string]int{}}
// Simulate traffic.
for i := 0; i < 100; i++ {
	if i%10 == 0 {
		reqs.inc("500")
	} else {
		reqs.inc("200")
	}
}

// Expose /metrics, then scrape it like Prometheus would.
h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	fmt.Fprint(w, reqs.render())
})
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
fmt.Print(rec.Body.String())
}

That http_requests_total{status="..."} output is exactly what Prometheus scrapes; rate(http_requests_total{status="500"}[5m]) then gives your error rate. In real code you’d use the client library:

// Fenced: github.com/prometheus/client_golang
var reqs = promauto.NewCounterVec(prometheus.CounterOpts{
	Name: "http_requests_total", Help: "Total HTTP requests.",
}, []string{"method", "route", "status"}) // LOW-cardinality labels only

http.Handle("/metrics", promhttp.Handler())
// in your middleware: reqs.WithLabelValues(r.Method, route, status).Inc()

Instrument with RED

For request-driven services, track RED: Rate (requests/sec), Errors (failures/sec), Duration (latency histogram) — per endpoint. Those three answer “is the service healthy?” and are what you alert on. (For resources — CPU, memory, disk — the complementary USE method tracks Utilization, Saturation, Errors.)

See also

• Structured logging — events, not aggregates.
• Distributed tracing — following one request across services.
• Profiling in production — when metrics say “slow” and you need to know why.
• Benchmarks (stdlib) — measuring before you ship.

Next: following a single request across service boundaries — distributed tracing.

Related topics