HTTP Reconnaissance

A recon client, built right

Everything starts with an HTTP client you control — custom headers, a real timeout, and your own redirect policy. The build is fenced (it needs the network) but it’s the foundation of every web tool:

client := &http.Client{
	Timeout: 8 * time.Second, // NEVER use the zero-timeout default
	CheckRedirect: func(r *http.Request, via []*http.Request) error {
		return http.ErrUseLastResponse // see redirects, don't auto-follow
	},
}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
req.Header.Set("User-Agent", "recon/1.0 (authorized testing)")
resp, err := client.Do(req)

The two non-negotiables: a timeout (the default client has none — one hung host stalls a goroutine forever) and an honest User-Agent for authorized work.

Fingerprinting from the response

The headers and status code reveal the stack:

graph TD
R["HTTP response"] --> H["Server: nginx/1.18"]
R --> P["X-Powered-By: PHP/7.4"]
R --> C["Set-Cookie: PHPSESSID"]
R --> CDN["CF-RAY / X-Amz-Cf-Id"]
H --> MAP["map to known CVEs"]
P --> MAP
C --> MAP
CDN --> WAF["CDN / WAF in front?"]

See it: parse and fingerprint a response

You can analyze a raw HTTP response with http.ReadResponse — no network needed — which is exactly how this runs in the sandbox. The same parsing logic works on a real resp from client.Do:

package main

import (
"bufio"
"fmt"
"net/http"
"sort"
"strings"
)

// A captured raw HTTP response (as if read from the wire).
const raw = "HTTP/1.1 200 OK\r\n" +
"Server: nginx/1.18.0\r\n" +
"X-Powered-By: PHP/7.4.3\r\n" +
"Set-Cookie: PHPSESSID=abc; HttpOnly\r\n" +
"Content-Type: text/html\r\n" +
"Content-Length: 0\r\n" +
"\r\n"

func main() {
resp, err := http.ReadResponse(bufio.NewReader(strings.NewReader(raw)), nil)
if err != nil {
	fmt.Println("parse error:", err)
	return
}
defer resp.Body.Close()

fmt.Println("status:", resp.StatusCode)

// Simple tech fingerprint from headers.
var tech []string
if s := resp.Header.Get("Server"); s != "" {
	tech = append(tech, "web server: "+s)
}
if p := resp.Header.Get("X-Powered-By"); p != "" {
	tech = append(tech, "language: "+p)
}
for _, c := range resp.Cookies() {
	if c.Name == "PHPSESSID" {
		tech = append(tech, "session: PHP")
	}
}
sort.Strings(tech)
fmt.Println("fingerprint:")
for _, t := range tech {
	fmt.Println("  -", t)
}
}

That nginx/1.18.0 + PHP/7.4.3 fingerprint is the first thing an attacker maps to a CVE list — and the first thing a defender should make less informative.

Content and path discovery

The interesting paths aren’t linked — /admin, /.git/config, /backup.zip, /api/v1/docs. Path discovery requests each candidate from a wordlist (the same worker pool) and flags non-boring responses. The one subtlety is soft-404 calibration:

// Calibrate: fetch a path that definitely doesn't exist, record its
// status + body length as the "not found" fingerprint.
notFound := fetch(base + "/zzq-" + random())
// A real hit differs from that baseline (different status OR length).
if resp.StatusCode != notFound.status || resp.length != notFound.length {
	report(path, resp.StatusCode)
}

Without calibration, a site that serves a friendly 200 page for missing routes makes every guess look like a hit. Always baseline first, then throttle (don’t hammer the target), and honor robots.txt as a hint about interesting paths — it often lists exactly what the owner wanted hidden.

Defending your web service

See also

• DNS enumeration — finding the hosts whose web services you then profile.
• Fuzzing for bugs — pushing past recon into finding actual flaws.
• Input validation & injection defense — the bugs recon is looking for.
• Web security — the security headers and hardening referenced above.

Next: from reading what a service says to finding where it breaks — fuzzing for bugs.

Related topics