Go Concurrency Patterns: Worker Pools, Fan-Out & Pipelines

You already know the slogan: “Don’t communicate by sharing memory; share memory by communicating.” But slogans don’t process 10,000 webhook payloads. Patterns do. Go’s concurrency toolkit — goroutines, channels, select, and context — is simple enough to learn in an afternoon and dangerous enough to misuse for years.

This guide covers the four Go concurrency patterns that actually show up in production backends: the worker pool, fan-out/fan-in, pipelines, and errgroup — each with working code you can steal, plus the mistakes that cause goroutine leaks and panics at 3am. Verified against current Go behaviour; code examples are complete programs, not fragments.

Table of Contents

Go concurrency basics in 60 seconds

A goroutine is a lightweight function execution managed by the Go runtime — you start one with the go keyword, and it costs a couple of kilobytes of stack instead of a megabyte OS thread. Channels are the typed pipes goroutines use to pass values to each other. An unbuffered channel blocks the sender until a receiver is ready, which makes the handoff itself the synchronisation.

package main

import "fmt"

func main() {
    results := make(chan string) // unbuffered: send blocks until received

    go func() {
        results <- "server-1 is healthy"
    }()

    fmt.Println(<-results) // blocks until the goroutine sends
}

The select statement lets one goroutine wait on several channel operations at once — including timeouts, which is how you stop waiting on something that never responds:

select {
case v := <-results:
    fmt.Println("got:", v)
case <-time.After(2 * time.Second):
    fmt.Println("timed out")
}

That’s the whole mental model. Everything below is built from these three pieces.

Pattern 1: The worker pool (bounded concurrency)

The worker pool is the single most useful Go concurrency pattern, and it exists for one reason: goroutines are cheap, but the things they touch aren’t. Spawning 100,000 goroutines to call an API 100,000 times will exhaust your file descriptors, hammer the database pool, and get you rate-limited into oblivion. A worker pool fixes the count: a fixed number of goroutines pull jobs from a shared channel, so concurrency stays bounded no matter how many jobs arrive.

Size the pool by workload: for CPU-bound work, use runtime.NumCPU(); for I/O-bound work (HTTP calls, DB queries), 10–100 workers is the typical range, tuned by latency and connection limits.

package main

import (
    "fmt"
    "sync"
)

func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
    defer wg.Done()
    for j := range jobs { // ranges until jobs is closed
        fmt.Printf("worker %d processing job %d\n", id, j)
        results <- j * 2 // send result back
    }
}

func main() {
    const numJobs = 9
    jobs := make(chan int, numJobs)
    results := make(chan int, numJobs)

    var wg sync.WaitGroup
    for w := 1; w <= 3; w++ { // exactly 3 workers
        wg.Add(1)
        go worker(w, jobs, results, &wg)
    }

    for j := 1; j <= numJobs; j++ {
        jobs <- j
    }
    close(jobs) // signal: no more jobs

    wg.Wait()      // wait for all workers to finish
    close(results) // now safe: no one will send again

    for r := range results {
        fmt.Println("result:", r)
    }
}

Three rules make worker pools safe: the sender closes the jobs channel exactly once; you close the results channel only after all workers are done (the WaitGroup guarantees this); and the results channel is buffered or consumed so workers never block forever.

Pattern 2: Fan-out / fan-in

Fan-out starts many goroutines on one stream of work; fan-in merges their results back into a single channel. This is the pattern for “call these 50 URLs and give me all the responses” — each call runs concurrently, and a single consumer reads the merged results.

The tricky part is knowing when to close the merged channel. The canonical answer: a dedicated goroutine waits on a sync.WaitGroup, then closes. Never close from the fan-out workers — exactly one goroutine should own each close.

package main

import (
    "fmt"
    "sync"
)

func main() {
    urls := []string{"https://a.example", "https://b.example", "https://c.example"}

    results := make(chan string)
    var wg sync.WaitGroup

    for _, u := range urls {
        wg.Add(1)
        go func(url string) { // fan-out: one goroutine per URL
            defer wg.Done()
            // imagine real HTTP work here
            results <- "status 200: " + url
        }(u)
    }

    go func() { // closer goroutine: owns the close
        wg.Wait()
        close(results)
    }()

    for r := range results { // fan-in: single consumer
        fmt.Println(r)
    }
}

One subtle bug this avoids: wg.Wait() cannot run in main before the range loop on an unbuffered channel — main would block waiting while nobody reads. The closer goroutine keeps the receive side alive.

Pattern 3: Pipelines

A pipeline chains stages together: each stage is a goroutine that reads from an input channel and writes to an output channel, and it owns — and closes — its own output when its input is drained. This models ETL flows, streaming transforms, and any “stage A, then stage B” workload where the stages can overlap.

package main

import "fmt"

func gen(nums ...int) <-chan int { // stage 1: generator
    out := make(chan int)
    go func() {
        defer close(out) // owner closes
        for _, n := range nums {
            out <- n
        }
    }()
    return out
}

func square(in <-chan int) <-chan int { // stage 2: transform
    out := make(chan int)
    go func() {
        defer close(out)
        for n := range in {
            out <- n * n
        }
    }()
    return out
}

func main() {
    for n := range square(gen(1, 2, 3, 4)) { // 1 4 9 16
        fmt.Println(n)
    }
}

The directional channel types (<-chan) are not decoration — they make the data flow a compile-time contract. A stage that only receives can never accidentally send. Combine a pipeline with fan-out between stages (multiple goroutines reading one stage’s output) and you get parallel processing with clean shutdown semantics, as long as every stage still closes its own output exactly once.

Pattern 4: errgroup — structured concurrency with errors

Raw WaitGroup + channels handle happy paths, but production code needs error collection and cancellation. errgroup, from the golang.org/x/sync module, is the standard answer: it runs a group of goroutines, returns the first error, and cancels a derived context so the rest of the group can stop early.

package main

import (
    "context"
    "fmt"
    "net/http"

    "golang.org/x/sync/errgroup"
)

func fetchStatus(ctx context.Context, url string) (int, error) {
    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
    if err != nil {
        return 0, err
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return 0, err
    }
    defer resp.Body.Close()
    return resp.StatusCode, nil
}

func main() {
    urls := []string{"https://go.dev", "https://example.com"}

    g, ctx := errgroup.WithContext(context.Background())
    g.SetLimit(10) // bounded concurrency: first error also cancels the rest

    for _, u := range urls {
        g.Go(func() error {
            code, err := fetchStatus(ctx, u)
            if err != nil {
                return err // g.Wait() returns this; ctx cancels
            }
            fmt.Println(u, "->", code)
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        fmt.Println("failed:", err)
    }
}

Install the module with go get golang.org/x/sync. Two details matter: SetLimit(10) caps how many goroutines run at once, which turns errgroup into a bounded worker pool with errors; and every Go function must actually respect the derived ctx — cancellation is cooperative, so an HTTP call built with http.NewRequestWithContext aborts when ctx is cancelled, but a function that ignores ctx will keep running anyway.

Go concurrency mistakes that will bite you

Every one of these shows up in real codebases. Check your own before the race detector does.

Goroutine leaks. A goroutine blocked forever on a channel send or receive is never garbage collected. Give every long-running goroutine an exit path — usually ctx.Done() in a select — and monitor goroutine counts in production.
Closing a channel you don’t own. Only the sending side closes a channel, exactly once. Sending on a closed channel panics; closing twice panics. If multiple writers exist, let a separate closer goroutine own the close (see the fan-in example).
Unbounded goroutines. go inside a loop over unknown-sized input is a resource leak wearing a trench coat. Use a worker pool, semaphore channel, or errgroup.SetLimit.
Fire-and-forget errors. A goroutine that swallows its error makes failures invisible. Propagate errors through channels or use errgroup.
Forgetting defer cancel(). Every context.WithTimeout / WithCancel must be followed by defer cancel(), or the runtime’s internal timer and goroutine leak until the timeout fires.
Sharing memory instead of communicating. Two goroutines writing to one variable without a lock or channel is a race. Run go test -race on everything — it exists precisely for this and catches what code review misses.

When NOT to use concurrency

Go makes concurrency easy, which tempts you to use it everywhere. Don’t. Sequential code is easier to read, test, and debug — and on a single fast path, a mutex-guarded counter is about an order of magnitude faster than a channel handoff. Reach for these patterns when you have genuine parallelism to exploit: I/O-bound fan-out (API calls, downloads), throughput-oriented processing (queues, streams), or responsiveness (background work that must not block a request). If the work is small, sequential, and fast, keep it boring.

The pattern cheat sheet

Worker pool — bounded concurrency over homogeneous jobs. Fixed goroutines, job channel, WaitGroup.
Fan-out / fan-in — divide work across goroutines, merge results into one channel. One closer goroutine owns the close.
Pipeline — chained stages, each owning and closing its output. Directional channels make flow a compile-time contract.
errgroup — fan-out with error collection and first-error cancellation. golang.org/x/sync/errgroup, always SetLimit.
Always — go test -race, defer cancel(), and an exit path for every goroutine.

Verified October 4, 2026. Code targets current Go behaviour (latest stable: Go 1.27) and compiles as written — run it with go run and go vet before adapting.

Go Concurrency Patterns: Worker Pools, Fan-Out/Fan-In, and Pipelines Explained