Making Sense Golang Worker Pattern

Source: Dev.to

Spawn workers

intNumWorkers := 3

for w := 0; w < intNumWorkers; w++ {
    go InsertStudentData(ctx, tx, insertedStudentJobs, insertedStudentjobsErrCh)
}

func InsertStudentData(ctx context.Context, tx *sql.Tx, data <-chan models.StudentItem, errCh chan<- error) {
    // worker implementation
}

Three goroutine workers are started; each reads from a channel of StudentItem values and reports errors via errCh.

Publish task/job to workers

numJobs := len(StudentItems)
jobs := make(chan models.StudentItem, numJobs)

for i := 0; i < numJobs; i++ {
    jobs <- StudentItems[i]
}
close(jobs)

func InsertStudentWorker(ctx context.Context, tx *sql.Tx, data <-chan models.StudentItem, errCh chan<- error) {
    for d := range data {
        err := repository.InsertStudentItemFromWorker(ctx, tx, d)
        // handle err if needed
    }
}

The caller fills a buffered channel with all StudentItem jobs and then closes it. Workers pull jobs from this read‑only channel (data <-chan models.StudentItem) and process them concurrently.

Collecting results

numJobs := len(StudentItems)
jobs := make(chan models.StudentItem, numJobs)

for i := 0; i < numJobs; i++ {
    jobs <- StudentItems[i]
}
close(jobs)

// read from the error channel
for j := 0; j < numJobs; j++ {
    err := <-jobsErrCh
    if err != nil {
        _ = tx.Rollback()
        return err
    }
}

func InsertStudentWorker(ctx context.Context, tx *sql.Tx, data <-chan models.StudentItem, errCh chan<- error) {
    for d := range data {
        err := repository.InsertStudentItemFromWorker(ctx, tx, d)
        errCh <- err
    }
}

Each worker sends its result (or error) to a write‑only channel (errCh). Because the caller knows how many jobs were dispatched, it reads exactly that many results, handling any errors without needing to know which specific worker produced them.