DevPill 10 - Fault tolerance: adding retries to your Go code

Source: Dev.to

Adding retries to your API

Adding retries to your API is a must to make your system more resilient. You can add them in database operations, communication with external APIs, and any other operation that might depend on a response from a third party.

1. Retry function

The function receives the number of attempts, a delay (interval between attempts), and the function that represents the operation you want to perform.

func Retry(attempts int, delay time.Duration, fn func() error) error {
    err := fn()
    if err == nil {
        return nil
    }

    if !IsTransientError(err) {
        // log.Println("--- retry not needed")
        return err
    }

    if attempts--; attempts > 0 {
        time.Sleep(delay)
        // log.Println("--- trying again...")
        return Retry(attempts, delay*2, fn) // exponential backoff
    }

    return err
}

2. IsTransientError function

This helper checks whether an error is transient and worth retrying. For example, sql.ErrNoRows indicates no rows were found and should not be retried.

func IsTransientError(err error) bool {
    if err == nil {
        return false
    }

    // example: deadlock postgres
    if strings.Contains(err.Error(), "deadlock detected") {
        return true
    }

    // closed connection
    if strings.Contains(err.Error(), "connection reset by peer") {
        return true
    }

    // network/postgres timeout
    if strings.Contains(err.Error(), "timeout") {
        return true
    }

    // max connections exceeded
    if strings.Contains(err.Error(), "too many connections") {
        return true
    }

    return false
}

3. Using the function

Here’s an example of retrying a database operation in a login method within the service layer:

func (s *UserService) Login(ctx context.Context, email, password string) (*UserOutput, error) {
    var user *domain.User
    err := retry.Retry(3, 200*time.Millisecond, func() error {
        var repoError error
        user, repoError = s.repo.GetUserByEmail(ctx, email)
        if repoError != nil {
            if repoError == sql.ErrNoRows {
                return ErrUserNotFound
            }
            return repoError
        }
        return nil
    })

    if err != nil {
        return nil, err
    }
    // ... further processing ...
    return &UserOutput{/* fields */}, nil
}