Nick Logvynenko

Retry Pattern is a popular pattern to achieve Resilience in Distributed Systems. But people use it wrong all the time. Here are 4 things to keep in mind when using the Retry Pattern correctly. The Retry Pattern is a design approach used to enhance the reliability and resilience of software systems by automatically reattempting a failed operation or request. 1. Set a Reasonable Retry Limit: Determining the right number of retries is critical. Too few retries might not allow temporary issues to be resolved, while too many retries could lead to excessive load or long delays in recognizing a persistent problem. Consider your operation and the expected duration of transient failures to decide on an appropriate retry limit. I never go over 3 retries. 2. Implement Exponential Backoff: Instead of retrying immediately, implement an exponential backoff strategy. This means that after each failed attempt, you increase the time before the next retry. This prevents overwhelming the system and gives it time to recover. Exponential backoff helps avoid a stampeding herd effect where all failed requests suddenly hit the system simultaneously after a short time. 3. Identify Retriable Errors: Not all errors are worth retrying. Focus on retrying only transient errors: - 408 Request Timeout - 5XX (Server did something bad) Avoid responses like: - 400 (Bad Request) - 403 (Forbidden) They are not recoverable, so retry logic shouldn’t retry them. 4. Combine with Circuit Breaker: The Retry Pattern works well when combined with a circuit breaker mechanism. A circuit breaker monitors the health of a service and prevents repeated calls to a failing service, which could worsen the situation. If a certain threshold of failures is reached, the circuit breaker opens, temporarily preventing further requests. This gives the service time to recover before attempting retries again. Remember that the goal is to gracefully handle transient failures and maintain the overall stability of your application.