Retry idiom for Python

A relatively little known feature of Python is the else block for control flow statements other than if.

If you haven’t heard about it before, you can provide such a block for both while and for loops, as well as any variant of the try statement, Its functionality is roughly analogous in both cases:

• in loops, the else block is executed if the loop didn’t exit abnormally (i.e. with break)
• in try constructs, the else block runs if no exception happened

Likely because of the unique semantics that don’t exist in other languages, neither of those constructs has been seen much in real world code. Recently, however, I’ve found they can be combined into a very pythonic pattern that’s also quite useful.

The trick

Say you have a task that won’t always succeed. Perhaps it’s a request made to a janky server, or other network operartion that’s prone to timeouts. Since failures are likely to be transient, you’d like to retry it several more times before giving up permanently.

With try/except block, you can detect those half-expected failures. With a simple loop, you can repeat the attempt for as many times as you deem feasible. Combined, they can solve the problem rather neatly:

for _ in range(MAX_RETRIES):
    try:
        # ... do stuff ...
    except SomeTransientError:
        # ... log it, sleep, etc. ...
        continue
    else:
        break
else:
    raise PermanentError()

But why?

What’s the deal with the elses here, though? Are they both necessary?

The simple answer is of course no. else after either a loop or try/except block is always a syntactic sugar. Any code that contains it can be transformed into an equivalent snippet that utilizes different techniques to achieve the same effect.

But this view isn’t very useful, for many of the essential features in any programming languages can be dismissed as superfluous using this reasoning. The real question is whether the above idiom is more readable and understandable than the alternatives.

To that, I posit, the answer is: absolutely.

Desugaring

Without the double else, this example would have to be written in a considerably more convoluted way:

retries = MAX_RETRIES
while retries > 0:
    try:
        # ... do stuff ...
        break
    except SomeTransientError:
        # ... log it, sleep, etc. ...
        retries -= 1
if retries == 0:
    raise PermanentError()

Although at first glance the difference may be minuscule, this version adds significant extra busywork the programmer has to pay careful attention to:

• The retries variable now has to be explicit, because the final conditional statement must look at its value.
• We can’t use a for loop anymore (e.g. for retries in range(MAX_RETRIES)), because we wouldn’t distinguish the “success at last try” and “retry limit exceeded” cases: they’d both result in retries equal to MAX_RETRIES - 1 after the loop¹.
• As a result, we have to remember to decrement the counter ourselves upon an error.

Additionally, the break is easy to miss amidst the actual logic within the try block, both for developer who writes the code and for any subsequent readers. An alternative is to move it outside of the try/except clause, but that in turn reintroduces continue into the except branch and further complicates the whole flow.

In short, the desugared version is more error-prone (all those off-by-ones!) and also quite inscrutable.