Asynchronous programming came to python awhile back, formally encapsulated in the asyncio module. This is a python library that allows developers to utilise an asynchronous pattern in writing code using the async-await pattern. While full-featured and mature, I personally encountered a confusing edge case when using create_task to spin off background tasks.
Let’s start off with a working example:
The above code runs some asynchronous task code, and initiates an event loop that polls an api periodically. We save the asynchronous task to a variable. This task runs in the background; we save it because maybe we want to use it later.
All looks good.
Silent Failure
But what if the background task fails:
The above code is exactly the same, except now we raise an exception in the standalone background task.
The polling continues to work, but (and?) strangely the exception does not raise. The application has just failed silently. Frustrating.
Why does this happen?
Interestingly, asyncio actually does register that an error has occurred. But the error is buffered in the coroutine, as explained in the documentation.
The “silent” failure occurs specifically when
- We save the awaitable task to a variable and never do anything with it.
- There is code that runs after the coroutine is created that doesn’t terminate the enclosing function. This code is the while loop in our example.
In other words, the error is registered in the coroutine, but because we don’t do anything with the coroutine, or don’t terminate the enclosing function, the error stays in the coroutine and we never get to see it.
Which makes it seem like the application code is failing silently.
Very frustrating.
Solutions
Fortunately for us, once we understand what the problem is, the solutions are clear — although none of them perfect.
Here are some suggested approaches.
Solution 1: await asyncio.create_task directly
Interestingly, the exception gets raised explicitly when we await asyncio.create_task directly. This is because we do do something with the awaitable task.
This approach can be useful if we are willing to sacrifice some parallel processing for determinism. Maybe the background task is small enough to not have to worry about processing time?
Solution 2: Await task eventually
If we await the saved task eventually, the exception is also raised explicitly.
Same as above, but we delay non-parallel processing a little — or a lot depending on where you await the task eventually.
Solution 3: Use side-effects
Instead of relying on a failure at the level of the enclosing function, we can also catch the error in the function and handle it using side-effects.
This approach might not work if we do want the error to propagate. For example in an application where an error in the background task should force a restart of the application.
Solution 4: Live with it, expecting the exception to throw eventually
This do-nothing approach only works without the presence of an endless while loop like in our original example.
Of course, if the fact that a failure not being caught early enough will be a problem, this solution is out of the picture.
Hope this post helped save you a lot of head-wrangling!