os.walk has a major gotcha

Python's os.walk is a very useful idiom for recursively iterating over a directory. However it has one big gotcha that bit us recently: by default, it silently no-ops if the directory doesn't exist:

However os.walk does support an optional onerror param, which takes a callable to handle any exceptions from the underlying walk:

Note that this callback will also be invoked for errors deep in the walk. These can happen, for example, if you modify the walk in-flight, as described in the docs, and erroneously add a non-existent directory.

In our codebase we plan to ban os.walk in favor of a wrapper function, safe_walk, that enforces the strict error handling above.  We'll use a custom Flake8 plugin to enforce this. Fortunately, our build system makes it easy to run Flake8, and many other Python linters and formatters.

Send us an email if you'd like to learn more about this!