Toolchain is a build-and-test acceleration system for orchestrating fast, stable workflows on your existing CI infrastructure.

Toolchain uses smart analysis to understand the fine-grained structure and dependencies of your codebase and all changes made to it. It applies this data to find the smallest amount of work necessary to keep your builds up to date. Toolchain then consults a shared cache to avoid repeated effort and leverages distributed execution to dramatically speed up the remaining work. Doing significantly less work means your CI workflows are faster and less expensive.

Pants OSS is a state-of-the-art build system that runs on CI machines and developer laptops is the open-source core of the Toolchain technology stack. Pants has many performance and consistency benefits as a standalone tool, but it really shines for growing teams and codebases when used as the client to the Toolchain system.

Toolchain’s remote caching and execution system can speed up your CI workflows by as much as 95%. Because CI systems don’t understand your code, they must do a huge amount of repeated work on every run. Their limited caching and concurrency capabilities must be manually configured and are often ineffective. However, Toolchain uses dependency and change analysis to apply automatic, precise fine-grained invalidation, caching, and concurrency to speed up your workflows and use fewer resources.

Toolchain saves you money and time by simplifying your CI configuration and using significantly less CI resources. Toolchain doesn’t replace your existing CI provider but instead acts as a layer between your CI infrastructure—CircleCI, GitHub Actions, BuildKite, Jenkins, and other underlying tools for building, testing, and packaging your code. When you configure your CI workflows to run simple Pants commands, they will in turn send work requests to the Toolchain system.

CI systems don’t understand your code. So caching and concurrency, if used at all, are coarse-grained and must be manually configured. Typically, you must cache and restore huge directories against a manually constructed cache key. This process is prone to error and more time-consuming than simply redoing the cached work. Concurrent jobs must be manually partitioned, which is also error-prone, and each job requires repeated, heavyweight bootstrapping such as cloning the repo and installing tools. As a result, CI typically involves 2-3 huge, expensive cache lookups and expensive concurrent jobs.

Toolchain, on the other hand, understands your code’s structure and dependencies. It can automatically perform thousands of precise, fine-grained cache lookups and run hundreds of distributed processes quickly and correctly, with no laborious manual configuration. With the power of code analysis, Toolchain makes your builds faster and more accurate.

Toolchain has advanced support and fine-grained code analysis for several languages and frameworks, including Python, Go, Java, Scala, Kotlin, Shell, Docker, Terraform, and Helm (with more on the way). These languages and frameworks gain the greatest benefit from Toolchain. However Toolchain also has generic workflow mechanisms that can be used with any language, tool, or custom script. The system can be customized via a plugin API to add advanced support for those cases.

Toolchain was originally designed with the intricacies of the Python ecosystem in mind. Therefore, it has particularly strong support for Python codebases for uses like data science, web applications, and platform engineering. Toolchain provides very robust solutions for scaling Python codebases that involve multiple deployables and shared libraries.