CI

Our test suite consists of a common part that tests basic SDK functionality, and an integrations part that tests individual integrations.

Since the number of integrations to test is so large, the tests in CI are split into multiple groups (roughly by integration type) so that they can run in parallel. So there is, for example, a GitHub workflow that runs all our feature flag integrations, and another one that runs the test suites of all GraphQL integrations, and so on. These workflow YAMLs are auto-generated by this script.

The ultimate source of truth of what should be tested and on which versions (both Python versions as well as package versions) is stored in tox.ini. This file is also auto-generated each week by a GitHub cron job that scans PyPI for new releases and updates the test matrix with them. The script that does this lives in scripts/populate_tox and is often referred to as "toxgen".

The Big Picture

There are multiple components to the test setup that runs in CI:

• on a weekly basis, scripts/populate_tox creates a PR that updates tox.ini with new releases
• tox.ini is where the configuration (what to test and what versions) is stored and read from
• scripts/split-tox-gh-actions/split-tox-gh-actions.py is the script that defines the test groups and generates the resulting YAML files
• scripts/split-tox-gh-actions/templates contains the jinja2 template snippets used to generate the YAML files
• .github/workflows/test-integrations-*.yml are the resulting workflow configuration YAMLs

If you update any of the components without keeping the rest in sync, for instance by making a change in one of the test-integrations-*.yml files that is not reflected in the templates in scripts/split-tox-gh-actions/templates, they will be overwritten next time they're auto-generated.