Reviewing AI-Generated Code
This playbook extends the standard code review process with AI-specific checks for common failure modes in AI-generated code. It covers hallucinated imports, meaningless tests, over-engineering, speculative changes, missing context, and subtle behavior changes. By following these steps, reviewers will catch issues that automated tools miss while maintaining the same quality standards as human-written code.
Related resources:
- Reviewing a PR — base review process
- Code Quality Standards — test quality requirements
- Sentry Skills — find-bugs skill for systematic detection
Apply the full review checklist from Reviewing a PR:
What, why, linked issue.
You MUST NOT review failing code.
Runtime errors, performance, side effects, backwards compatibility, security, test coverage (Test requirements by change type), test quality (Test quality).
Public API, dependencies, schema changes, security-sensitive code, frameworks.
AI-generated code has specific failure modes. You MUST check for these in addition to the standard review:
Verify every import and function call actually exists. AI tools sometimes reference packages, modules, or functions that don't exist or have different signatures than expected.
You MUST check that test assertions would actually fail if the feature broke (Test quality). Watch for: hardcoded expected values that happen to match the output, assert True or equivalents, testing mock behavior instead of real behavior, asserting only that no exception was thrown.
AI tools frequently add unnecessary abstractions, configuration options, and error handling for impossible cases. Ask: "does this need to be this complex?" If a simpler approach works, request it.
Code changes beyond what the issue or PR describes (One logical change per PR). If the PR is "fix null check" but also reorganizes imports and adds docstrings, request a split.
AI tools may not understand SDK-specific patterns and conventions. Check that the change fits the SDK's existing architecture, not just generic "good code" patterns.
Pay extra attention to edge cases in any "cleanup" or "refactor" PR. AI refactors sometimes change semantics in ways that aren't obvious from a quick scan.
You SHOULD use the sentry-skills:find-bugs skill for systematic bug and vulnerability detection in the diff.
- Review feedback conventions — LOGAF scale and blocking criteria
- Test requirements by change type — test coverage expectations
- Test quality — meaningful assertion requirements
- AI attribution — Co-Authored-By footer requirement
- One logical change per PR — focused PR scope
| Version | Date | Summary |
|---|---|---|
1.0.0 | 2026-02-21 | Initial playbook — specialized review techniques for AI-generated code with common failure modes |
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").