CodeQL 2.26.3 Changes Findings: Rebaseline AI-Generated Code Scan Cost

A static-analysis upgrade changes the economics of AI-generated code even when compute pricing stays flat. New source and sink models may find defects that older scans missed; query refinements may remove noisy alerts. Teams should rebaseline scan and triage cost after CodeQL 2.26.3 instead of comparing raw alert counts across versions.

GitHub's August 19 changelog describes new JavaScript, TypeScript, Vue, Sails, Windows, and Ruby modeling plus several GitHub Actions query changes. It also removes codeql.actions.security.SelfHostedQuery, improves cache-poisoning and untrusted-checkout paths, and reduces false positives in selected cases. GitHub.com receives versions automatically; other environments may require an upgrade.

Freeze a Comparison Cohort

Select repositories that represent generated and human-written code, JavaScript frameworks, Actions workflows, and typical pull-request sizes. Record commit SHAs and configuration. Scan the identical cohort before and after the engine update. Comparing different weeks or changing code mixes version effects with workload drift.

Preserve alerts by query ID, severity, path, reachability, disposition, and reviewer minutes. A new alert count is not automatically regression or improvement. Classify each delta as new modeling, changed precision, code change, configuration change, or analysis failure.

Calculate Cost per Useful Alert

Include runner minutes, cache and database storage, upload time, engineer triage, remediation, rescans, and blocked merge time. Divide by alerts confirmed as actionable, not by all emitted results. If a scan costs $40 in compute and 12 reviewer hours at $100 per hour to find eight valid issues, the useful-alert cost is $155 before remediation.

Also estimate avoided loss. A finding that prevents credential theft or workflow cache poisoning can justify substantial analysis cost. Keep expected loss separate from direct spend and show uncertainty. Security value should not become an invented savings number added to every alert.

Treat AI Output as a Segment

Tag pull requests by generation assistance without storing sensitive prompt content. Compare finding density, severity, triage time, and remediation latency for AI-assisted and baseline cohorts. Control for repository and change size. AI-generated code often changes volume and pattern, so alerts per thousand changed lines are more useful than total alerts.

Feed structured, confirmed findings back to the coding agent only after sanitizing untrusted content. Limit repair attempts and rescan the actual patch. Do not ask the model to reinterpret an entire database when query ID, location, flow path, and remediation guidance are available.

Plan for Query Breakage

• Inventory custom queries that import the removed self-hosted module.
• Compile query packs before changing the production scanner.
• Run a canary cohort and retain the previous results for comparison.
• Document accepted alert changes and update suppression ownership.

A broken custom query can silently reduce coverage or stop a pipeline. Price migration labor and temporary dual scans into the upgrade. Do not weaken a gate merely to restore speed; fix the query or explicitly accept the changed coverage with an owner and expiry.

Use Versioned Security Metrics

Dashboards should display CodeQL version beside alert and cost trends. Track analysis duration, failure rate, confirmed-positive rate, time to first triage, time to remediation, and cost per confirmed issue. Report p95 scan time because large generated changes can dominate merge queues.

Review the baseline whenever queries, languages, build modes, runner types, or repository composition change. A stable metric with an invisible engine change is not comparable. Version metadata is part of the denominator.

Build an Upgrade Scorecard

Create one row per repository and language with old and new engine versions, scan duration, database size, alerts added, alerts removed, confirmed positives, dismissals, custom-query failures, and reviewer minutes. Require owners to explain material deltas before removing the older baseline. This prevents an automatic deployment from becoming an invisible measurement discontinuity.

For GitHub Actions, include merge-group, scheduled, pull-request, and low-trust cache scenarios because the release changes their modeling. For JavaScript and Vue, include route parameters, reactive values, framework inputs, and promise-wrapped responses. The cohort should exercise the announced changes rather than relying only on historical repositories that never use them.

Set an acceptance threshold for analysis failures and confirmed-positive coverage, not a target alert count. If scan time rises, profile the affected queries before buying runners. If findings drop, verify that improved precision—not missing extraction or broken packs—caused the change. Archive the scorecard with the engine rollout decision.

Revisit the scorecard after query-pack, build-mode, framework, or repository changes so later cost comparisons retain the same evidence boundary.

Bottom Line

CodeQL 2.26.3 can improve coverage and precision, but its value appears only after a controlled rebaseline. Compare the same commits, migrate custom queries, price triage, and segment AI-assisted changes. Optimize for confirmed risk removed per dollar and minute, not the smallest or largest alert count.