Falsifying Swarm Orchestrator
Version updated for https://github.com/moonrunnerkc/swarm-orchestrator to version v10.4.0-advisory.
- This action is used across all versions by 0 repositories.
Action Type
This is a Docker action.
Go to the GitHub Marketplace to find the latest changes.
Action Summary
Swarm Orchestrator is a GitHub Action and CLI tool designed to audit AI-generated pull requests by analyzing their diffs, scoring them with advisory-grade cheat detectors, and providing suspicion-score comments along with compliance artifacts (e.g., CycloneDX-ML and SPDX AI-Profile). It automates the detection of potential issues in PRs and supports both advisory (non-blocking) and gate (merge-blocking) modes, addressing the need for enhanced code review and compliance with AI-related regulations. Key capabilities include advanced detection, compliance reporting, and compatibility with AI procurement standards like the EU AI Act.
What’s Changed
Detectors are now high-recall candidate generators, with two stages after them deciding what reaches a reviewer. The goal is precision without silencing detectors or losing recall.
Measured on the 205-PR real corpus
- Advisory findings on the 195 clean PRs: ~1146 → 145 (−87%). Label-independent, the most reliable win.
- Deterministic path: precision 0.091, recall 0.300, F1 0.140.
- Judge gate: precision 1.000 / recall 0.000 — it refuses to block anything it cannot confirm. Trustworthy recall waits on a human-labeled corpus.
Honest caveat: the corpus “broken” labels are AI-generated and unverified, so block-level recall is not yet provable (the judge refuted all 10 “broken” PRs, consistent with either bad labels or missed detection). This release makes the tool a safe, low-noise advisory that will not false-block; it does not by itself prove the tool catches real cheats. The gate to that verdict is the human-labeled labels-v2 corpus.
Added
- Deterministic verification stage (
verify-findings.ts): refutes a candidate when the diff itself shows it is legitimate; demotes fix-claim-gated detectors to info when no fix is claimed. - LLM-judge confirmation gate (
confirm-findings.ts): a block must be judge-confirmed or it drops to advisory. Off by default. - Precision-based promotion gate with a Wilson lower bound, plus a CI freshness guard (
npm run promotions:check). - Per-finding confidence and a per-category cascade cap in the PR comment.
- Detector evasion-cost harness (
npm run evasion).
Changed
mock-of-hallucinationresolves internal roots from the diff, and the GitHub Actions version-ceiling finding drops from block to advisory.- The judge caps the diff it sends to Haiku so large PRs no longer fail the call.
- Published precision numbers repinned to the v10.4 deterministic snapshot.
Full detail in CHANGELOG.md.