Market-Leading Accuracy.
High Recall. Near-Zero Noise.
Find vulnerabilities without accepting a flood of false alarms. Cyfendo detected 94.06% of vulnerable Java test cases and achieved 93.78% precision on Python. Explore the results, then put it to work on your code.
Cyfendo-reported OWASP evaluations. Java and Python measured separately. Sources and methods below.
Measured detection fidelity across 3,970 test cases
Vulnerability scanning requires balancing high detection coverage against false positive suppression. Youden’s J measures that balance directly.
High detection is only half the story.
A scanner needs to find vulnerabilities and distinguish them from safe code. Youden’s J measures that balance. See Cyfendo’s results alongside selected published evaluations of established SAST tools.
| Solution / evaluated configuration | Youden’s J Pooled | Recall | Precision | FPR | Evidence |
|---|---|---|---|---|---|
| Cyfendo Evaluation | |||||
| Cyfendo; scan run September 2026 | 90.29% | 94.06% | 96.38% | 3.77% | Cyfendo Whitepaper |
| Research & Open-Source Evaluations (Ranked by Youden’s J) | |||||
| Semgrep CE; auto rules; binary unspecified | 47.70% | 89.96% | 69.45% | 42.26% | QASecClaw (May 3, 2026) |
| FindBugs w/ FindSecBugs v1.4.6 | 43.76% | 96.82% | 66.09% | 53.06% | OWASP Scorecard v1.2 |
| SonarQube Java Plugin v3.14 | 32.26% | 42.90% | 81.15% | 10.64% | OWASP Scorecard v1.2 |
| CodeQL 2.23.7 | 28.8% | 97.0% | 60.3% | 68.2% | Sifting the Noise (Jan 30, 2026) |
| Semgrep 1.145.0 | 15.6% | 90.4% | 56.3% | 74.8% | Sifting the Noise (Jan 30, 2026) |
| Joern 4.0.454 | 1.0% | 8.2% | 54.7% | 7.2% | Sifting the Noise (Jan 30, 2026) |
| SonarQube 9.9.8 Community | 1.0% | 95.6% | 51.9% | 94.6% | Sifting the Noise (Jan 30, 2026) |
| Industry Commercial Tools Baseline (OWASP Project Cohort) | |||||
| Commercial SAST Tools Average (6 Commercial Tools) | 25.50% | — | — | — | OWASP Commercial Baseline |
| Other Vendor-Reported Results | |||||
| Fluid Attacks SAST; build not pinned | —* | 100% | — | 0% | Vendor docs (July 17, 2026) |
| Static Reviewer (Security Reviewer) | —* | 98.0% | — | 1.2% | Vendor claims (2025/2026) |
| Kiuwan; historical 2017 run | 83.30% | 100.00% | — | 16.70% | Vendor report (Oct 30, 2025) |
| Xygeni; build unspecified | 83.3% | 100% | — | 16.7% | Vendor blog (May 6, 2026) |
* Vendor-Reported Notes: Fluid Attacks vendor reports J = 100%; Static Reviewer vendor
reports 98% TPR, 1.2% FPR (derived J = 96.8%); Kiuwan reports category-averaged J = 83.30%; Xygeni reports
100% recall, 16.7% FPR (derived J = 83.3%). Commercial SAST Tools Average (25.50%) represents OWASP's
category-averaged benchmark baseline across 6 anonymized commercial tools (SAST-01 to SAST-06, ranging
from 16.74% to 32.93%). Precision is not reproduced where unpublished. “—” does not mean zero.
Evaluation Qualifications: Separate evaluations, with different scoring methods, rule
sets, and test harness configurations. These are reported numbers, not a matched head-to-head test or a
current-product ranking.
Python: 407 vulnerable cases found. 27 false positives.
Of 434 flagged test cases, 407 were true positives: 93.78% precision, with 90.04% recall across the full suite.
| Solution / evaluated configuration | Youden’s J Pooled | Recall | Precision | FPR | Evidence |
|---|---|---|---|---|---|
| Cyfendo; scan run September 2026 | 86.57% | 90.04% | 93.78% | 3.47% | Cyfendo Whitepaper |
Note on Competitor Data: This review did not establish attributable full-suite Python scores for the established alternatives in the Java table. Missing published competitor data is an evidence limitation and is not presented as a performance claim.
Find it. Validate it. Work toward a fix.
Your team needs more than another list of alerts. Cyfendo connects code analysis, exploitability investigation and contextual patches in one workflow.
Follow the code.
Analyze the codebase and trace how data moves through the application to identify potential vulnerabilities.
Challenge the finding.
Investigate the context and, where applicable, validate exploitability in a sandbox.
Move toward a fix.
Generate contextual patches and retest to help developers address the underlying issue.
The benchmark results on this page measure detection performance. They do not measure exploit-validation or patch quality. Explore our technical whitepaper for architecture and validation methodology.
Every percentage has a denominator.
Benchmark percentages depend entirely on the underlying test suite, categorization, and aggregation methods. Inspect the exact counts, formulas, and evaluation differences below.
Full Test Counts & Confusion Matrices (Java & Python)
OWASP Benchmark Java v1.2 Confusion Matrix (2,740 Tests)
- Vulnerable Test Cases: 1,415
- Negative (Safe) Cases: 1,325
- Total Scored Test Cases: 2,740
- Recall (Pooled / Whitepaper): 94.06% (1,331 / 1,415)
- Precision (Pooled / Whitepaper): 96.38% (1,331 / 1,381)
- Specificity (TNR): 96.23% (1,275 / 1,325)
- False Positive Rate (Pooled): 3.77% (50 / 1,325)
- Youden’s J (Whitepaper Primary): 90.29% (0.90 Youden Index)
- Youden’s J (Category-Averaged): 90.55%
- F1 Score (Pooled, Secondary): 95.21%
OWASP Benchmark Python v0.1 Confusion Matrix (1,230 Tests)
- Vulnerable Test Cases: 452
- Negative (Safe) Cases: 778
- Total Scored Test Cases: 1,230
- Recall (Pooled): 90.04% (407 / 452)
- Precision (Pooled): 93.78% (407 / 434)
- False Positive Rate (Pooled): 3.47% (27 / 778)
- Youden’s J (Pooled): 86.57% (Recall − FPR)
- F1 Score (Pooled, Secondary): 91.87%
helpers/resources/insecureCmd.sh) outside the formal BenchmarkTestNNNNN
numbered test cases. That out-of-scope helper-file finding was excluded, leaving 434 in-scope positive
alerts (407 TP + 27 FP). It is not counted in the displayed confusion matrix.
Youden’s J Aggregation Explained (Category-Averaged vs. Pooled)
Methodology & Alignment with the Technical Whitepaper:
- Primary Whitepaper Score — Pooled Youden’s J (90.29% / 0.90): Calculated by treating all 2,740 test cases across the entire benchmark suite as one unified dataset: overall recall (94.06%) minus overall FPR (3.77%) = 90.29% (published as a Youden Index of 0.90 in the Cyfendo Technical Whitepaper). Pooled J is also the standard metric used in the comparative table to ensure direct consistency with published academic evaluations reporting whole-suite aggregate counts.
- Alternative OWASP Aggregation — Category-Averaged J (90.55%): The legacy OWASP Benchmark scoring script calculates recall and false positive rate individually across each of the 11 vulnerability categories (e.g., SQLi, XSS, Path Traversal, Command Injection, Weak Cryptography), subtracts FPR from recall for each category, and computes the unweighted arithmetic mean. Cyfendo’s category-averaged J of 90.55% is derived directly from the per-category counts detailed in our whitepaper.
- Python v0.1 Headline Score (86.57%): Uses pooled counts (overall recall 90.04% − overall FPR 3.47% = 86.57%). A category-averaged score is not established for this preliminary suite.
Interpretation: A Youden’s J of 100% represents perfect detection without false positives (100% recall, 0% FPR). A J of 0% indicates a tool whose true positive rate equals its false positive rate (no diagnostic power). J can be negative if a tool generates more false alarms than true detections.
Metric Definitions & Mathematical Formulas
Note on Precision vs. FPR: False Positive Rate (FPR) measures the fraction of safe control cases incorrectly flagged, not the fraction of reported alerts that are false. Precision measures the fraction of reported alerts that are real vulnerabilities. All metrics are calculated from unrounded counts, then rounded to displayed precision; subtracting displayed rounded percentages directly can introduce a 0.01-point rounding artifact.
Source Differences & Evaluation Configurations
- Sifting the Noise (arXiv:2601.22952, Jan 30, 2026): Evaluated CodeQL 2.23.7, Semgrep 1.145.0, SonarQube 9.9.8 Community Edition, and Joern 4.0.454. Uses file-level positive scoring rather than confirmed intended-CWE detection. SonarQube scores count vulnerabilities and exclude security hotspots. Joern uses its default Java scanner ruleset (6 rule files), yielding low false positives (7.2% FPR) but very low recall (8.2%). Pooled J values are calculated from the paper's displayed rounded recall and FPR figures, preserving one decimal place. Scanner execution and exact rule pin configurations were not independently rerun in this review.
- QASecClaw (arXiv:2605.01885, May 3, 2026): Evaluated Semgrep CE with automatic rule selection (unspecified binary version). Scorer counts are TP 1,273, FN 142, FP 560, TN 765. The scoring script accepts any finding in a numbered benchmark file rather than strictly verifying CWE match. Recounting published IDs verifies tally arithmetic, not scanner execution.
-
OWASP Benchmark v1.2 Official Scorecards: Baseline reports generated by the OWASP
project using the official
BenchmarkScoreutility for open-source tools against the 2,740 test cases of OWASP Benchmark Java v1.2. FindBugs with FindSecBugs v1.4.6 scored 1,370 TP, 45 FN, 703 FP, 622 TN (43.76% pooled J; 39.10% category-averaged J). SonarQube Java Plugin v3.14 scored 607 TP, 808 FN, 141 FP, 1,184 TN (32.26% pooled J; 33.34% category-averaged J). - OWASP Commercial Tools Baseline (SAST-01 – SAST-06): Aggregated benchmark results across 6 leading commercial SAST tools anonymized in accordance with commercial software license terms. The commercial cohort averaged a 25.50% category-averaged Youden's J score, with individual tool scores spanning from 16.74% (SAST-01) to 32.93% (SAST-06).
- Fluid Attacks SAST: Vendor documentation updated July 17, 2026; execution date unspecified. The vendor reproduction recipe includes selected checks, an unpinned container, and a custom parser. Fluid Attacks reports J = 100%; aggregation is unspecified. Documentation freshness date is not an execution date.
- Static Reviewer (Security Reviewer): Vendor-published performance claims report 98% TPR and 1.2% FPR on the OWASP Benchmark. Test suite version and execution environment are vendor-reported.
Benchmark Limitations & Independent Verification Notice
Synthetic Test Case Scope: OWASP Benchmark suites consist of synthetic, labeled test cases. While valuable for evaluating taint tracking and syntactic discrimination under controlled conditions, benchmark scores alone cannot establish real-world vulnerability coverage, complex business-logic flaw detection, developer triage time saved, or the correctness of proposed code fixes.
Evaluation Status: Cyfendo's figures are vendor-reported measurements conducted in September 2026 against the public benchmark suites. The specific evaluated build commit hash is unpinned; results are not independently reproduced.
OWASP Independence Notice: OWASP is a registered trademark of the OWASP Foundation, Inc. Cyfendo's evaluation is an independent study against open benchmark suites and does not imply endorsement, sponsorship, or certification by OWASP.
How we count
Supporting evidence & source register
Full citation register and additional published vendor evaluations reviewed through September 22, 2026.
Additional vendor evidence: Kolega.dev (Java and Python)
Evaluation Context: Kolega.dev published self-reported OWASP Benchmark evaluations on February 26, 2026. Its evaluated build was designated unreleased at publication time. These are separate vendor evaluations.
| Suite / run | Pooled J | Recall | Precision | FPR | F1 (Secondary) | Evidence |
|---|---|---|---|---|---|---|
| Java v1.2, run 1; selected best J of 3 | 87.38% | 96.89% | 91.58% | 9.51% | 94.16% | Vendor blog (Feb 26, 2026) |
| Python v0.1, run 1 | 89.79% | 99.56% | 85.55% | 9.77% | 92.02% | Artifact JSON (Feb 26, 2026) |
| Python v0.1, run 2; selected best J of 3 | 90.90% | 98.23% | 88.62% | 7.33% | 93.18% | Artifact JSON (Feb 26, 2026) |
| Python v0.1, run 3 | 89.59% | 100.00% | 84.80% | 10.41% | 91.78% | Artifact JSON (Feb 26, 2026) |
• Java run 1: TP 1,371; FN 44; FP 126; TN 1,199 (Total: 2,740)
• Python run 1: TP 450; FN 2; FP 76; TN 702 (Total: 1,230)
• Python run 2: TP 444; FN 8; FP 57; TN 721 (Total: 1,230)
• Python run 3: TP 452; FN 0; FP 81; TN 697 (Total: 1,230)
Comparison Disclosures: For Java, Cyfendo reports higher precision (96.38% vs. 91.58%) and higher pooled J (90.29% vs. 87.38%) than Kolega’s selected Java evaluation, while Kolega reports higher recall (96.89% vs. 94.06%). For Python, Cyfendo reports higher precision (93.78% vs. 84.80%–88.62%) and lower FPR (3.47% vs. 7.33%–10.41%) than all three Kolega runs, but lower recall (90.04% vs. 98.23%–100.00%); Kolega’s selected run has higher J and F1. All three Python runs are preserved to show run-to-run variability.
Additional Java Context & Enterprise Tools Policy
- Kiuwan: Recall 100.00%, FPR 16.70%, category-averaged Youden’s J 83.30%. This is a historical 2017 result republished on October 30, 2025. While the article mentions a subsequent rerun, the displayed historical numbers are not established as the result of that rerun.
- Xygeni: Recall 100%, FPR 16.7%, derived Youden’s J 83.3% cited in a May 6, 2026 article. The evaluated product version, aggregation method, and run metadata remain unresolved. This J is not established as pooled or category-averaged.
- Enterprise Tool Policy: Commercial proprietary SAST vendors (including Snyk, Checkmarx, Veracode, Mend, and Fortify) include restrictive benchmarking clauses in their End User License Agreements (EULAs) prohibiting the publication of comparative benchmark scorecards. In strict adherence to fair reporting standards, Cyfendo does not publish unverified scores or reuse obsolete free-tier results to represent current enterprise commercial products. Missing evidence does not imply zero performance.
Primary Sources & Public References
Evidence inventory reviewed through September 22, 2026.
Strong on the benchmark. See it in your code.
Start a scan. Inspect the findings. Evaluate the proposed fixes. See how Cyfendo fits the way your team builds and ships software.