False Positives and False Negatives in AppSec

A pull request turns red on a string that only looks like a secret. The reviewer clears it, then clears the next three the same way. By the end of the week, the authorization bug that matters is sitting in that same list, unread.

That is what false positives and false negatives do to a delivery team. One error burns reviewer time on a benign change. The other lets a real violation through. A high accuracy score can hide both, especially when almost every change is fine.

The decision is narrower than "turn the scanner up." It is which checks stay comments, and which ones are allowed to stop a merge.

Why Scanner Accuracy Misleads Security Teams

The 2025 SANS Detection and Response Survey reported that 73% of respondents identified false positives as their leading detection challenge, compared with 64% in 2024. That queue used to live in a SOC. It now lands on a pull request.

Say the check flags a fixture in a test file. The change is harmless. The reviewer clicks through, then starts clicking through the next alerts without reading them. A false positive treats a benign change as a violation. A false negative lets a real violation through.

Why a quiet detector still fails

When almost every change is fine, a detector that rarely speaks looks accurate. It can still flood review, or it can miss the authorization bug that matters. Accuracy counts the easy cases. It does not price the minutes a reviewer spends, or the exploit that ships.

Application security automation only helps if the gate matches the risk. The path from here is a worked comparison of the two errors, then how rare violations change whether the same rule is trustworthy, which bugs a file-level scan cannot see, and how to budget each error by control and release stage.

Practical rule: Keep noisy checks as comments. Allow a check to block a merge only when a miss is unacceptable or a person has confirmed the finding.

False Positives Versus False Negatives With the Metrics That Matter

A scanner can catch most real violations and still send reviewers a queue dominated by noise. To understand the trade-off, track four outcomes: a true positive is a real violation caught, a false positive is a benign change flagged, a true negative is a benign change left alone, and a false negative is a real violation missed.

The Google machine learning crash course's definitions of classification metrics distinguish three useful ratios. Precision is TP/(TP+FP), the share of alerts that are real. Recall, or sensitivity, is TP/(TP+FN), the share of real violations caught. Specificity is TN/(TN+FP), the share of benign changes left alone.

One queue of 100,000 changes

Consider 100,000 reviewed changes, with 100 genuine violations and 99,900 benign changes. A detector with 95% recall catches 95 violations and misses 5. At a 1% false-positive rate, it also flags about 999 benign changes, leaving precision at about 8.7%. Lowering the false-positive rate to 0.1% produces about 100 false alarms and raises precision to about 49%.

The lower false-positive rate cuts the review burden, but it does not recover the five missed violations. Those require better detection or a control that can see design context the scanner lacks. A high overall accuracy score would obscure both costs because benign changes make up nearly all of this queue.

Where the check is allowed to block

Use high-recall advisory checks while a design is exploratory, when reviewers can investigate findings without stopping every merge. For a blocking policy, require stronger precision or human confirmation, and weigh the reviewer workload against the cost of a missed violation. A single global threshold cannot serve every control and repository when violation prevalence differs.

The same distinction matters when teams compare scanners with manual testing. False positive validation for MSSPs describes how pentesting can help validate scanner findings. Set the merge gate according to the risk and the evidence behind each finding, not accuracy alone.

How Rare Violations Destroy Precision

Sensitivity and a false-positive rate are not the outcome a reviewer feels. Positive predictive value is the share of alerts that are real.

A review of predictive values in low-prevalence testing gives the relationship teams should use. PPV equals sensitivity times prevalence, divided by that quantity plus the false-positive rate times one minus prevalence. Negative predictive value depends on specificity and prevalence in the same way.

The same rule, two repositories

Hold the control fixed at 99% detection and a 1% false-positive rate.

• Broad changes at 0.1% prevalence: Across 100,000 changes the control finds about 99 true violations and about 999 benign alerts. PPV sits near 9%.
• A targeted high-risk repository at 10% prevalence: The same control finds about 9,900 true violations and about 900 false alarms. PPV rises to about 92%.

One organization-wide cutoff is the wrong tool. Broad repositories need suppression, deduplication, risk ranking, and non-blocking feedback. High-risk services can justify a stricter gate and a deeper look.

A clean bill of health can look excellent when violations are rare, even while the detector still misses a meaningful number of true cases.

What to report

Report a confusion matrix by repository, policy type, and release stage. Price reviewer minutes separately from the cost of a missed vulnerability. A false-negative rate needs ground truth, so sample changes that passed, and back-label escaped issues from incidents and penetration tests. Put the miss count beside each repository's matrix. Which control deserves which budget is a later decision, not a single slider.

Why Local Scans Miss Real Design Flaws

A local scan can judge a code pattern without seeing the conditions that make it safe or dangerous. A secret-shaped string in a fixture, a call protected by framework configuration, or code restricted to a test path can trigger a false positive.

The harder misses cross boundaries. An authorization check may look valid in one function while another service lets a user alter someone else's record.

What the studies actually measured

Research on secure code generation, summarized in an ACM paper on detection limits, found that three static analyzers missed 12–19 of 44 vulnerabilities, while LLM-based methods had false-positive rates near 50%. A separate study of hardcoded-secret detection found that test-path context contributed to generative models' misclassifications, and that prompt changes reduced errors.

Adding scanners does not resolve these limits. More detectors can create duplicate or conflicting findings, while flaws spread across functions or services remain difficult to detect from snippets alone.

Evidence you already have

Repository analysis shows what the code does. Tickets and design decisions show the intended control; runtime or deployment metadata shows what is exposed. A threat modeling process can connect those sources when the flaw crosses system boundaries. Defer an alert only when its rationale is recorded and reviewable.

Error Budgets by Control and Release Stage

Maximum recall is not the safest setting. Harvard researchers on test thresholds report that lowering false negatives generally creates more false positives. Preferred cutoffs differed from standard methods because people weigh harm, unnecessary cost, and their own risk tolerance.

The security question is the same shape. Which missed vulnerability is unacceptable, and which alert is safe to defer? A queue full of low-value findings trains reviewers to ignore alerts. An aggressive false-positive rate can raise the false negatives that ship.

Budgets follow the asset, not the scanner brand

Build the model by asset criticality and exploitability. Authentication bypasses, payment-authorization flaws, and exposure of health data get a near-zero miss budget. Ambiguous findings on those controls go to a person. Low-impact code-quality notes can stay unblocked.

Suppression is allowed only when the rationale is recorded and reviewable later. That matters for test-path secrets, a common false-alarm source. Early design exploration can tolerate noisy, high-recall comments. The release candidate cannot.

I would not share one global scanner threshold across those rows. The miss you cannot accept sets the gate. Everything else stays a comment until someone records why it was suppressed.

Set Merge-Time Checks Before the Next Release

Five moves, in order, before the next release candidate.

1. Split checks by stage. High-recall comments while a design is exploratory. A short list of high-precision or human-confirmed rules at merge time.
2. Retune each rule from a confusion matrix. Do it per repository and policy. Track precision, recall, and reviewer minutes. Remeasure after every threshold change.
3. Add context before you add another scanner. Tickets, design decisions, service metadata, and deployment exposure show whether an identity boundary holds.
4. Encode the approved decision as policy on the pull request. Policy as code should block only controls in the strict budget, and it should store a reason for every exception.
5. Refresh the threat model as tickets and repositories change. A model that froze at design review will not catch the boundary that moved last sprint.

Three lists to leave the room with

Must not ship, with near-zero tolerance for a miss: an authentication bypass, a payment-authorization flaw, and health-data exposure.

Stay comments: low-impact quality notes, and test-path secrets once a reason is stored.

Blocks nothing yet: any rule whose alerts are mostly benign in rare-violation repositories. Sample last quarter's blocked merges and escaped issues so you can see which error the team paid for.

DevArmor is one place this can live, where threat models, merge enforcement, and guardrails for AI coding agents meet the pull request. It is unnecessary when one over-broad rule is the only problem. Fix that rule first.

Agree the operating line before the branch cuts. Advisory while the design moves. Precise, or confirmed by a person, before merge. A recorded reason for every suppressed alert.

DevArmor keeps a living security context across tickets, design decisions, and pull requests, then enforces the policies you have already agreed. If your merge gate is drowning in benign alerts while a real authorization flaw can still ship, start by separating advisory checks from the few controls allowed to block. Visit DevArmor to see how continuous threat modeling and policy enforcement fit the checks you set before the next release.