How to test fraud rules in shadow mode before they touch customers

Shadow mode means running a fraud rule or score threshold on live payments while it changes nothing: it only records what it would have done. After enough time for chargebacks and refunds to arrive, you compare those would-be decisions with real outcomes and enforce only the rules that catch fraud without blocking too many good customers.

Updated 1 November 2026 · RiskCogs team

Every fraud rule is a guess about what fraud looks like. Most guesses are partly right. A rule that blocks orders where the card country differs from the IP country will catch some fraud, and it will also catch travellers, expats, people on VPNs and customers on corporate networks. You cannot know the balance until you see it on your own traffic.

Enforcing a rule straight away means learning that balance from angry customers and lost sales. Shadow mode lets you learn it from logs instead. The rule sees the same payments it would see in production, makes the same decision, and writes it down. Nothing changes for the customer.

It also makes discussions easier. Instead of debating whether a rule is too strict, the team can look at what it would have done over a month and decide together.

1. Write down the hypothesis in one sentence: which fraud the rule targets and which signal it uses.
2. Keep the rule narrow. One condition set per rule makes results easy to interpret.
3. Run it on all live traffic in shadow, alongside your current rules, so the comparison is fair.
4. Log for each payment: the payment ID, the shadow decision, the reason, the score and the time.
5. Make sure outcomes can be joined to those logs later: chargebacks, refunds for fraud, manual review results.

The main constraint is time, not volume. A card payment can be disputed weeks after it was made, and many disputes arrive in the first month or two. If you judge a rule after three days, you will see the payments it flagged but not which of them turned into chargebacks.

A practical approach is to run the rule for a few weeks to collect flagged payments, then keep watching those payments for outcomes for a further period before deciding. For fast-moving attacks such as card testing, you can judge sooner, because the pattern is obvious in the logs: bursts of small attempts, mostly declined by issuers, from a handful of devices.

Also make sure the test window is normal. A shadow test that covers only a holiday sale or a marketing push will not represent ordinary traffic.

For each rule, build a simple table. Of the payments the rule would have blocked, how many later became fraud chargebacks or were confirmed as fraud? How many were clean orders? And of all the fraud in the period, how much did the rule catch?

The first ratio tells you how precise the rule is: when it fires, how often it is right. The second tells you its coverage. A rule that catches a small share of fraud but is almost always right is a good block rule. A rule that catches a lot of fraud but also many good customers belongs in a review or authentication band, not a hard block.

Look at the good orders it would have stopped, not just the count. Ten blocked first-time buyers of low-value items are a different cost from ten blocked repeat customers placing large orders.

Imagine a store testing a rule that blocks first orders over €300 when the card country differs from the IP country. These numbers are made up to show the method, not real results.

Over four weeks, the rule would have blocked 120 orders. Over the next weeks, 18 of those turn into fraud chargebacks and 4 more are refunded after the customer reported card misuse. That is 22 fraud cases out of 120, so the rule is right a little under one time in five. The other 98 were good customers, many of them travellers and expats buying gifts.

In the same period, the store had 40 fraud chargebacks in total, so the rule would have caught about half of them. Good coverage, poor precision. The sensible decision is not to block, but to send these orders through 3-D Secure, which keeps most of the protection and lets the 98 good customers complete their purchase.

The team then narrows the rule: add a data-center IP or a newly created email as a second condition, and run the narrower version in shadow for another cycle.

• Enforce as a block when the rule is highly precise and the cost of a false decline is lower than the fraud it stops.
• Enforce as review or 3-D Secure when it catches meaningful fraud but also stops good customers.
• Narrow it when a sub-group is causing the false positives, such as a specific country or card type, and test again.
• Drop it when it adds nothing that existing rules or the score do not already catch.
• Keep a record of each decision and the numbers behind it, so the next person understands why the rule exists.

The same method works for a risk score. Instead of one rule, you test cut-off points. Score every payment in shadow, wait for outcomes, then look at fraud and good orders in each score band. Where does fraud start to concentrate? How many good customers sit just above the line you are considering?

Moving the block threshold down a few points often stops a little more fraud and many more good customers, because most payments sit in the low and middle ranges. Seeing the actual distribution on your own traffic makes the choice much easier than picking a round number.

• Judging the rule before chargebacks have had time to arrive.
• Changing several rules at once and not knowing which change caused the effect.
• Testing only on a slice of traffic that is not representative.
• Forgetting that your existing rules already block some fraud, so the shadow rule never sees it.
• Not logging the reason with each decision, which makes later review guesswork.
• Leaving a rule in shadow for ever; set a review date when you start.

Shadow mode is not only for new rules. Teams that keep a shadow copy of their next candidate rules running all the time can react to new attacks faster, because they already have data when the pattern appears. It is also a safe way to test the impact of loosening a rule, which is how false declines get fixed.

Think of it as a staging environment for risk decisions. Production rules protect the business today; shadow rules are where tomorrow's rules prove themselves.

RiskCogs starts every account in shadow mode. Each payment sent to POST /v1/risk-score is scored and given a recommendation and reason codes, but your payment flow does not change. The would-have-blocked report shows which payments the default bands (allow under 40, review 40 to 69, block from 70) or your rules would have stopped.

Send chargebacks and refunds to POST /v1/risk-feedback with the riskId, and the report shows which of those would-be blocks were real fraud. Switch to enforcement when the numbers look right. The Shadow plan is free; enforcement starts on Starter at €49 a month.

• Visa, Visa Core Rules and Visa Product and Service Rules (public edition), dispute time limits.
• Mastercard, Chargeback Guide (merchant edition).

• Fraud rules engine guide
• Transaction risk scoring
• How shadow mode works
• False declines
• Pricing
• Talk to the RiskCogs team