Verifying trading scripts and claimed backtests — mine included. Author, Trading as a System. Free: the 10 checks I run before I believe a backtest ↓
Joined February 2021
- Tweets299
- Following2.5K
- Followers739
- Likes7.3K
Pinned Tweet
Most backtests fail on arithmetic, not strategy.
60% wins over 25 trades: the 95% interval is roughly 41–77%. A profit factor of 2.0 at 35% wins needs winners 3.7× the size of losers.
The 10 checks I run before I believe a backtest, free on one page:
quanticator.app/go/checklist…
Ship log: I killed my Alpaca paper crypto bot on 30 Sep, then wrote the rule that allowed it on 26 Oct.
Frozen config, replayed on 145 unseen days (7 May to 29 Sep 2026): 93 trades, gross +4.7 ± 8.3 bps a trade, net -55 against a 60 bps round trip.
Evidence first, rule second.
🤖 Made with AI
Four hundred trades is not four hundred times the evidence of one. It is twenty times.
That ratio decides how many of the backtests you are shown mean anything, and almost nobody does the arithmetic. Seven posts.
ALT Card: Same 58% win rate, three sample sizes. 50 trades: 44 to 71%. 100 trades: 48 to 67%. 400 trades: 53 to 63%. Wilson 95% score intervals on 29/50, 58/100 and 232/400. Width falls with the square root of n.
Trade count is not sample size. Twenty fills taken over one weekend in two instruments that move together are closer to one observation than twenty.
Count independent events. I have made exactly this mistake on my own ledger.
Nine winners out of ten. The textbook interval puts the win rate between 71% and 109%.
A hundred and nine percent. That is the normal approximation falling over at small n, and it is the formula most calculators use.
Wilson gives 60 to 98: still wide, and possible.
🤖 Made with AI
A big trade count is good evidence right up until you try to audit it.
TradingView's docs say it plainly: on the default testing range, a strategy preserves individual data for up to the latest 9000 trades. Past that, the list you'd check is a tail.
Large n, small audit.
ALT Quote from the TradingView Pine Script docs, Strategies, Trades tab: 'If a strategy uses the default testing range, it preserves individual data for up to the latest 9000 trades.' Note: Deep Backtesting keeps every trade; on the default range, past 9000 the list you can audit is the latest tail.
I'd want the trade count next to any PF. A true PF of 1.5 (half the trades win 1.5R, half lose 1R) prints anywhere from 0.84 to 2.67 over 50 trades, and below 1.0 about 6% of the time. Over 500 trades: 1.26 to 1.79. nitter.cf/PeterLBrandt/status/21…
Unlike all the braggarts on X touting their "moon-shot" success, I do not release company ROR data - but, by example, highly encourage aspiring traders to be data rich on the much more meaningful measures such as PF, Calmar, EV, DDs and other things.
I have released data in the past - a long time ago, because my trading performance from 1981 through 2010 had been audited by a CPA/auditing firm using month end brokerage statements, a CFTC guidelined performance table and IRS tax statements.
I don't release this anymore because it is old and I am against releasing un-audited ROR data.
We do our trading through ADMIS, IBKR and plan to test-drive Stone soon.
The company enters trades in units sized to multiples of $1M - so perhaps, six Bean Oil contracts per $1M notional.
The notional is all backed by cash and T-Bills, but we only keep about 20% of notional value on deposit with our brokers.
Seldom is the margin-to-equity ratio of our open trades more than 10-15% of total notional.
We are angled to make about 150 or so trades per year across all the globe's liquid commodity futures exchanges from Chicago to NY to London to Zurich to Hong Kong to Paris to Malaysia.
We make all of our decisions based on monthly and weekly price charts. No fundamentals. No indicators. Simple charts. We trade about 5 to 10 different set ups that we have traded hundreds of times each over the 50 years.
I gave a speech to the MTA (now CMT certification) in 1991. I recently came across a copy and our trading style has changed almost none since 1981. Whereas correct interpretation was the payoff for us then, sizing, risk and trade management and engineered processes are now where our edge lies.
I believe many of you young chartists way way way over-complicate your charting. Somehow many of you think that adding "just one more" indicator or tool to your efforts will offer the magic sauce.
The magic sauce is SIMPLICITY.
You've seen the charts I post. Any more than one marked line is too often too many to be worthwhile.
A final comment to you if you are a day trader. Good luck. The data is solid - no more than 3 in 1,000 of you who seek to make trading your livelihood will succeed. Disbelieve me at your own risk (and years wasted trying)
Only you have the answer for you. Believing this last sentence and not wasting time time, money and energy looking for shortcuts will shorten your search.
That's it for today
My paper equity record read +$91.88 over six closed trades. Then I read the logs and found five more nobody had written down. All five lost.
IBIT and FBTC, 9/14 to 10/6, Alpaca paper, bot-recorded prices before fees: +$45.16 over eleven.
A trade count is a reading, not a fact.
🤖 Made with AI
Fifty trades, 56% winners.
The 95% interval on that runs 42.3% to 68.8%. At the bottom of that range, a 1:1 payoff loses money.
Two minutes: find the trade count, find the win rate, put an interval on it.
That's one of ten. The other nine are on one page, in my profile.
🤖 Made with AI
One question ran through this whole week: how many configurations were tried before the one you were shown?
Searching isn't cheating. It reprices the bar. And the count that sets the new bar is the number a report almost never carries, so you have to ask for it.
Selection is check 9 of the ten I run before I believe a backtest. The whole page, free: quanticator.app/go/checklist…
Count the inputs before you read the result.
Three tunable inputs, five plausible values each, and the sweep behind that one chart is 125 variants. At a 5% family-wise rate the winner needs a t near 3.54, not 1.96.
Nobody reports 125. The dialog does.
ALT Card: Count the inputs, not the tests. 3 inputs x 5 values = 125 variants. 0.05 / 125 = 0.0004, which is a two-sided t of 3.54, not 1.96. Two-sided normal quantiles, computed; Bonferroni is conservative when the variants correlate.
Ship log: I wrote a research tool that refuses to test an idea I had not declared in advance.
Three horizons, frozen in a protocol file before the data was read. Anything off that list is rejected and every override is logged.
I don't trust myself to count attempts afterwards.
ALT Card: The protocol was frozen before the data was read. 1: horizons declared up front, 15, 60 and 240 bars. 2: the tool rejects any horizon off the list. 3: overriding it is logged, not silent. Market Logger research protocol, adopted 2026-09-05; sixty declared equity tests set the bar at t 3.34.
A report shows you one configuration. The number it almost never gives you is how many were tried before that one.
Eight posts on the arithmetic of that question, which is harsher than most people expect.
ALT Card: One configuration shown, forty tried. 87%: the chance a zero-edge sweep of 40 variants finds a winner. 1 - 0.95^40 = 87.15%, computed. The expected best t from 40 noise variants is 2.16, so a reported 2.0 is below average for nothing.
The honest bar sits somewhere between 1.96 and 3.23, and 1.96 is the one value it certainly is not. If you want the figure rather than the bracket, bootstrap the whole set of variants together and let it price the correlation.