Backtest Limitations

A backtest is a simulation — not a prediction. Understanding where backtests can mislead you is just as important as knowing how to run them.

Warning

Past performance does not guarantee future results. Every backtest result is subject to the limitations described on this page. Always validate on out-of-sample data before trading any strategy live.

Survivorship bias 

What it is: Historical stock datasets tend to include only companies that survived — those still listed today. Companies that went bankrupt, were acquired, or were delisted are often missing from the data.

How it distorts results: If your strategy randomly held 10 NASDAQ stocks in 2001–2003, it might have held some that later went bankrupt. A dataset without those companies would show your strategy avoided those losses — but you had no way to know in advance. This makes every backtest result slightly optimistic.

In Quantlens: The NASDAQ dataset includes delisted stocks up to their final trading date, which partially mitigates this bias. However, stocks removed from the universe due to illiquidity (rather than bankruptcy) may still be absent.

Practical impact: The more speculative or small-cap your universe, the larger the survivorship bias. For large-cap, liquid stocks (top 500 by market cap), the impact is small.

Look-ahead bias 

What it is: The strategy inadvertently uses information that would not have been available at the time of the trade.

Common causes:

Using closing price to make a decision and execute at the same close. In reality, a market order at exactly the closing price is difficult to achieve. Use Execution = sod (next day’s open) for more realistic fills.
Indicator offset = 0 when it should be 1. An indicator computed on today’s close and used to trigger a trade on today’s close means you knew the close before placing the order. For daily-close signals, this is usually acceptable (end-of-day systems are standard practice), but be aware of the assumption.
Reusing data across train/test splits incorrectly. If you optimise on the full date range and then report those optimised results as “out-of-sample”, you have look-ahead bias at the parameter level.

Transaction cost assumptions 

Backtests assume frictionless execution unless you explicitly add costs. Real trading always has friction:

Cost type	Description
Commission	Broker fee per trade. Set in backtest settings (e.g. 0.1%). Compounds over many trades.
Slippage	Difference between expected and actual fill price. Large orders move the market against you.
Bid-ask spread	You buy at the Ask and sell at the Bid. The spread is an implicit cost on every trade.
Market impact	Large orders in illiquid stocks push prices, especially at entry and exit.

Recommendation: Always include realistic commission (0.05–0.1% for retail) and at least 0.1% slippage in your backtest settings. Zero-cost backtests are unreliable.

Commission drag example:

Commission:    0.1% per trade (round-trip = 0.2%)
Rebalance:     weekly (50 rebalances/year)
Avg turnover:  50% of portfolio per rebalance

Annual drag = 0.2% × 50 × 50% = 5% per year

A strategy returning 12% gross would return only 7% net after this commission drag.

Liquidity assumptions 

Backtests assume you can buy and sell any stock in the universe at the recorded price, in any quantity, at any time.

In reality:

Thinly traded stocks may not have enough liquidity to fill a $10,000 order at the close price.
Opening gaps — a stock that gaps down at open may be impossible to exit at yesterday’s close price.
Position concentration — if your strategy holds 5 slots of $20,000 each in small-cap stocks, your orders may represent a significant fraction of daily volume, moving the price against you.

Mitigation: The Quantlens universe filters out illiquid stocks. Stick to the filtered universe and do not manually force inclusion of micro-cap names.

Regime dependency 

Every strategy performs differently in different market regimes:

Regime	Favours	Struggles
Bull market (rising trend)	Momentum, trend-following	Mean-reversion short signals
Bear market (falling trend)	Short strategies, cash holding	Buy-only momentum
Sideways / choppy	Mean-reversion, range-trading	Trend-following (many false signals)
High-volatility crash	Cash or hedged positions	Most long-only strategies

A strategy optimised exclusively on a 10-year bull market (e.g. 2010–2020) will look exceptional but may collapse when the regime changes.

Best practice: Always test across multiple market regimes. Include at least one bear market period (2000–2002, 2007–2009, 2020 crash) in your test window.

Execution timing 

Quantlens executes orders at either end-of-day (eod) or start-of-day (sod). Real brokers execute differently:

Market orders fill immediately at the best available price, which may differ from the recorded OHLCV data.
End-of-day orders are submitted as “market on close” — which is a real order type, but may have a small slippage from the official close.
Limit orders (via Enter Next-Week Limit Orders) may not fill if the price never reaches the limit, or may fill at a slightly different price due to order book depth.

The backtest model assumes perfect fill at the specified price. This is a reasonable approximation for liquid instruments but less accurate for volatile or illiquid names.

Data quality 

Quantlens uses institutional-grade data providers with adjusted prices and corporate action handling. However, no dataset is perfect:

Dividend adjustments are applied retroactively, so historical prices are restated. This means the data you see today for 2010 is not identical to what you would have seen in 2010.
Split adjustments are similarly retroactive.
Occasional missing bars occur for public holidays, trading halts, or exchange errors. The backtest engine skips missing bars gracefully.

Summary: What backtests can and cannot tell you 

Backtests CAN tell you	Backtests CANNOT tell you
Whether a strategy had an edge historically	Whether that edge will persist in the future
The approximate scale of drawdowns to expect	The exact timing of the next drawdown
Whether your logic is correctly implemented	Whether the market regime will stay the same
Which parameter ranges look promising	The single “correct” parameter value
How commission and slippage affect returns	How your specific broker’s execution compares

Use backtesting as a filter — to discard ideas that clearly don’t work — not as a proof that an idea will work in the future.