Backtest overfitting and Optuna

Estimate selection overfitting with PBO, or record a search for the deflated-Sharpe calculation. The Optuna integration is optional; install purgedcv[optuna] to run an Optuna study.

Backtest overfitting

purgedcv.PBOResult dataclass

PBOResult(pbo: float, logits: NDArrayAny, slope: float, is_oos_performance: NDArrayAny, n_combos: int)

Return type of probability_of_backtest_overfitting.

A frozen dataclass: read fields by attribute (result.pbo), and call dataclasses.asdict if you need a plain dict to serialise.

Attributes:

Name	Type	Description
pbo	float	Probability of backtest overfitting in [0, 1]. The fraction of CSCV combinations whose in-sample-best configuration ranked below the OOS median (equivalently, logit < 0). Read it as 0.0 = selection never overfits, 0.5 = selection is no better than chance, 1.0 = the in-sample best is always an out-of-sample loser. Anything materially above 0.5 means the search is mostly fitting noise.
logits	NDArrayAny	Per-combination logit of the OOS relative rank of the in-sample-best configuration. Negative values are the overfit cases; the distribution's mass below 0 is pbo.
slope	float	OLS slope of OOS performance on IS performance across the combinations (for the IS-best configuration). Direction is the signal: > 0 means in-sample strength carries over (no overfit), ~ 0 means the link is random, and < 0 means in-sample strength predicts out-of-sample weakness (severe overfit). A positive slope below 1 still indicates some decay.
is_oos_performance	NDArrayAny	(n_combos, 2) array; column 0 is the IS performance of the IS-best configuration, column 1 its OOS performance, one row per combination.
n_combos	int	Number of CSCV combinations scored, C(n_splits, n_splits // 2) minus any folds dropped for an empty IS set.

purgedcv.PerformanceMetric module-attribute

PerformanceMetric = Callable[[NDArrayAny], float]

A configuration-performance metric: maps a 1-D return slice to a scalar where larger is better.

Pass this callback as metric to probability_of_backtest_overfitting to score and rank candidate configurations. The default is a Sharpe ratio computed as mean return divided by sample standard deviation. Negate a loss if smaller values would otherwise mean better performance.

purgedcv.probability_of_backtest_overfitting

probability_of_backtest_overfitting(returns: NDArrayAny, n_splits: int = 16, *, metric: PerformanceMetric = sharpe, prediction_times: TimesLike | None = None, evaluation_times: TimesLike | None = None, purge_horizon: HorizonLike | None = None, embargo: HorizonLike | None = None, embargo_observations: int | None = None, embargo_fraction: float | None = None) -> PBOResult

Estimate the Probability of Backtest Overfitting (PBO) via CSCV.

Given the per-period returns of n_configs candidate configurations, PBO is the probability that the configuration selected as best in-sample performs below the median out-of-sample. The estimate is the relative frequency of that event across all CSCV combinations.

Parameters:

Name	Type	Description	Default
returns	NDArrayAny	(n_configs, n_obs) array; returns[j] is the per- period return series of configuration j over a shared time axis. At least 2 configurations and n_obs >= n_splits are required; all values must be finite.	required
n_splits	int	Number of contiguous blocks the time axis is cut into (S in CSCV). Must be even and >= 2; the number of combinations is C(n_splits, n_splits // 2). The standard choice is 16.	16
metric	PerformanceMetric	Performance metric mapping a 1-D return slice to a scalar (larger is better). Defaults to sharpe.	sharpe
prediction_times	TimesLike | None	Optional per-period prediction times. Supply with evaluation_times to enable purge/embargo at IS/OOS boundaries. Length must equal n_obs.	None
evaluation_times	TimesLike | None	Optional per-period evaluation times; see prediction_times.	None
purge_horizon	HorizonLike | None	Optional purge horizon (requires the time series).	None
embargo	HorizonLike | None	Optional embargo horizon (requires the time series).	None
embargo_observations	int | None	Optional number of row positions to embargo after each OOS block (requires the time series).	None
embargo_fraction	float | None	Optional fraction of rows to embargo after each OOS block (requires the time series).	None

Returns:

Type	Description
PBOResult	A PBOResult (frozen dataclass; read fields by attribute,
PBOResult	e.g. result.pbo, result.slope).

Raises:

Type	Description
ValueError	on a malformed returns matrix, an odd or too-small n_splits, a time series whose length disagrees with n_obs, or purge/embargo requested without times.

Examples:

>>> import numpy as np
>>> from purgedcv import probability_of_backtest_overfitting
>>> rng = np.random.default_rng(0)
>>> returns = rng.standard_normal((10, 240))  # 10 configs, pure noise
>>> result = probability_of_backtest_overfitting(returns, n_splits=8)
>>> 0.0 <= result.pbo <= 1.0
True
>>> result.n_combos
70

Optuna integration

purgedcv.optuna_integration.TrialSharpeRecorder

TrialSharpeRecorder(attr: str = 'sharpe')

Optuna study callback that collects per-trial Sharpe ratios.

Pass an instance as a callback to study.optimize and store each trial's Sharpe in a user attribute (default key "sharpe"); the recorder accumulates them and reports the variance for deflated_sharpe_ratio. If a trial has no such user attribute, the recorder falls back to that trial's objective value. Non-finite and missing values are ignored.

Parameters:

Name	Type	Description	Default
attr	str	The trial.user_attrs key under which the Sharpe ratio is stored. Defaults to "sharpe".	'sharpe'

Examples:

>>> from types import SimpleNamespace
>>> from purgedcv.optuna_integration import TrialSharpeRecorder
>>> recorder = TrialSharpeRecorder()
>>> # Optuna calls the recorder with (study, frozen_trial) per trial;
>>> # SimpleNamespace stands in for the trial here.
>>> for s in (1.2, 0.8, 1.5):
...     recorder(study=None, trial=SimpleNamespace(value=s, user_attrs={"sharpe": s}))
>>> recorder.n_trials()
3
>>> round(recorder.var_sharpe(), 4)
0.1233

Usage with a real study::

recorder = TrialSharpeRecorder()
study.optimize(objective, n_trials=300, callbacks=[recorder])
dsr = deflated_sharpe_ratio(
    best_returns, n_trials=recorder.n_trials(), var_sharpe=recorder.var_sharpe()
)

__call__

__call__(study: object, trial: _TrialLike) -> None

Record one trial's Sharpe ratio. Matches the Optuna callback signature.

Missing, None, non-numeric, and non-finite values are skipped rather than raised: a callback that throws would abort the whole study.optimize run over a single malformed trial.

sharpes

sharpes() -> NDArrayAny

The recorded Sharpe ratios, in trial order, as a 1-D array.

n_trials

n_trials() -> int

Number of trials with a usable (finite) Sharpe ratio recorded.

n_effective

n_effective(method: str = 'autocorr') -> int

Effective number of independent trials, for correlated samplers.

TPE and CMA-ES draw each trial conditioned on the previous ones, so the raw trial count overstates the independent search effort and makes deflated_sharpe_ratio overly conservative. This applies effective_n_trials to the recorded Sharpe series; pass the result as n_trials to deflate more realistically. Returns 0 when nothing has been recorded yet. See effective_n_trials for the heuristic's caveats.

var_sharpe

var_sharpe(ddof: int = 1) -> float

Variance of the recorded Sharpe ratios, for deflated_sharpe_ratio.

Returns nan until more than ddof trials have been recorded.

UNITS: the variance is in the units of whatever Sharpe you stored per trial. deflated_sharpe_ratio compares against the per-period Sharpe of its returns argument, so record a per-period Sharpe in each trial (the same convention as the returns you will later deflate). Do that and the recorder feeds deflated_sharpe_ratio directly, with no unit conversion.