Repository navigation
Add return_samples: expose the full Monte-Carlo distribution instead of averaging it away - #395
Open
brodsky754 wants to merge 1 commit into
Open
brodsky754 wants to merge 1 commit into
brodsky754 wants to merge 1 commit into
Conversation
auto_regressive_inference averages the sample_count paths before returning (np.mean(preds, axis=1)), so the predictive distribution is destroyed before any caller can see it: quantile cones, P(up), and sample-agreement statistics cannot be computed from the public API, and statistics of the averaged path systematically understate uncertainty. return_samples=False everywhere keeps behaviour byte-identical. With return_samples=True, predict returns one DataFrame per sampled path and predict_batch a list of the same per series. Motivation: running Kronos-small live and grading 249 directional forecasts, probabilities derived from path agreement (via a batch-replication workaround) stated 93% average confidence against a 47.8% realized hit rate (Brier 0.45). Exposing the raw paths lets users sample uncertainty with top_p=1.0 and calibrate agreement against realized outcomes, per the new docstring note.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
auto_regressive_inferenceaverages thesample_countpaths before returning (preds = np.mean(preds, axis=1)), so the predictive distribution is destroyed before any caller can see it. Quantile cones, P(up), and sample-agreement statistics cannot be computed from the public API — and statistics computed on the averaged path systematically understate uncertainty.Change
Adds
return_samples=Falsethreaded throughauto_regressive_inference→generate→predict/predict_batch:False: behavior byte-identical to today;predict(..., return_samples=True)→List[pd.DataFrame], one per sampled path;predict_batch(..., return_samples=True)→List[List[pd.DataFrame]], one list per series.Docstrings note the two uncertainty pitfalls we hit in practice: truncated sampling (
top_p < 1) narrows every path so probabilities should be sampled attop_p=1.0, and sample agreement should be calibrated against realized outcomes before being trusted.Motivation (real-world data)
We run Kronos-small live on hourly crypto klines and grade every forecast against realized returns. Deriving P(up) from path agreement (via a batch-replication workaround for the averaging) produced severe miscalibration: stated confidence 0.93 average vs realized hit rate 0.478 over 249 graded directional forecasts (Brier 0.45; the 184 forecasts claiming >90% landed at 0.505). Part of that is the truncated-sampling effect the docstring now warns about — but users can neither see nor fix any of it while the API averages the distribution away. This change makes honest uncertainty estimation possible without touching the model.
🤖 Generated with Claude Code