Bottom line
No local model has yet earned a trading role; one thin Qwen signal on mid-cap stocks is the only lead left.
Do this next
- Run H5's next gate on the mid-cap lead. The pre-registration's order is: validation book net of 20 bps, then the Qwen2.5 robustness check, then one test look.
- Commit and hash the freezes. H5 and H6 still carry the placeholder sha
abc1234, anddatagate_runs.jsonlandh5_run.pyare untracked. - Start the nightly archiver. Forward headlines and scores cannot be bought back, and the text roles only count on days after the cutoff.
What this report adds
- One timeline built from git history, run logs and file dates, which no single repo document has.
- One ledger of 15 attempts with the number that ended each.
- An honest "not proved" list, separating no edge found from no edge exists.
Decisions only you can make
- Whether to open the H6 test window (vol-sized SPY). Validation gave +0.05 [-0.24, +0.34], so the likely outcome is "inconclusive".
- Whether to freeze
PREREGISTRATION_H3_DRAFT.mdso Laya's event reader can be judged on forward days. - Whether to freeze the other drafts for futures, commodities and FX. See open, closed, later.
Context
The cast and the method
It began as a paper-trading bot for SPY and QQQ and became a research program asking one question: can a small local model add predictive power that a plain regression cannot?
Every attempt was held to the same method. A hypothesis and a variant budget are written down first. Data is split into train, validation and test by date. Any new data goes to a ridge logistic before a model sees it (bot/datagate.py), and a result counts only if its 95% day-block interval excludes zero.
The local models
- Laya: a ModernBERT encoder with a typed head, served locally by
laya-serve. Checkpointtyped-decisionsis pinned; fine-tunes areft1,ft2a/b/c,laya-tb,stocks_ftandfx_ft. - Qwen3-4B-Instruct-2507: a 4-bit model run through Ollama on a 4 GB GPU. It fills fixed fields per headline (battery
tx1). - Qwen fine-tunes
d1andd2: adapters trained on headline direction.
Not local, and why it is here
- TypeSafe's Jev gated the first replay. It was removed when the project committed to Laya, so reports call it "the earlier model".
- The ridge logistic is not a local AI model, but it is the yardstick every model had to beat.
Timeline
Nine days, eight phases
Times come from run logs and file dates, in the PC's local time. Commits have dates only. The azure phase marks work that is still open.
Build the bot and pick an idea
Build
- Analytics dashboard with a roadmap view, bot modules, tests and config.
- SPY/QQQ intraday momentum philosophy drafted.
Research
- Cost-wall study: crypto and market making lose to fees (30–52 bps round trip against 12–22 bps moves).
- Chose momentum on SPY/QQQ, with Laya as a gate only.
By end of day: a bot that can observe and replay, and a first model role: gate the entry rule.
Rule, replay and the first verdict
Build
- Baseline rule, risk engine, decision ladder, replay simulator, review agent, main loop, order manager and chaos tests.
Results
- 05:44: one-change screen, 12 variants. None positive; the rule's gross was -0.04 bps per trade.
- 06:27: H1 frozen. 06:50–07:31: Phase 6 replay with the earlier hosted model as gate. Stress result -5.13 bps/trade on train, -3.16 on validation.
By midday: the rule had nothing for a gate to save. The earlier model was dropped the same day.
Laya takes over and is fine-tuned
Overlay tests
- 09:51–15:33: H2, hold SPY and step aside on days the model rates disorderly. 33 variants, all worse than buy-and-hold.
- 15:33: H2-L, same overlay with a logistic gate: -4.31 bps/day against buy-and-hold.
Laya training
- Fine-tunes
ft1,ft2a,ft2b,ft2c: calibrated, no skill. - Stocks (4,399 rows, 12 epochs) and FX checkpoints trained: no measured edge.
Live
- One partial live session, 12:29–15:59 ET: 422 snapshots, 0 model calls, 0 orders.
- A PC clock-skew bug stopped the collector; a guard was added.
By evening: five strategy families and six schools all negative on validation; SIP and spread checks done.
Laya's roles are redesigned
Gates
- 06:10:
contextandnewsdata gates both FAIL. - 06:49: roles redesigned. The logistic judges numbers; Laya reads text and ties.
Tie-breaker
- 07:42–08:50: tie-break replay on validation, worse than the gate alone.
- 09:22: stack test and nonlinear ceiling FAIL. The tie-breaker was retired the day it was built.
Reorganise
- Indexes concluded: no edge found, not proved none.
- Repo split into five systems. Stocks daily Phase 1 (S&P 500, 2016–2026) measured the cost wall.
By noon: the numeric role of local AI was closed and the project turned to text and to single stocks.
Daily scenarios and the Qwen build
Daily scenarios
- Slow runners built for indexes, futures, commodities and FX. All tested on synthetic data only; real-data runners refuse until a pre-registration is frozen.
- 22:57–23:04: H6 frozen and its validation look run five times with identical output.
Qwen
- 12:45: news scorer written. 17:03: Benzinga headline set built for fine-tuning. 18:50: fine-tune smoke test.
By midnight: H5 (Qwen as headline reader) frozen on paper; scoring and fine-tuning ready to run.
Qwen fine-tunes finish
Training
d1(375 steps, about 2.1 hours) saved 01:55.d2(2 epochs, 1,000 steps) saved 08:03.- Both learned the public task and neither added signal on Benzinga headlines.
By midday: fine-tuning Qwen on headline direction was a dead end at this data size.
The H5 gate runs
Scoring
- 02:24: scoring finished. 87,432 stories across 528 tickers, 0 invalid answers.
Gate
- 09:04: V1 (S&P 500) FAIL.
- 09:45: V3 (mid-cap, 20 bps) PASS, narrowly.
By midday: one lead, two of three variant slots spent, and the next gate not yet run.
Open work
Next
- No run is logged after that in the files read.
- Pending: V3 validation book, V2 robustness, one test look, then a 120-day forward test.
Open: nothing has been tested on a test window yet.
Ledger
Every attempt, and how it ended
Red means the attempt failed on evidence. Blue means retired, untested or inconclusive. Green means it worked or is a live lead. Intervals are 95% day-block bootstrap unless stated.
| Attempt | Status | What happened |
|---|---|---|
| 1. Earlier hosted model as the entry gate | Retired | All five Phase 6 pass conditions failed or were inconclusive. Deflated Sharpe 0.0. Removed from code and docs the same day. |
2. Laya typed-decisions reading 17 snapshot numbers | Failed | Brier skill below zero on all four questions (-0.403, -1.146, -0.236, -1.066 on validation); calibration error 0.165–0.484. Its answers sat near 25% when the truth was 55%. |
3. Laya fine-tunes ft1, ft2a/b/c | Failed | Calibrated but no skill. ft2a collapsed to the class prior; 36 times more rows (ft2c) removed the collapse and plateaued at ft1's level. Regime AUC 0.627–0.685 against the logistic's 0.925. |
| 4. Laya as a veto in the strategy "schools" | Inconclusive | Momentum under a Laya veto went from -0.31 bps/day on train to +0.46 on validation (interval [-6.77, +12.28], 99 trades). A sign flip is what a gate fitted on a 0.02-wide answer band looks like. |
| 5. H2: Laya gate to step aside on disorderly days | Failed | -1.73 bps/day against buy-and-hold, interval [-5.5, +2.17]. Flagged days did better than kept days (+2.78 against +2.20 bps), which tripped H2's own kill rule. |
| 6. Plain ridge logistic on the same numbers | Worked | Regime AUC 0.925, liquidity 0.851. The edge is one feature, vol_vs_typical: volatility clustering, which says nothing about direction. Used as the numeric judge. |
| 7. H2-L: the logistic gate as an overlay | Failed | -4.31 bps/day against buy-and-hold, interval [-7.63, -0.95]. Flagged days rose +13.37 bps against -0.42 on kept days, so stepping aside gave up return. |
8. Laya tie-breaker (laya-tb) | Retired | Better calibrated (error 0.31 to 0.04) but worse at ranking inside the tie band: AUC 0.35 and 0.42 against the logistic's 0.78 and 0.73. Replay cost 0.76 bps/trade against the gate alone. |
| 9. Stacking Laya answers beside the logistic; boosted trees | Failed | Adding ft1, ft2c or laya-tb lowered validation AUC (disorderly -0.020 to -0.030, liquidity -0.054 to -0.068). Trees and interactions did not beat the ridge. The 12 snapshot numbers hold nothing more. |
| 10. New data kinds for the logistic: context and headline counts | Failed | context: AUC 0.926 to 0.907, interval [-0.052, +0.008]. news (40,273 headlines): 0.840 to 0.852, interval [-0.012, +0.038]. |
| 11. Stocks and FX Laya checkpoints | Failed | Stocks: 2,018 validation moments, calibration error 0.02–0.06, Brier skill -0.008 to +0.021. FX: 189 signals, Brier skill -0.031 to +0.007. Calibrated priors with no measured edge. |
| 12. Laya event reader and auditor | Untested | Event reader is off; its pre-registration (H3) is still a draft and only forward days count. The auditor is usable now because nothing rides on it, and its two answers are uncalibrated. |
13. Qwen3 fine-tunes d1, d2 on Benzinga headlines | Failed | Public test accuracy rose 0.708 to 0.878, but Benzinga test AUC went 0.493 to 0.497. Difference +0.004, interval [-0.017, +0.023]. d2 validation AUC 0.5140 against base 0.5166. |
| 14. H5 V1: Qwen3 fields, S&P 500, 5-day label | Failed | AUC gain +0.0014, interval [-0.0054, +0.0073], 51,235 validation rows over 102 days. The 3-day (+0.0026) and 1-day (+0.0083) gains also include zero. |
| 15. H5 V3: same fields, mid-cap tercile, 20 bps | Lead | 5-day gain +0.0118, interval [+0.0001, +0.0228], 17,082 rows. The 3-day and 1-day intervals include zero. It passes the gate by a hair and has not faced the book test. |
Measurement
The numbers behind the verdicts
Validation AUC by question, 262 moments on SPY and QQQ. 0.5 is a coin flip. The two questions that carry signal are about volatility and volume, not direction.
| Question | Ridge logistic | typed-decisions | ft1 | ft2c |
|---|---|---|---|---|
| regime (disorderly) | 0.925 | 0.631 | 0.685 | 0.627 |
| liquidity_stressed | 0.851 | 0.564 | 0.702 | 0.657 |
| follow_through | 0.466 | 0.523 | 0.519 | 0.533 |
| move_beats_cost | 0.639 | 0.527 | 0.553 | 0.574 |
The intraday rule, stress cost 4 bps round trip. Buy-and-hold SPY made +2.24 bps/day on train and +4.50 on validation.
| System | Window | Trades | Gross bps | Net stress bps |
|---|---|---|---|---|
| Rule alone | train | 562 | -0.04 | -4.04 |
| Rule alone | validation | 262 | +0.54 | -3.46 |
| Rule plus earlier-model gate | train | 262 | -1.13 | -5.13 |
| Rule plus earlier-model gate | validation | 117 | +0.84 | -3.16 |
| Rule plus numeric gate | validation | n/a | n/a | -2.50 |
| Numeric gate plus Laya tie-break | validation | n/a | n/a | -3.27 |
The Qwen headline reader on a 5-day excess-return label, validation.
| Variant | Base AUC | With Qwen | Gain | 95% interval | Verdict |
|---|---|---|---|---|---|
| V1, S&P 500 | 0.4844 | 0.4857 | +0.0014 | [-0.0054, +0.0073] | FAIL |
| V3, mid-cap | 0.4700 | 0.4818 | +0.0118 | [+0.0001, +0.0228] | PASS |
Disposition
What is open, what is closed, what waits
Each "later" carries the trigger that starts it.
| Direction | Verdict | Why, and the trigger |
|---|---|---|
| H5 V3: Qwen fields, mid-cap book test | Now | The only live lead. Next gate: top-decile book net of 20 bps must beat SPY and the equal-weight universe. |
| Laya event reader (H3) | Later | Needs a frozen H3 and forward days. Trigger: you freeze the draft. |
| Laya auditor | Optional | No lookahead risk and no money rides on it. It guards against bad prints (407 masked on FX). |
| H6 vol-sized SPY, test window | Later | No model involved. Validation does not exclude zero, and 407 test days will probably be inconclusive. Trigger: you decide a probable "inconclusive" is worth the one look. |
| Futures, commodities and FX daily scenarios | Later | Built, unrun, drafts unfrozen. The Qwen-dependent ones wait on the H5 book result; the numeric ones can start once you freeze. |
| Real-fill measurement on SPY/QQQ | Optional | Every cost claim is assumed (2–4 bps) while the quoted spread is 0.25–0.32 bps. A few weeks of small paper fills would settle it. |
| Laya reading snapshot numbers; Laya tie-break | Closed | The logistic beats every checkpoint, and stacking made it worse. |
| SPY/QQQ 1-minute momentum and its variants | Closed | About 96 variants, none positive net. Do not re-test. |
| Qwen fine-tuning on headline direction; H5 V1 on the S&P 500 | Closed | No gain on Benzinga test; V1's interval includes zero at every horizon. |
Judgment
Rules and risks
The kill rules below come from the pre-registrations. Read them before changing a threshold.
Ifthe V3 validation book is not above zero after 20 bps, or does not beat the equal-weight universe.
ThenH5 ends as "no text edge found". The next idea is a new pre-registration.
Ifany scored day falls on or before a model's cutoff.
Thenthe run is void.
Ifthe book is positive gross but not net.
Thenlengthen the hold once; never lower the cost.
Ifthe forward test is below -200 bps cumulative net excess after day 60.
Thenabandon it early. Do not declare a win early either.
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| The V3 pass is luck. Its lower bound is +0.0001, and two of three Qwen variant slots are spent. | High | High | Run the book test and the Qwen2.5 check before believing it. Do not add variants. |
| Multiple testing. About 96 SPY/QQQ variants against a budget of 5. | High | Medium | Closed. The deflated Sharpe was 0.0 to 0.07 against a 0.95 bar. |
| Freeze provenance. H5 and H6 name a placeholder sha. The first screen ran at 05:44 and H1 froze at 06:27. | Medium | Medium | Commit the freeze files and record the real sha. |
| H6's written rule says stop only if the stressed Sharpe difference is at or below zero; the commit note stops on the lower bound. | Medium | Low | Decide which reading binds before any test look. |
| Thin evidence. Qwen has 14 months of honest history, and the test window is a fraction of it. | High | Medium | Expect "inconclusive" and rely on the forward test. |
| Data and costs. IEX carries 2–4% of volume; costs were assumed; the S&P 500 membership list is unaudited. | Medium | Medium | Measure fills; audit the membership file. |
Uncommitted work. datagate_runs.jsonl and h5_run.py are untracked. | High | Medium | Commit them. This holds the only gate evidence. |
Answer
What should happen next?
Run the one remaining gate on the mid-cap Qwen lead, and treat every other local-model role as closed until forward days exist.
Local models have not beaten a regression on numbers anywhere, so the open question is text on single stocks over days. If the V3 book test fails, H5 ends and the project's honest result is that no local model added tradable information. If it passes, the next evidence is the Qwen2.5 check, one test look, and 120 forward days.
References
Sources
Everything below was read from the repo; no backtest was re-run. The Qwen gate figures were read straight from datagate_runs.jsonl, and the story count came from the score file. Indexes figures, the Laya AUC table and the H1–H2 results are carried over from Indexes_Conclusion.md, and the Daily Phase 1 and win-score figures from their own reports. Times are from run logs and file dates.
Conclusions and plans
Pre-registrations
Results and run logs
History
- Git log of the repo, 21 commits (
git log --reverse) - File modification dates for Qwen scores, adapters and reports