In March 2026, someone on X ran a contest: two AI agents, $1,000 each, 48 hours to trade on Polymarket. One finished with $14,216. The other was liquidated. The post went viral. LLMs can trade! (I discuss the fine print the screenshots do not show in the original article.)
I took away a different lesson. The LLM was one component of the system.
It is easy to build a demo trading bot. Get an API key, write a prompt, let the model read the market, choose a side, place an order. Repeat. It works for a while. Then you hit a market with a thin order book, or a fee structure you did not model, or a resolution sentence nobody read carefully. The bot loses money, and you blame the model.
I recently built and operated a system that trades on prediction markets. The LLM was the least interesting part. The result was decided by the harness, the code around the model.
Prediction markets are an execution game
It is tempting to think of prediction markets as a forecasting competition. They are not. They are an execution competition where forecasting is one of the inputs.
Calibration. You need to output a probability, and you need to be right 70% of the time when you say 70%. LLMs are notoriously overconfident, especially in the tails. They will say 5% when it should be 20%, and the market will exploit your error. The top 1% of Polymarket traders succeed by trading small sizes, focusing on one or two categories, and exiting based on price, not the underlying outcome. None of that has to do with forecasting.
Information. The market has already priced in the obvious. You need data the market has not seen, or a correct reading of the resolution rules. The LLM does not create information. It helps you process it, and only if you can get it into the model in a useful form.
Execution. What is the price? The spread? The available liquidity? Can you actually execute on your edge at a size large enough to matter, after fees? The order book is part of the model. The weather bots on Polymarket won because of a well-oiled data pipeline and the discipline to trade small.
Risk management. You can be right 60% of the time and still go bust. Sizing and overall risk are a separate skill from forecasting.
Anatomy of a trading system
Data layer. Market feeds, order book data, resolution rules, external data. You need to replay what the market looked like when the system made a decision, or you will never debug the losses.
Model layer. Some code that outputs a probability. An LLM can do it, but so can a statistical model or an ensemble. Then a calibration layer converts raw confidence into a probability with the right frequency.
Edge. Model probability minus market probability minus fees. Only trade when the number is large enough.
Execution. Sizing, limit prices, order types, cancellation, a kill switch. Start with paper trading, graduate to real money, have a human approve large orders.
Evaluation. Backtests, live paper trading, traces of every decision, profit and loss attribution, calibration. If you cannot explain a loss, you cannot fix it.
I go into more detail on why the harness is the moat here.
Failure modes I have seen in production
The overconfident forecaster
The first version of my system used an LLM to output probabilities. It worked okay in the middle range and was overconfident in the tails. It was confident in long shots and sure things, and the market was less wrong than the model. The fix was a calibration layer trained on resolved markets. It helped more than a bigger model. An uncalibrated forecast is a liability.
The fee-blind strategy
My naive strategy appeared to make money in backtesting. Once I included fees, spread and position size, it was a loser. The fix was a fee-aware sizer and a liquidity filter. The order book is part of the model.
The thin book
I had a position that was small compared to my bankroll and large compared to the available liquidity. Getting in was easy. Getting out was impossible. The fix was a position cap tied to the book, not to the account.
None of these were model failures. They were harness failures.
The mistakes that look like progress
The hard part is not the model. It is knowing when the model is wrong.
Thinking you have a strategy when you do not
You ask an LLM for a probability, compare it with the price, place a trade. It feels like a strategy. It is a sentence with a number in it.
The model does not know the base rate for this market. It does not know the resolution rules. It does not know what the price already implies. A probability with no calibration history is a guess with a decimal point.
Leaky backtests
A backtest is a simulation of a decision, not an outcome. The most common error is temporal leakage. You read the market price at the close and trade at the open. You use a forecast published after the decision. The row knows the outcome because it was written after the market resolved.
A random train/test split makes it worse: you train on Tuesday and test on Monday. Use time-ordered folds, a locked test period, and a rule that no data after the decision timestamp enters the features.
Test set leakage
If you fit the calibration on the same period you evaluate on, the result is in-sample. If you tune the threshold on the test set, it is no longer a test set. If you try fifty variants and report the best, part of the result is luck.
Write the hypothesis first, lock the test set, report every variant, and correct for the number of attempts.
Ignoring costs
Fees, spread, slippage, gas and failed transactions. A four percent edge with a three percent spread and a one percent fee is not an edge. A backtest that fills at the mid price is a fantasy.
The resolution rule you did not read
Markets resolve on a sentence. The sentence has a source, a time zone and edge cases. If the model prices the headline and not the rule, it is trading a different contract.
None of these mistakes look like mistakes in a notebook. They look like progress.
The unglamorous 80%
In the systems I have built, the model was maybe twenty percent of the work. The rest was plumbing, checks and tools that never appear in a demo.
Data infrastructure. Several APIs, each with its own rate limit, schema and downtime. Normalization, time alignment, snapshots you can replay. Backfills, schema changes, a vendor that renames a field at 2 a.m.
Market resolution. Reading the rules, mapping the source, handling ambiguity, disputes, voided markets and settlement delays. The rule decides whether the position was right.
Internal oracles. Three views of the same position: the local ledger, the Polymarket API and the on-chain state. The oracle compares them. When they disagree, it applies a rule: the chain wins on settlement, the API wins on open orders, and everything else goes to a human with a ticket. Without this, the profit and loss number is a guess.
Reconciliation. Fills and balances are reconciled after every order. Idempotency and retries matter: an order sent twice is a different position.
Tests. Unit tests for the math, replay tests for the strategy, contract tests for the API, failure injection for the feed. Paper trading through the same execution path, not a separate script.
Internal tools. A market browser, a decision log, a calibration dashboard, a profit and loss attribution view, an alert console. None of this is customer-facing. All of it is the difference between a demo and a system.
The model is the part you can buy. The infrastructure is the part you have to build.
Beating the market
The market price is a forecast. A good one, made by people with money at stake. Your model has to beat it, not the headline.
Start with the baseline: the market price, the base rate, a simple model. Report the baseline next to your strategy. If the model cannot beat it after costs, on a locked test set, on the markets you would actually trade, it is not an edge. It is a story.
The test is not whether the model can explain the market. It is whether it can beat the price, out of sample, after fees and spread.
Start with an edge scanner
The first system you should build does not trade. It scans.
It ingests markets and resolution rules, builds a probability with evidence, compares that probability with the market price, and ranks the gaps. An edge scanner places no orders. It tells you whether a market is worth building a trading system for.
The order I would build the components:
- Data snapshots you can replay.
- A model that produces a calibrated probability, with a reliability curve.
- The edge scanner.
- Paper trading with the same execution code you will use with real money.
- Real money with limits, human approval and a kill switch.
- Monitoring, alerting and a daily brief.
If you are building
You do not need a better LLM. You need a system that can test, size, execute and explain. The model is a commodity. The harness is the moat.
Whether you are building a prediction market product, a trading desk or a research agent, the work is the same:
- Data pipelines that produce clean data.
- A model that produces a calibrated number.
- An execution path with limits.
- An evaluation loop that tells you the truth.
The model is the easiest component to buy and the hardest to trust.
P.S. If you are building a prediction market system and want advice on the harness, write to me or see how we build these systems.
