Projects / Built · Case study
Forecasting next-day electricity prices on the IEX
Every morning, within 30 to 60 minutes of the exchange publishing yesterday’s results, a trader needs tomorrow’s price for each of 96 fifteen-minute blocks. This is how I got a model to produce them from about a year and a half of data.
- Context
- Day-ahead price forecasting for the power traders of a large Indian energy group (client name withheld)
- My role
- Led the data science in a team of three: analysis, features, models and evaluation. Two teammates built ingestion, backend and frontend
- Stack
- Python, TensorFlow, Keras, Scikit-Learn, TabNet, Pandas, Django REST Framework, Celery, Redis, MySQL, Docker
- Data
- IEX auction data, regional load and transmission, and weather forecasts: about 1.5 years of history from April 2022
The job
On the Indian Energy Exchange, electricity for tomorrow is sold in an auction, one for each 15-minute block of the day, 96 in all. Buyers and sellers bid a day ahead and the price where supply meets demand, the market clearing price (MCP), is what everyone pays. It changes from block to block through the day. A trader who knows roughly where tomorrow’s curve will land bids with confidence. One who doesn’t is guessing.
The clock sets the shape of the problem. IEX publishes the previous day’s results around 9:30 AM, and the report is due at 10. Whatever the model needs has to be known by then, and the whole pipeline has to run in the gap.
Why it was hard
- Little data. Block-level history started in April 2022, so I trained on about a year and a half. That is too short for a model to learn seasonality by itself.
- 96 different problems. Overnight, the morning ramp and the evening peak behave differently, because people behave differently. One model has to be right at all of them.
- Prices that hit a ceiling. On some mornings and evenings the price sits pinned at the exchange’s cap, which makes a hit-or-miss tolerance easy to satisfy there and worth reading carefully.
- Nothing from today. The forecast for tomorrow can only use data up to yesterday, plus tomorrow’s weather forecast. Day D’s own results do not exist yet.
Result
A prediction counts as correct when it lands within ±10% of the actual MCP. The best single slot reached about 81%. I have no baseline comparison, such as a same-block-last-week forecast, and no business-impact figure like trading profit, so the number says how close the forecasts were, not how much they earned.
Turning a time series into a table
The obvious framing is a time series: one long line of prices to extend. I flattened it instead. Each row is one block of the day being forecast, with everything the model may know about it as columns, and one target column: that block’s MCP. There are no dates. Every feature is expressed relative to the target day, and a block index from 1 to 96 says where in the day the row sits.
block price_D-1 price_D-2 ... price_D-7 slope_7d load_spread temp_D+1 | target: MCP
29 ... ... ... ... ... ... ... | ?Illustrative layout, not real data.
That one change gave a single training set for all 96 slots and 96 times as many rows as one row per day, which matters when you only have a year and a half.
Feature engineering: squeezing a small dataset
With so little history, the lever was not more data but more signal from the same data. From a handful of raw series I derived the feature set below. Every column uses only information available before the 10 AM deadline.
| Family | Examples | What it captures |
|---|---|---|
| Lags | Price, purchase bids, sell bids and cleared volume for each of the last 7 days | Recent level and weekly rhythm |
| Supply-demand curve | Aggregate demand and supply per price band, per block | How steep the market is near the clearing price |
| Rolling statistics | Means, variances, min, max and median over the lag window | How volatile the recent past has been |
| Trend | Slope of the 7 daily prices; yesterday relative to a week ago and to the 7-day mean; day-over-day change; trend of the bid-minus-offer gap | Direction and momentum |
| Cross-region | Spread of regional load; each region’s share of national load; net transmission between regions; load as a share of transmission capacity | Congestion, the thing that makes one region’s price diverge |
| Weather | Next-day temperature and rainfall forecasts for the highest-consumption city in each of five regions | Demand, which follows heat and rain |
The 7-day window came from an autocorrelation analysis: correlation with the past decayed when I tested longer windows, so extra lags would have added columns and noise on a short history. I did not run a feature-by-feature ablation, so I cannot say which family contributed most.
Predict the multiplier, not the price
The recurrent network could not take prices on their raw scale, and standard normalisation assumes a stable range that these prices do not have. So every price from D-7 to D-1 is expressed as a multiple of the D-7 price, with D-7 equal to 1, and the model predicts tomorrow’s price as a multiplier of the same base. A price of 6 against a base of 4 becomes 1.5. Inputs and output then stay in a narrow band even when the absolute price level moves.
The catch. If the D-7 day was a spike, every ratio in that row is distorted. I accepted that trade for stable scales.
Three models, each best at a different hour
Consumption follows human routine, and routine differs by time of day. So I expected no single kind of model to be best everywhere, and built three families that see the problem differently:
- Random forest on the whole table, for general patterns shared across days and slots.
- 96 TabNet models, one per block, a neural network for tabular data that picks features with attention. Each learns the quirks of its own time of day.
- A two-branch GRU. One branch reads the price sequence from D-7 to D-1 in order, the other takes weather and holiday flags through a dense layer, and the two are joined to predict the multiplier.
Then I measured how much each family mattered in each of the 96 slots. The importance split really did vary across the day, matching the hypothesis. Those importances became the weights: for every slot, the final forecast is a weighted average of the three predictions using that slot’s weights.
The daily pipeline
What I tried and set aside
- Longer lag windows. Tested and dropped: correlation fell off beyond seven days.
- Gradient boosting. I compared it against the random forest, and the forest performed better.
- Final scheduled volume. A strong-looking field, but IEX publishes it two days late, so a 10 AM forecast cannot use it. I left it out rather than train on something unavailable at prediction time.
- Raw scales for the network. Replaced with the ratio scheme above. The random forest did not need it, since trees split on order and not magnitude.
My part
I led the data science in a team of three: the analysis, features, models and evaluation, end to end. Two teammates built the data ingestion, the backend (Django REST Framework, Celery and Redis scheduling, MySQL, Docker) and the frontend that delivered the report. My constraint on their side was the clock: features and models had to be cheap enough to run inside the 30-to-60-minute window.
Limits
- I trained on about a year and a half of history, which is still short for learning seasonality. A fixed 7-day window cannot see monthly or seasonal cycles.
- In some morning and evening windows the market price sits pinned at the exchange’s price cap. A ±10% tolerance is easy to hit when a forecast also sits near the cap, so accuracy in those hours may read differently from the rest of the day.
- With one row per day per block, each of the 96 TabNet models trained on roughly 550 rows, and independent benchmarks find tree models often still win on small tabular data. I did compare against gradient boosting and the random forest performed better, so the TabNet models earn their place through the diversity they add to the blend, not through beating trees on their own.