Fix ML predictions missing from report page by pookey · Pull Request #36 · johanzander/bess-manager

pookey · 2026-03-07T15:44:55Z

Summary

ML predictions were missing from the ML Report tab because the 23:55 next-day preparation wiped the cache populated at 23:00
Moved prediction generation from 23:00 to 23:55 (after cache clear) so predictions survive into the new day
The 23:00 cron now only retrains the model; predictions are generated at 23:55 alongside other next-day prep

Test plan

Verify ML Report page shows prediction line after overnight schedule run
Confirm model retrains at 23:00 and predictions appear after 23:55
Check startup still generates both retrain + predictions correctly

🤖 Generated with Claude Code

Extend the price system to support Octopus Energy alongside Nordpool. Octopus provides separate import/export rates at 30-minute resolution via HA event entities, with VAT-inclusive GBP/kWh prices. Key changes: - Add period_duration_hours and get_sell_prices_for_date() to PriceSource - Create OctopusEnergySource for fetching 48 half-hourly import/export rates - Add price_provider config field ("nordpool", "nordpool_official", "octopus") - Fix optimization path to use pre-calculated sell prices from price entries - Pass period_duration_hours through to DP algorithm - Generalize period count validation for different resolutions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Octopus Energy Agile tariff price support

Add Octopus Energy as a supported price source in README, Installation Guide, and User Guide. Include setup instructions for UK users with entity ID discovery, electricity_price adjustments, and GBP currency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Update docs for Octopus Energy support

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Bump version to 6.1.0

Fix get_tomorrow_prices() catching ValueError but not PriceDataUnavailableError, which meant the "return empty list" fallback never fired for Octopus Energy. Relax Octopus rate count validation to accept 46-48 rates instead of exactly 48. Octopus Energy publishes rates incrementally and the last couple of half-hours may arrive slightly later. 46 rates covers through 23:00, sufficient for optimization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Octopus Energy Agile tariff price support

Promise.all rejected entirely when /api/dashboard returned HTTP 500 (which happens when no optimization schedule exists), discarding all 4 fetch results and leaving every state variable as null. Switched to Promise.allSettled so each fetch succeeds or fails independently. Working endpoints (inverter status, schedule, battery settings) now populate their data even when dashboard fails. Bump version to 6.1.2.

…d-failure Fix Inverter page blank when dashboard endpoint fails

The system uses 15-minute periods internally (96/day) but Octopus Energy returns 48 half-hourly prices. When _gather_optimization_data received period_count=48 from len(prices), it created 48-element arrays, but the current_period index (e.g. 90) is a 15-minute index — causing "list assignment index out of range" and preventing any optimization. Normalize non-quarterly prices to 15-minute resolution in _get_price_data() by repeating each entry to fill the corresponding 15-min slots. The DP algorithm now always runs with period_duration_hours=0.25. Bump version to 6.1.3.

Fix Octopus Energy optimization crash due to period index mismatch

InfluxDB 1.x stores the unit of measurement in _measurement and the entity ID in an entity_id tag, while 2.x stores the entity ID directly in _measurement. Query filters now match on either field. CSV response parsers detect column positions from the header row instead of using hardcoded indices, preventing silent data loss when columns appear in a different order. Bump version to 6.2.0.

Fix InfluxDB queries for dual 1.x/2.x support

…ual rates Two bugs caused incorrect Octopus Energy pricing: 1. PriceManager retained Swedish default parameters (markup=0.08, VAT=1.25, additional=1.03) because update_settings() updated the PriceSettings dataclass but never propagated values to the running PriceManager. Formula: (0.1558 + 0.08) * 1.25 + 1.03 = 1.3248 matched the displayed ~1.33. Fix: propagate all pricing fields and clear the price cache. 2. Octopus 30-min rates (48/day) were not expanded to 15-min quarterly resolution (96/day) at the source level. The previous BSM-level normalization duplicated price-entry dicts with wrong timestamps. Fix: expand in OctopusEnergySource._fetch_rates() so PriceManager generates correct 15-minute timestamps for all 96 periods. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix Octopus Energy price display and period resolution

Promise.all rejected entirely when /api/dashboard returned HTTP 500 (which happens when no optimization schedule exists), discarding all 4 fetch results and leaving every state variable as null. Switched to Promise.allSettled so each fetch succeeds or fails independently. Working endpoints (inverter status, schedule, battery settings) now populate their data even when dashboard fails. Bump version to 6.1.2.

The system uses 15-minute periods internally (96/day) but Octopus Energy returns 48 half-hourly prices. When _gather_optimization_data received period_count=48 from len(prices), it created 48-element arrays, but the current_period index (e.g. 90) is a 15-minute index — causing "list assignment index out of range" and preventing any optimization. Normalize non-quarterly prices to 15-minute resolution in _get_price_data() by repeating each entry to fill the corresponding 15-min slots. The DP algorithm now always runs with period_duration_hours=0.25. Bump version to 6.1.3.

InfluxDB 1.x stores the unit of measurement in _measurement and the entity ID in an entity_id tag, while 2.x stores the entity ID directly in _measurement. Query filters now match on either field. CSV response parsers detect column positions from the header row instead of using hardcoded indices, preventing silent data loss when columns appear in a different order. Bump version to 6.2.0.

# Conflicts: # CHANGELOG.md # core/bess/battery_system_manager.py

InfluxDB 1.x stores entity_id without the "sensor." domain prefix (e.g. "urq2eyf00c_..." not "sensor.urq2eyf00c_..."), with the domain in a separate tag. The 6.2.0 filter was querying entity_id with the prefix, which matched zero rows. - Remove "sensor." prefix from entity_id filter in both get_sensor_data and get_sensor_data_batch queries - Update _extract_sensor_name() to normalize unprefixed 1.x entity_id values by prepending "sensor." for downstream consumers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix InfluxDB 1.x entity_id filter returning zero rows

SensorCollector, EnergyFlowCalculator, and HistoricalDataStore were initialized with the default 30 kWh capacity but update_settings() only updated BatterySettings without propagating the configured 10 kWh to these components. SOE was stored at 3x the correct value and then divided by the real capacity at display time, inflating SOC (e.g. 56% became 168%). Capacity is now propagated to all dependent components when battery settings are updated. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix battery SOC showing >100% due to capacity mismatch

Dashboard energy flow chart and Savings page lost all history on restart because HistoricalDataStore kept records only in memory. Added JSON file persistence to /config/bess_historical_data.json following the same pattern used by ScheduleStore: save after each record_period() call, load on init if data is from today, discard stale data. Deserialization filters init=False fields so EnergyData and EconomicData __post_init__ methods recalculate derived values correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Persist historical data across restarts

HA add-ons don't have /config mounted as persistent storage without a map directive. Both HistoricalDataStore and ScheduleStore were writing to /config which is an ephemeral container layer. Changed to /data which is always available as persistent add-on storage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When tomorrow's Octopus prices aren't yet published, the optimizer falls back to a terminal value to prevent dumping battery charge at end of day. Previously this used the average of all remaining buy prices, which included tonight's expensive peak periods and inflated the terminal value enough to prevent discharge at those same peaks - a circular and self-defeating calculation. Now uses the last known non-zero price in the horizon (typically an overnight off-peak rate), which is a much better proxy for what overnight recharging would cost tomorrow. Co-authored-by: Ian P. Christian <ian.christian@zoopla.co.uk>

XGBoost-based model predicts 24h energy consumption at 15-minute resolution. Integrates with the battery optimizer via `consumption_strategy: ml_prediction` setting, with daily retrain at 23:00 and cached predictions. - ml/ module: CLI tools (train, predict, evaluate, report), feature engineering, InfluxDB data fetching, configurable derived features - Backend: /api/ml-report endpoint serving model metrics, predictions, and feature importance from a training sidecar file - Frontend: ML Report tab with forecast chart, metrics comparison table, and feature importance visualization - Config: ML settings nested under options.ml in config.yaml; model_path is a hardcoded default, not user-configurable - ml_config.yaml removed from tracking (dev-only convenience file) Co-authored-by: Ian P. Christian <ian.christian@zoopla.co.uk> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

XGBoost-based model predicts 24h energy consumption at 15-minute resolution. Integrates with the battery optimizer via `consumption_strategy: ml_prediction` setting, with daily retrain at 23:00 and cached predictions. - ml/ module: CLI tools (train, predict, evaluate, report), feature engineering, InfluxDB data fetching, configurable derived features - Backend: /api/ml-report endpoint serving model metrics, predictions, and feature importance from a training sidecar file - Frontend: ML Report tab with forecast chart, metrics comparison table, and feature importance visualization - Config: ML settings nested under options.ml in config.yaml; model_path is a hardcoded default, not user-configurable - ml_config.yaml removed from tracking (dev-only convenience file) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ning

… image The v7.0.0 release failed to build because the Dockerfile was missing COPY ml/ and the Alpine build dependencies (g++, cmake, make, libgomp) needed to compile xgboost from source on musl. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Alpine (musl) has no prebuilt wheels for xgboost, scikit-learn, or pandas, forcing compilation from source on every install (~15+ min). Debian bookworm uses glibc so manylinux wheels install in under a minute. Also fixes the xgboost build failure caused by missing linux/mempolicy.h on Alpine. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The ml schema section had pv_power, import_power, etc. that belong under sensors, not ml. HA schema validation rejected user configs missing these non-existent ML options. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Thread currency from config through the optimization pipeline so economic chain explanations, log messages, and UI displays use the configured currency instead of hardcoded "SEK". Key changes: - Add currency parameter to generate_economic_chain(), create_decision_data(), _calculate_reward(), _run_dynamic_programming(), and optimize_battery_schedule() - battery_system_manager passes home_settings.currency to optimizer - Frontend derives currency from API data instead of hardcoding 'SEK' - Neutralize Swedish-specific comments (öre, överföringsavgift, etc.) - Rename BATTERY_CHARGE_CYCLE_COST_SEK to BATTERY_CHARGE_CYCLE_COST - Widen ElectricitySettings.area type from 'SE1'|'SE2'|'SE3'|'SE4' to string - Clean up config.yaml comments to be locale-neutral Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The power sensor fields (pv_power, local_load_power, import_power, export_power, output_power, self_power, system_power) were in the ml schema section instead of the sensors section. When removed from ml in v7.0.3, they weren't added back to sensors. HA strips options not in the schema, so all power sensor values were None at runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…strategy ML Report now shows the weekly average (influxdb_profile) as a third line on the forecast chart alongside ML predictions and yesterday's actual. The report page is also accessible when consumption_strategy is set to influxdb_profile. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Previously the ML model only trained when consumption_strategy was ml_prediction. Now it trains whenever the ml config section is present, so the ML Report page has data even when using influxdb_profile or other strategies. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In the HA add-on, InfluxDB credentials live in options.json, not environment variables. The ML config builder now checks env vars first (dev) and falls back to the influxdb section in app_options (production). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The ML forecast cache was only populated when consumption_strategy was ml_prediction. The report endpoint now generates predictions on demand when the cache is empty, so the ML line always appears on the chart. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Backend: Fill overnight energy data gaps using power (W) sensors when cumulative kWh sensors show zero due to 0.1 kWh reporting resolution. Averages instantaneous power readings per 15-min period and converts to kWh, giving continuous data even at low overnight consumption. Frontend: Fix tomorrow data causing charts to render 3x wider than expected. The API returns tomorrow periods numbered 96-183 but the charts were using these raw numbers for x-axis positioning. Normalize to 0-95 before offsetting by 24h. Also fixes tooltip showing times like "28:00" instead of "04:00" in both EnergyFlowChart and BatteryLevelChart. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix overnight chart gaps and tomorrow data rendering

Remove hardcoded SEK currency and Swedish locale references

- Guard missing 'ml' section in load_config() with descriptive KeyError - Validate strategy-config compatibility on startup, fall back to 'fixed' - Generate ML predictions proactively after training (boot + daily 23:00) - Remove on-demand prediction generation from /api/ml-report endpoint - Wrap ML boot block in try/except so training failures don't crash system - Bump version to 7.0.9 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Make ML config optional & generate predictions proactively

The ML config builder used HA_TOKEN for the HA API token, but the HA supervisor sets HASSIO_TOKEN. Weather forecast calls failed with 401 auth errors, leaving _ml_forecast_cache as None. Now checks HASSIO_TOKEN first (matching app.py controller pattern), falling back to HA_TOKEN for local development. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The JSON file persistence (_save_to_disk/_load_from_disk) was added as a workaround while InfluxDB 1.x/2.x data-fetching bugs were being diagnosed. Those bugs have since been fixed (header-aware CSV parsing, dual-filter queries, batch parser prefix mismatch). On every startup, _fetch_and_initialize_historical_data() rebuilds all periods from InfluxDB, overwriting any persisted data — making the JSON file redundant. Strategic intent persistence (ScheduleStore) is unaffected. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove redundant disk persistence from HistoricalDataStore

The "last known price" approach was provider-specific — it worked for Octopus Agile by accident (last price before the gap is off-peak) but for Nordpool the last price is hour 23 which could be anything. The median is outlier-resistant, works across all price providers, and correctly handles negative prices (which the p > 0 filter skipped). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use median price for terminal value calculation

…r last period After a restart, the last completed period was collected via live sensors (runtime path) instead of InfluxDB. Live sensor values at e.g. 21:28 include energy from the in-progress period (21:15-21:30), inflating the last completed period and leaving the next period nearly empty on the chart. Fix: when _last_readings cache is None (startup/restart), always use InfluxDB which has correct 15-minute boundary data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Read timezone from Home Assistant instead of hardcoding Europe/Stockholm HA already knows the timezone via /api/config, so read it at startup instead of forcing users to configure it or hardcoding a value. - Add get_ha_config() to HomeAssistantAPIController - Add set_timezone() to time_utils, called during BESSController init - Convert all TIMEZONE imports to attribute access (time_utils.TIMEZONE) so modules see the updated value after set_timezone() runs - Replace hardcoded ZoneInfo("Europe/Stockholm") in influxdb_helper - Make docker-compose TZ a per-developer setting via .env Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Bump version to 7.1.0 and update CHANGELOG Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Ian P. Christian <ian.christian@zoopla.co.uk> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

The 23:00 cron job generated ML predictions but the 23:55 next-day preparation cleared them without regenerating. Now the 23:00 job only retrains the model, and the 23:55 preparation generates predictions after clearing the cache, so they survive into the new day. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pookey · 2026-03-07T16:09:19Z

Fix folded into PR #34 (ml-prediction branch).

ipc-zpg and others added 30 commits February 27, 2026 21:02

Merge pull request #1 from pookey/feature/octopus-energy-support

720063a

Add Octopus Energy Agile tariff price support

Merge pull request #2 from pookey/feature/octopus-energy-support

ad7a506

Update docs for Octopus Energy support

Bump version to 6.1.0 for Octopus Energy support

f9f8bad

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge pull request #3 from pookey/feature/octopus-energy-support

e0b8424

Bump version to 6.1.0

Bump version to 6.1.1

fe401d5

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge pull request #4 from pookey/feature/octopus-energy-support

6a0c7e0

Add Octopus Energy Agile tariff price support

Merge pull request #5 from pookey/fix/inverter-page-blank-on-dashboar…

7e7808e

…d-failure Fix Inverter page blank when dashboard endpoint fails

Merge pull request #6 from pookey/fix/octopus-price-period-normalization

6b2fd06

Fix Octopus Energy optimization crash due to period index mismatch

Merge pull request #7 from pookey/fix/influxdb-dual-version-support

899e81c

Fix InfluxDB queries for dual 1.x/2.x support

Merge pull request #8 from pookey/fix/price-settings-not-propagated

b5d2ffd

Fix Octopus Energy price display and period resolution

Merge remote-tracking branch 'fork/main'

a118182

# Conflicts: # CHANGELOG.md # core/bess/battery_system_manager.py

Bump version to 6.2.1 for Home Assistant add-on detection

4835d19

Merge pull request #9 from pookey/fix/influxdb-1x-entity-id-filter

9297fe4

Fix InfluxDB 1.x entity_id filter returning zero rows

Merge pull request #11 from pookey/worktree-battery_soc

fd48f43

Fix battery SOC showing >100% due to capacity mismatch

chore: pointless version bump

0b342c7

Merge pull request #12 from pookey/worktree-first_loaded

bc0cb25

Persist historical data across restarts

pookey and others added 27 commits March 5, 2026 17:22

Merge remote-tracking branch 'origin/main' into worktree-machine_lear…

6defdbe

…ning

Bump version to 7.0.6

143c1c1

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge pull request #34 from pookey/fix/chart-gaps-and-tomorrow-data

ec449c2

Fix overnight chart gaps and tomorrow data rendering

Merge pull request #33 from pookey/locale-cleanup

e2233e8

Remove hardcoded SEK currency and Swedish locale references

Merge pull request #35 from pookey/worktree-machine_learning

754521e

Make ML config optional & generate predictions proactively

Merge pull request #36 from pookey/remove/historical-data-persistence

8e7638b

Remove redundant disk persistence from HistoricalDataStore

Merge pull request #37 from pookey/fix/terminal-value-median

2d516c3

Use median price for terminal value calculation

pookey force-pushed the fix/ml-predictions-wiped-at-2355 branch from d5f0c69 to 103f05c Compare March 7, 2026 15:51

pookey closed this Mar 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ML predictions missing from report page#36

Fix ML predictions missing from report page#36
pookey wants to merge 92 commits intojohanzander:mainfrom
pookey:fix/ml-predictions-wiped-at-2355

pookey commented Mar 7, 2026

Uh oh!

pookey commented Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

pookey commented Mar 7, 2026

Summary

Test plan

Uh oh!

pookey commented Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants