Skip to content

Conversation

@petersalomonsen
Copy link
Collaborator

@petersalomonsen petersalomonsen commented Dec 30, 2025

Summary

This PR adds two new API endpoints for querying historical balance data:

  • Chart API (/api/balance-history/chart) - Returns time-series data for charting balance changes
  • CSV Export API (/api/balance-history/csv) - Exports balance history as downloadable CSV

Additionally includes optimizations, database migrations, and improvements from PR #17 review feedback.

Key Features

1. Balance History Chart API

  • Time-series snapshots at configurable intervals (hourly, daily, weekly, monthly)
  • Groups data by token for easy charting
  • Correctly handles prior balances (data from before the query timeframe)
  • Returns BigDecimal values for precise balance representation
  • Example: GET /api/balance-history/chart?account_id=dao.near&start_time=2025-01-01&end_time=2025-12-31&interval=daily

2. CSV Export API

  • Downloads complete transaction history with metadata
  • Includes token symbols from counterparties table
  • Filters out internal records (SNAPSHOT, NOT_REGISTERED)
  • Example: GET /api/balance-history/csv?account_id=dao.near&start_time=2025-01-01&end_time=2025-12-31

3. Prior Balance Support

  • Critical fix: Chart API now loads most recent balance before the query timeframe
  • Ensures correct starting balances instead of defaulting to 0
  • Query uses DISTINCT ON (token_id) with ORDER BY block_height DESC for efficiency

4. BigDecimal Refactoring

  • Changed all balance query functions to return BigDecimal instead of raw token amounts
  • Provides decimal-formatted values (e.g., "9.99998" instead of "9999980")
  • Ensures correct precision handling across all APIs

5. Intents Token Metadata Discovery

  • Added FT metadata discovery for intents.near multi-tokens
  • Supports long token identifiers like intents.near:nep141:base-0x833589fcd6edb6e08f4c7c32d4f71b54bda02913.omft.near
  • Fetches metadata including symbol, name, and decimals for proper display

6. Database Migrations

  • Increased VARCHAR column lengths in counterparties and balance_changes tables
  • Supports long intents.near token identifiers (up to 128 characters)
  • Migrations:
    • 20251231000001_increase_counterparties_account_id_length.sql
    • 20251231000002_increase_balance_changes_varchar_columns.sql

7. Monitoring Optimization

  • Optimized monitoring cycle to run only once per interval
  • Reduces unnecessary database queries and API calls

8. Review Feedback Addressed

From akorchyn's review on PR #17:

  • ✅ Use U128 directly instead of serde_json::Value in ft.rs
  • ✅ Use Tokens::ft_metadata from near-api-rs for cleaner API
  • ✅ Add sputnik-dao account validation to prevent monitoring abuse
  • ✅ Document N+1 query pattern as intentional for background jobs
  • ✅ Use tokio::time::interval for accurate timing
  • ✅ Move block_timestamp_to_datetime to dedicated utils module
  • ✅ Document MONITOR_INTERVAL_MINUTES in .env.example

Testing

Comprehensive integration tests using real webassemblymusic-treasury data:

  • ✅ Snapshot-based testing for all chart intervals (hourly, daily, weekly, monthly)
  • ✅ CSV export validation with token metadata
  • ✅ Test data regenerated from current database state (773 balance changes, 10 counterparties)
  • ✅ Validates 11 different token types with specific decimal-formatted balances
  • ✅ All 24 unit tests pass
  • ✅ Integration tests pass

Test Plan

  • Chart API returns correct balance snapshots for all intervals
  • CSV export includes token symbols and filters internal records
  • Prior balances are correctly loaded from before query timeframe
  • BigDecimal values are correctly formatted
  • Intents token metadata is discovered and cached
  • Database migrations run successfully
  • sputnik-dao validation rejects non-DAO accounts
  • All unit and integration tests pass

🤖 Generated with Claude Code

petersalomonsen and others added 14 commits December 30, 2025 16:11
- Add comprehensive API specification for balance history charts and CSV export
- Document FT registration status check optimization using storage_balance_of
- Add NOT_REGISTERED counterparty marker to prevent unnecessary backward searches
- Clarify snapshot calculation algorithm and response formats
Add chart and CSV export endpoints for querying historical balance data:
- GET /api/balance-history/chart - Returns balance snapshots at intervals
- GET /api/balance-history/csv - Exports balance changes as downloadable CSV

Key features:
- Support for hourly, daily, weekly, and monthly intervals
- Optional token filtering via query params
- Prior balance loading to show correct starting balances
- CSV includes token metadata (symbols) from counterparties table
- Excludes SNAPSHOT and NOT_REGISTERED records from exports

The critical fix loads the most recent balance_after for each token
before the query timeframe, ensuring snapshots show actual balances
instead of defaulting to 0 when no changes occur during the period.

Integration tests use real webassemblymusic-treasury data loaded from
SQL dumps to verify API correctness with production-like scenarios.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Create CLAUDE.md that references the shared copilot-instructions.md
file to ensure all AI coding assistants follow the same guidelines.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1. Move block_timestamp_to_datetime to utils module
   - Created dedicated utils module for shared functions
   - Removed duplicate timestamp conversion from gap_filler
   - Added tests for timestamp conversion

2. Use tokio::time::interval for monitoring loop
   - Replaced manual sleep loop with tokio::time::interval
   - Provides more accurate timing that accounts for processing time
   - Ensures consistent interval between cycles

3. Document MONITOR_INTERVAL_MINUTES in .env.example
   - Added comment explaining the environment variable
   - Default value is 5 minutes if not set

Addresses PR NEAR-DevHub#17 review comments from akorchyn:
- Comment: "Can we put it into utils?"
- Comment: "Personally, I prefer tokio::time::interval"
- Comment: "Should we move to env too?"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Use U128 directly instead of serde_json::Value in ft.rs
- Use Tokens::ft_metadata from near-api-rs for cleaner API
- Add sputnik-dao account validation to prevent abuse
- Document N+1 query pattern as intentional for background jobs
- Fix import path for block_timestamp_to_datetime utility

Addresses feedback from akorchyn on PR NEAR-DevHub#17:
- NEAR-DevHub#17 (comment)
- NEAR-DevHub#17 (comment)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Resolved conflict in src/main.rs by combining:
- Kept interval_minutes == 0 check for disabling monitoring
- Kept tokio::time::interval for accurate timing
- Integrated latest changes from origin/main

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Use FungibleTokenMetadata directly from near-api-rs instead of a
type alias. This makes the code cleaner and more direct.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Move MONITOR_INTERVAL_MINUTES from direct env reading to the EnvVars
struct for consistency with other environment variables.

Addresses review comment from akorchyn.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated test account from test-account.near to test-treasury.sputnik-dao.near to match the API validation requirement that only allows accounts ending with .sputnik-dao.near to be monitored.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add exact row count assertions (204 rows for CSV export)
- Create snapshot files for regression testing of all API outputs
- Snapshots only generated when GENERATE_NEW_TEST_SNAPSHOTS=1
- Remove redundant test_csv_includes_token_metadata (covered by CSV snapshot)
- Add comprehensive documentation for snapshot testing workflow

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix issue where intents token metadata (symbol, decimals, etc.) wasn't
being stored in the counterparties table during monitoring cycles.

Changes:
- Modified intents::get_balance_at_block to call ensure_ft_metadata
- Added pool parameter to intents balance query function
- ensure_ft_metadata already had logic to extract actual FT contract
  from intents tokens (e.g., eth.omft.near from intents.near:nep141:eth.omft.near)
- Added integration test to verify metadata discovery
- Centralized network config creation in tests/common/mod.rs

The monitoring cycle now correctly:
1. Discovers intents tokens via mt_tokens_for_owner
2. Creates snapshot records for new tokens
3. Fills gaps which queries balances
4. Balance queries trigger metadata fetching and storage

Test results show successful metadata discovery for tokens like:
- ETH (18 decimals)
- BTC (8 decimals)
- SOL (9 decimals)
- wNEAR (24 decimals)
- XRP (6 decimals)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated all balance-related functions to return BigDecimal instead of String
to properly map to PostgreSQL NUMERIC type without precision loss.

Changes:
- Updated get_balance_at_block in near.rs, ft.rs, intents.rs, and mod.rs
- Updated convert_raw_to_decimal to return BigDecimal
- Updated BalanceGap and FilledGap structs to use BigDecimal fields
- Updated binary_search::find_balance_change_block to accept BigDecimal
- Removed ::TEXT casts from gap_detector SQL queries
- Updated all tests to use BigDecimal comparisons
- Fixed intents token test values to use decimal-adjusted balances

All 33 library tests passing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Regenerate SQL dumps from current dev database
  - webassemblymusic_counterparties.sql: 10 FT tokens
  - webassemblymusic_balance_changes.sql: 773 balance records
- Regenerate all test snapshots (hourly, daily, weekly, monthly, CSV)
- Update test expectations to match new data:
  - Remove obsolete token from expected list
  - Use decimal-formatted balance values from API
  - Update CSV row count from 204 to 173

All balance history API tests now passing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@petersalomonsen petersalomonsen force-pushed the feat/balance-apis-and-optimizations branch from 149617b to a469eda Compare December 31, 2025 14:40
@petersalomonsen
Copy link
Collaborator Author

@race-of-sloths include

@petersalomonsen petersalomonsen marked this pull request as ready for review December 31, 2025 14:46
@race-of-sloths
Copy link

race-of-sloths commented Dec 31, 2025

@petersalomonsen Thank you for your contribution! Your pull request is now a part of the Race of Sloths!
Weekly streak is on the road, smart strategy! Secure your streak with another PR!

Shows inviting banner with latest news.

Shows profile picture for the author of the PR

Current status: executed
Reviewer Score
@akorchyn 8

Your contribution is much appreciated with a final score of 8!
You have received 108 (80 base + 10 weekly bonus + 20% lifetime bonus) Sloth points for this contribution

@akorchyn received 25 Sloth Points for reviewing and scoring this pull request.

Another weekly streak completed, well done @petersalomonsen! To keep your weekly streak and get another bonus make pull request next week! Looking forward to see you in race-of-sloths

What is the Race of Sloths

Race of Sloths is a friendly competition where you can participate in challenges and compete with other open-source contributors within your normal workflow

For contributors:

  • Tag @race-of-sloths inside your pull requests
  • Wait for the maintainer to review and score your pull request
  • Check out your position in the Leaderboard
  • Keep weekly and monthly streaks to reach higher positions
  • Boast your contributions with a dynamic picture of your Profile

For maintainers:

  • Score pull requests that participate in the Race of Sloths and receive a reward
  • Engage contributors with fair scoring and fast responses so they keep their streaks
  • Promote the Race to the point where the Race starts promoting you
  • Grow the community of your contributors

Feel free to check our website for additional details!

Bot commands
  • For contributors
    • Include a PR: @race-of-sloths include to enter the Race with your PR
  • For maintainers:
    • Invite contributor @race-of-sloths invite to invite the contributor to participate in a race or include it, if it's already a runner.
    • Assign points: @race-of-sloths score [1/2/3/5/8/13] to award points based on your assessment.
    • Reject this PR: @race-of-sloths exclude to send this PR back to the drawing board.
    • Exclude repo: @race-of-sloths pause to stop bot activity in this repo until @race-of-sloths unpause command is called

petersalomonsen and others added 9 commits December 31, 2025 15:59
The test was expecting raw satoshi values (20000) but after the
BigDecimal refactoring, all amounts are stored as decimal-formatted
values (0.0002 BTC). Updated the assertion to compare against the
decimal value instead of the raw value.

This fixes the CI test failure where BigDecimal scale mismatch
occurred (scale=8 vs scale=0).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Updated TestServer to use DATABASE_URL_TEST for integration tests
- Added filtering for pg_dump v18 security commands (\restrict, \unrestrict)
- Skip SET and SELECT pg_catalog commands that are incompatible with Postgres 16
- This ensures tests run correctly both locally and in CI

The pg_dump v18.1 output includes commands that aren't supported by
PostgreSQL 16, so we filter them out when loading test data via sqlx.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Update all integration tests to use consistent pattern for loading
environment variables that works both locally and in CI:

1. Load .env first (base configuration)
2. Load .env.test (overrides DATABASE_URL to test database)
3. Read DATABASE_URL (not DATABASE_URL_TEST)

This matches the CI configuration where .env is created with DATABASE_URL
pointing to the test database at port 5433.

Changes:
- tests/common/mod.rs: TestServer::start() now loads env files correctly
- tests/balance_history_apis_test.rs: load_test_data() uses standard pattern
- tests/intents_tokens_metadata_test.rs: Updated to match other tests

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add ON CONFLICT clause to counterparties INSERT statements and clear
partial data before loading. This fixes test failures in CI where
multiple tests run in parallel and attempt to insert the same test data.

Changes:
- Add `ON CONFLICT (account_id) DO NOTHING` to counterparties inserts
- Clear existing balance_changes for test account before loading
- Improve error messages to show database errors

Fixes issue where tests were failing with duplicate key violations
when running in parallel in CI environment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix string replacement logic to properly append ON CONFLICT clause
without creating syntax errors. The previous logic was doing a simple
replace of ");", which could match in the middle of VALUES clauses.

Now uses proper string slicing to replace only the final ");".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…use to INSERT statements

The previous implementation was checking if the trimmed line ends with ");",
but then slicing the original untrimmed line. This caused the string slicing
to be at the wrong position when there was trailing whitespace, resulting in
malformed SQL syntax errors in CI.

The fix ensures we trim the line first, then both check and slice the trimmed
version, making the behavior consistent regardless of trailing whitespace.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Instead of trying to make INSERT statements idempotent with ON CONFLICT,
we now simply delete all test data (both counterparties and balance_changes)
before loading. This is simpler, more reliable, and works perfectly since
tests run serially with #[serial].

This approach:
- Eliminates complex SQL string manipulation
- Avoids potential edge cases with trailing whitespace or line endings
- Is more straightforward and easier to understand
- Ensures a clean slate for each test run

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@petersalomonsen
Copy link
Collaborator Author

petersalomonsen commented Dec 31, 2025

Tests pass in

https://github.com/petersalomonsen/treasury26/actions/runs/20625010671/job/59233994186?pr=3

( fastnear api key is needed in CI )

akorchyn
akorchyn approved these changes Jan 2, 2026
) -> Result<Vec<BalanceChange>, Box<dyn std::error::Error>> {
let rows = if let Some(tokens) = token_ids {
sqlx::query!(
r#"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kind of worry that we might edit one query and forget about another, so maybe some generilized version would be prefered

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure how to combine them when using the sqlx:query macro with compile time verification. Leaving it as is for now.

csv.push_str("block_height,block_time,token_id,token_symbol,counterparty,amount,balance_before,balance_after,transaction_hashes,receipt_id\n");

// Rows (exclude SNAPSHOT and NOT_REGISTERED)
for change in changes {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might consider using https://github.com/BurntSushi/rust-csv

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah good point to use a library for this, but since it is quite simple let's look at it in a separate PR

@akorchyn
Copy link
Member

akorchyn commented Jan 2, 2026

@race-of-sloths score 8
oopsie

@NEAR-DevHub NEAR-DevHub deleted a comment from race-of-sloths Jan 2, 2026
@akorchyn
Copy link
Member

akorchyn commented Jan 2, 2026

@petersalomonsen can you tell me what I need to put?
image
As you can see I put a secret

@petersalomonsen
Copy link
Collaborator Author

@petersalomonsen can you tell me what I need to put? image As you can see I put a secret

hmm.. that looks correct. Same as in mine.. Not sure what is wrong then

image

- Move imports to top level in balance module
- Use BigDecimal::from(0) instead of parsing "0" string
- Remove ::TEXT casts from SQL queries for proper BigDecimal handling
- Add Interval enum with to_duration() method for type-safe intervals
- Use chrono's native RFC3339 deserialization for DateTime<Utc>
- Refactor to use into_iter().map().collect() pattern
- Move interval_timer.tick().await to beginning of loop (Rust convention)
- Update tests to use RFC3339 format with timezone indicator

All changes improve type safety, reduce code complexity, and follow Rust conventions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@akorchyn
Copy link
Member

akorchyn commented Jan 2, 2026

Do we need PIKESPEAK key? I don't have one. @petersalomonsen I'll make you admin and can you add yours?

It's green on main

@petersalomonsen
Copy link
Collaborator Author

Do we need PIKESPEAK key? I don't have one. @petersalomonsen I'll make you admin and can you add yours?

It's green on main

I think the reason is that the PR comes from my fork, and then it does not have access to repo secrets. Since it pass on main, which would have the same problem if the API key was not there, I expect that to be reason.

petersalomonsen and others added 4 commits January 2, 2026 21:29
Add comments to load_prior_balances and load_balance_changes
explaining why the SQL queries are intentionally duplicated.

The sqlx::query! macro requires compile-time verification, which
provides type safety but necessitates literal SQL strings. The
trade-off is acceptable because:
- Schema changes will cause both queries to fail compilation
- Editing one query and not the other will cause type mismatches
- The queries are adjacent, making updates straightforward

This addresses the maintainability concern while preserving the
benefits of compile-time type checking.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed amount, balance_before, and balance_after fields from String to
BigDecimal for better type safety and consistency. Also updated
BalanceSnapshot to use BigDecimal.

Changes:
- Updated BalanceChange struct to use BigDecimal for amount, balance_before, balance_after
- Updated BalanceSnapshot to use BigDecimal for balance field
- Removed ::TEXT casts from SQL queries (PostgreSQL numeric maps directly to BigDecimal)
- Updated test expectations to match BigDecimal serialization (includes trailing zeros)
- Regenerated test snapshots for chart and CSV APIs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Removed the first interval_timer.tick().await before the loop as it
caused a double-tick on the first iteration - the interval would tick
once before the loop and again at the start of the loop body, causing an
unnecessary sleep for the configured interval duration.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed Interval from returning a Duration to having an increment()
method that properly advances DateTime values. This ensures monthly
intervals use correct month boundaries (e.g., Jan 1 -> Feb 1 -> Mar 1)
instead of approximating with 30-day periods.

Changes:
- Replaced to_duration() with increment(datetime) method
- Monthly intervals now properly handle year/month boundaries
- Updated calculate_snapshots to use interval.increment()
- Regenerated test snapshots to reflect accurate monthly intervals
  (7 snapshots for June-December instead of 8 with 30-day periods)
- Fixed snapshot data to show correct balance states at month boundaries

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@petersalomonsen
Copy link
Collaborator Author

Latest test run can be found here: petersalomonsen#3

@petersalomonsen
Copy link
Collaborator Author

I believe it is ready @akorchyn

// Keep the same day, time, and timezone
datetime
.with_year(new_year)
.and_then(|dt| dt.with_month(new_month))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it work properly if it is 31st ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for seeing that. Added unit tests and a fix here: #24

@akorchyn akorchyn merged commit 4e9b634 into NEAR-DevHub:main Jan 2, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants