-
Notifications
You must be signed in to change notification settings - Fork 1
feat: Balance History APIs with optimizations #22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Balance History APIs with optimizations #22
Conversation
- Add comprehensive API specification for balance history charts and CSV export - Document FT registration status check optimization using storage_balance_of - Add NOT_REGISTERED counterparty marker to prevent unnecessary backward searches - Clarify snapshot calculation algorithm and response formats
Add chart and CSV export endpoints for querying historical balance data: - GET /api/balance-history/chart - Returns balance snapshots at intervals - GET /api/balance-history/csv - Exports balance changes as downloadable CSV Key features: - Support for hourly, daily, weekly, and monthly intervals - Optional token filtering via query params - Prior balance loading to show correct starting balances - CSV includes token metadata (symbols) from counterparties table - Excludes SNAPSHOT and NOT_REGISTERED records from exports The critical fix loads the most recent balance_after for each token before the query timeframe, ensuring snapshots show actual balances instead of defaulting to 0 when no changes occur during the period. Integration tests use real webassemblymusic-treasury data loaded from SQL dumps to verify API correctness with production-like scenarios. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Create CLAUDE.md that references the shared copilot-instructions.md file to ensure all AI coding assistants follow the same guidelines. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1. Move block_timestamp_to_datetime to utils module - Created dedicated utils module for shared functions - Removed duplicate timestamp conversion from gap_filler - Added tests for timestamp conversion 2. Use tokio::time::interval for monitoring loop - Replaced manual sleep loop with tokio::time::interval - Provides more accurate timing that accounts for processing time - Ensures consistent interval between cycles 3. Document MONITOR_INTERVAL_MINUTES in .env.example - Added comment explaining the environment variable - Default value is 5 minutes if not set Addresses PR NEAR-DevHub#17 review comments from akorchyn: - Comment: "Can we put it into utils?" - Comment: "Personally, I prefer tokio::time::interval" - Comment: "Should we move to env too?" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Use U128 directly instead of serde_json::Value in ft.rs - Use Tokens::ft_metadata from near-api-rs for cleaner API - Add sputnik-dao account validation to prevent abuse - Document N+1 query pattern as intentional for background jobs - Fix import path for block_timestamp_to_datetime utility Addresses feedback from akorchyn on PR NEAR-DevHub#17: - NEAR-DevHub#17 (comment) - NEAR-DevHub#17 (comment) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Resolved conflict in src/main.rs by combining: - Kept interval_minutes == 0 check for disabling monitoring - Kept tokio::time::interval for accurate timing - Integrated latest changes from origin/main 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Use FungibleTokenMetadata directly from near-api-rs instead of a type alias. This makes the code cleaner and more direct. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Move MONITOR_INTERVAL_MINUTES from direct env reading to the EnvVars struct for consistency with other environment variables. Addresses review comment from akorchyn. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated test account from test-account.near to test-treasury.sputnik-dao.near to match the API validation requirement that only allows accounts ending with .sputnik-dao.near to be monitored. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add exact row count assertions (204 rows for CSV export) - Create snapshot files for regression testing of all API outputs - Snapshots only generated when GENERATE_NEW_TEST_SNAPSHOTS=1 - Remove redundant test_csv_includes_token_metadata (covered by CSV snapshot) - Add comprehensive documentation for snapshot testing workflow 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix issue where intents token metadata (symbol, decimals, etc.) wasn't being stored in the counterparties table during monitoring cycles. Changes: - Modified intents::get_balance_at_block to call ensure_ft_metadata - Added pool parameter to intents balance query function - ensure_ft_metadata already had logic to extract actual FT contract from intents tokens (e.g., eth.omft.near from intents.near:nep141:eth.omft.near) - Added integration test to verify metadata discovery - Centralized network config creation in tests/common/mod.rs The monitoring cycle now correctly: 1. Discovers intents tokens via mt_tokens_for_owner 2. Creates snapshot records for new tokens 3. Fills gaps which queries balances 4. Balance queries trigger metadata fetching and storage Test results show successful metadata discovery for tokens like: - ETH (18 decimals) - BTC (8 decimals) - SOL (9 decimals) - wNEAR (24 decimals) - XRP (6 decimals) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated all balance-related functions to return BigDecimal instead of String to properly map to PostgreSQL NUMERIC type without precision loss. Changes: - Updated get_balance_at_block in near.rs, ft.rs, intents.rs, and mod.rs - Updated convert_raw_to_decimal to return BigDecimal - Updated BalanceGap and FilledGap structs to use BigDecimal fields - Updated binary_search::find_balance_change_block to accept BigDecimal - Removed ::TEXT casts from gap_detector SQL queries - Updated all tests to use BigDecimal comparisons - Fixed intents token test values to use decimal-adjusted balances All 33 library tests passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Regenerate SQL dumps from current dev database - webassemblymusic_counterparties.sql: 10 FT tokens - webassemblymusic_balance_changes.sql: 773 balance records - Regenerate all test snapshots (hourly, daily, weekly, monthly, CSV) - Update test expectations to match new data: - Remove obsolete token from expected list - Use decimal-formatted balance values from API - Update CSV row count from 204 to 173 All balance history API tests now passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
149617b to
a469eda
Compare
|
@race-of-sloths include |
|
@petersalomonsen Thank you for your contribution! Your pull request is now a part of the Race of Sloths! Current status: executed
Your contribution is much appreciated with a final score of 8! @akorchyn received 25 Sloth Points for reviewing and scoring this pull request. Another weekly streak completed, well done @petersalomonsen! To keep your weekly streak and get another bonus make pull request next week! Looking forward to see you in race-of-sloths What is the Race of SlothsRace of Sloths is a friendly competition where you can participate in challenges and compete with other open-source contributors within your normal workflow For contributors:
For maintainers:
Feel free to check our website for additional details! Bot commands
|
The test was expecting raw satoshi values (20000) but after the BigDecimal refactoring, all amounts are stored as decimal-formatted values (0.0002 BTC). Updated the assertion to compare against the decimal value instead of the raw value. This fixes the CI test failure where BigDecimal scale mismatch occurred (scale=8 vs scale=0). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Updated TestServer to use DATABASE_URL_TEST for integration tests - Added filtering for pg_dump v18 security commands (\restrict, \unrestrict) - Skip SET and SELECT pg_catalog commands that are incompatible with Postgres 16 - This ensures tests run correctly both locally and in CI The pg_dump v18.1 output includes commands that aren't supported by PostgreSQL 16, so we filter them out when loading test data via sqlx. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Update all integration tests to use consistent pattern for loading environment variables that works both locally and in CI: 1. Load .env first (base configuration) 2. Load .env.test (overrides DATABASE_URL to test database) 3. Read DATABASE_URL (not DATABASE_URL_TEST) This matches the CI configuration where .env is created with DATABASE_URL pointing to the test database at port 5433. Changes: - tests/common/mod.rs: TestServer::start() now loads env files correctly - tests/balance_history_apis_test.rs: load_test_data() uses standard pattern - tests/intents_tokens_metadata_test.rs: Updated to match other tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add ON CONFLICT clause to counterparties INSERT statements and clear partial data before loading. This fixes test failures in CI where multiple tests run in parallel and attempt to insert the same test data. Changes: - Add `ON CONFLICT (account_id) DO NOTHING` to counterparties inserts - Clear existing balance_changes for test account before loading - Improve error messages to show database errors Fixes issue where tests were failing with duplicate key violations when running in parallel in CI environment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix string replacement logic to properly append ON CONFLICT clause without creating syntax errors. The previous logic was doing a simple replace of ");", which could match in the middle of VALUES clauses. Now uses proper string slicing to replace only the final ");". 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…use to INSERT statements The previous implementation was checking if the trimmed line ends with ");", but then slicing the original untrimmed line. This caused the string slicing to be at the wrong position when there was trailing whitespace, resulting in malformed SQL syntax errors in CI. The fix ensures we trim the line first, then both check and slice the trimmed version, making the behavior consistent regardless of trailing whitespace. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Instead of trying to make INSERT statements idempotent with ON CONFLICT, we now simply delete all test data (both counterparties and balance_changes) before loading. This is simpler, more reliable, and works perfectly since tests run serially with #[serial]. This approach: - Eliminates complex SQL string manipulation - Avoids potential edge cases with trailing whitespace or line endings - Is more straightforward and easier to understand - Ensures a clean slate for each test run 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
Tests pass in https://github.com/petersalomonsen/treasury26/actions/runs/20625010671/job/59233994186?pr=3 ( fastnear api key is needed in CI ) |
| ) -> Result<Vec<BalanceChange>, Box<dyn std::error::Error>> { | ||
| let rows = if let Some(tokens) = token_ids { | ||
| sqlx::query!( | ||
| r#" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kind of worry that we might edit one query and forget about another, so maybe some generilized version would be prefered
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure how to combine them when using the sqlx:query macro with compile time verification. Leaving it as is for now.
| csv.push_str("block_height,block_time,token_id,token_symbol,counterparty,amount,balance_before,balance_after,transaction_hashes,receipt_id\n"); | ||
|
|
||
| // Rows (exclude SNAPSHOT and NOT_REGISTERED) | ||
| for change in changes { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might consider using https://github.com/BurntSushi/rust-csv
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah good point to use a library for this, but since it is quite simple let's look at it in a separate PR
|
@race-of-sloths score 8 |
|
@petersalomonsen can you tell me what I need to put? |
hmm.. that looks correct. Same as in mine.. Not sure what is wrong then
|
- Move imports to top level in balance module - Use BigDecimal::from(0) instead of parsing "0" string - Remove ::TEXT casts from SQL queries for proper BigDecimal handling - Add Interval enum with to_duration() method for type-safe intervals - Use chrono's native RFC3339 deserialization for DateTime<Utc> - Refactor to use into_iter().map().collect() pattern - Move interval_timer.tick().await to beginning of loop (Rust convention) - Update tests to use RFC3339 format with timezone indicator All changes improve type safety, reduce code complexity, and follow Rust conventions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
Do we need PIKESPEAK key? I don't have one. @petersalomonsen I'll make you admin and can you add yours? It's green on main |
I think the reason is that the PR comes from my fork, and then it does not have access to repo secrets. Since it pass on main, which would have the same problem if the API key was not there, I expect that to be reason. |
Add comments to load_prior_balances and load_balance_changes explaining why the SQL queries are intentionally duplicated. The sqlx::query! macro requires compile-time verification, which provides type safety but necessitates literal SQL strings. The trade-off is acceptable because: - Schema changes will cause both queries to fail compilation - Editing one query and not the other will cause type mismatches - The queries are adjacent, making updates straightforward This addresses the maintainability concern while preserving the benefits of compile-time type checking. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed amount, balance_before, and balance_after fields from String to BigDecimal for better type safety and consistency. Also updated BalanceSnapshot to use BigDecimal. Changes: - Updated BalanceChange struct to use BigDecimal for amount, balance_before, balance_after - Updated BalanceSnapshot to use BigDecimal for balance field - Removed ::TEXT casts from SQL queries (PostgreSQL numeric maps directly to BigDecimal) - Updated test expectations to match BigDecimal serialization (includes trailing zeros) - Regenerated test snapshots for chart and CSV APIs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Removed the first interval_timer.tick().await before the loop as it caused a double-tick on the first iteration - the interval would tick once before the loop and again at the start of the loop body, causing an unnecessary sleep for the configured interval duration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed Interval from returning a Duration to having an increment() method that properly advances DateTime values. This ensures monthly intervals use correct month boundaries (e.g., Jan 1 -> Feb 1 -> Mar 1) instead of approximating with 30-day periods. Changes: - Replaced to_duration() with increment(datetime) method - Monthly intervals now properly handle year/month boundaries - Updated calculate_snapshots to use interval.increment() - Regenerated test snapshots to reflect accurate monthly intervals (7 snapshots for June-December instead of 8 with 30-day periods) - Fixed snapshot data to show correct balance states at month boundaries 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
Latest test run can be found here: petersalomonsen#3 |
|
I believe it is ready @akorchyn |
| // Keep the same day, time, and timezone | ||
| datetime | ||
| .with_year(new_year) | ||
| .and_then(|dt| dt.with_month(new_month)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it work properly if it is 31st ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for seeing that. Added unit tests and a fix here: #24



Summary
This PR adds two new API endpoints for querying historical balance data:
/api/balance-history/chart) - Returns time-series data for charting balance changes/api/balance-history/csv) - Exports balance history as downloadable CSVAdditionally includes optimizations, database migrations, and improvements from PR #17 review feedback.
Key Features
1. Balance History Chart API
GET /api/balance-history/chart?account_id=dao.near&start_time=2025-01-01&end_time=2025-12-31&interval=daily2. CSV Export API
GET /api/balance-history/csv?account_id=dao.near&start_time=2025-01-01&end_time=2025-12-313. Prior Balance Support
DISTINCT ON (token_id)withORDER BY block_height DESCfor efficiency4. BigDecimal Refactoring
BigDecimalinstead of raw token amounts5. Intents Token Metadata Discovery
intents.near:nep141:base-0x833589fcd6edb6e08f4c7c32d4f71b54bda02913.omft.near6. Database Migrations
counterpartiesandbalance_changestables20251231000001_increase_counterparties_account_id_length.sql20251231000002_increase_balance_changes_varchar_columns.sql7. Monitoring Optimization
8. Review Feedback Addressed
From akorchyn's review on PR #17:
U128directly instead ofserde_json::Valuein ft.rsTokens::ft_metadatafrom near-api-rs for cleaner APItokio::time::intervalfor accurate timingblock_timestamp_to_datetimeto dedicated utils moduleMONITOR_INTERVAL_MINUTESin .env.exampleTesting
Comprehensive integration tests using real webassemblymusic-treasury data:
Test Plan
🤖 Generated with Claude Code