Skip to content

refactor(data): unify market data flow to Python daemon via gRPC#166

Merged
ChoKhoOu merged 1 commit intomainfrom
refactor/unify-data-flow
Feb 25, 2026
Merged

refactor(data): unify market data flow to Python daemon via gRPC#166
ChoKhoOu merged 1 commit intomainfrom
refactor/unify-data-flow

Conversation

@ChoKhoOu
Copy link
Copy Markdown
Owner

Summary

  • Extend proto/tino/data/v1/data.proto with new RPCs: GetMarketQuote, GetMarketKlines, GetMarketOverview, ListSupportedExchanges
  • Implement Python service methods in DataServiceServicer delegating to exchange connectors
  • Update TS gRPC data client to support new market data RPCs
  • Add deprecation markers on TS-side crypto finance providers (binance-public, bybit, okx)
  • Establishes the architecture pattern for gradual data flow migration to Python daemon

Closes #50

Test Plan

  • buf lint passes on proto changes
  • bun run typecheck passes
  • bun test passes
  • cd python && uv run pytest passes
  • New gRPC methods return market data from Python-side exchange connectors
  • TS data client can call new Python-side market data methods

🤖 Generated with Claude Code

@github-actions github-actions bot added type/tests Test-related changes area/daemon Python NautilusTrader daemon area/grpc gRPC / ConnectRPC layer area/tools Agent tools labels Feb 25, 2026
Copy link
Copy Markdown
Owner Author

@ChoKhoOu ChoKhoOu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COMPLETE

Requirement Verification: PR #166 vs Issue #50 (PRD-048)

All acceptance criteria from Issue #50 are fully met.

Acceptance Criteria Checklist

  • New CEX data sources implemented in Python side: DataServiceServicer in python/tino_daemon/services/data.py adds 4 new gRPC methods (GetMarketQuote, GetMarketKlines, GetMarketOverview, ListSupportedExchanges) that delegate to Python-side exchange connectors via get_connector(exchange).
  • TS tools call Python via gRPC for data: market-data-router.ts routes new crypto_exchange_quote, crypto_exchange_klines, crypto_exchange_overview, and crypto_supported_exchanges actions through DataClient gRPC calls. The data-client.ts exposes typed methods matching the new proto RPCs.
  • Exchange adapters only implemented once (Python side): The TS side creates zero new exchange-specific adapters. All exchange logic lives in Python connectors accessed via get_connector(). TS is purely a gRPC consumer.
  • Data format unified in Python side: Protobuf messages MarketQuote and MarketKline define the canonical format. The Python _to_market_quote() helper converts exchange-specific data into the unified protobuf format.

Gradual Migration Pattern

  • Old TS-side crypto providers (binance-public, bybit, okx) are preserved but marked with @deprecated JSDoc annotations
  • New crypto_exchange_* actions coexist alongside existing actions (e.g., crypto_prices, crypto_market_data)
  • No existing functionality is removed — only new paths added

Coverage

  • Proto layer: 4 new RPCs with proper request/response messages (61 additions to data.proto)
  • Python service: Full implementation with input validation, error handling (ValueError/NotImplementedError/generic), and proper gRPC status codes
  • Python tests: 6 new test cases covering success/error paths for all 4 methods (192 lines)
  • TS client: 4 new typed methods in DataClient
  • TS router: 4 new case branches with exchange validation and bigint JSON serialization support
  • TS tests: 4 new test cases verifying gRPC routing for each action (104 lines)
  • Tool schema: Updated with new actions, exchange and interval parameters
  • Descriptions: Documentation updated with new actions table and usage notes

Copy link
Copy Markdown
Owner Author

@ChoKhoOu ChoKhoOu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE

Code Review: PR #166 — Unify Market Data Flow to Python Daemon via gRPC

Overview

This PR implements Phase 1 of PRD-048 (Issue #50) by adding 4 new gRPC RPCs to the DataService (GetMarketQuote, GetMarketKlines, GetMarketOverview, ListSupportedExchanges), wiring them through the Python daemon's exchange connector infrastructure, and exposing them as new crypto_exchange_* actions in the TS market-data tool. The gradual migration strategy is well-executed — existing TS finance providers are preserved with deprecation markers, while new data flows route through the daemon.

Strengths

  • Clean proto design: Messages are well-named, field types are appropriate (double for prices, int64 for timestamps on klines), and field numbers don't conflict with existing definitions.
  • Solid error handling in Python service: Three-tier error mapping (ValueError -> INVALID_ARGUMENT, NotImplementedError -> UNIMPLEMENTED, Exception -> INTERNAL) with a shared _set_error helper for consistency.
  • GetMarketOverview uses asyncio.gather: Parallel ticker fetching is the right call for batch queries.
  • BigInt serializer in fmt(): Correctly handles int64 fields from protobuf that become bigint in TypeScript.
  • Test coverage: Both Python and TS sides have good coverage including error paths (invalid exchange, negative limit). TS tests use spyOn with proper finally cleanup.
  • Deprecation markers: @deprecated JSDoc on binance-public, bybit, okx providers is the right approach for gradual migration.

Suggestions (non-blocking)

  1. _to_market_quote type annotation (python/tino_daemon/services/data.py):
    The ticker parameter is typed as object, forcing the use of getattr(ticker, "symbol") etc. Since this is always called with a Ticker dataclass, typing it as Ticker would improve readability and enable IDE support:

    @staticmethod
    def _to_market_quote(exchange: str, ticker: Ticker) -> data_pb2.MarketQuote:
        return data_pb2.MarketQuote(
            exchange=exchange,
            symbol=ticker.symbol,
            ...
        )
  2. GetMarketOverview partial failure: If one symbol in the asyncio.gather call fails, the entire request fails. For robustness, consider return_exceptions=True and filtering out exceptions, returning partial results. Not required for Phase 1, but worth noting for future iterations.

  3. Timestamp type inconsistency in proto: MarketQuote.timestamp is string while MarketKline.open_time/close_time are int64. This is pragmatic (matches upstream exchange formats), but a brief comment in the proto file explaining the design choice would help future contributors.

Architecture Alignment

The PR correctly follows the gradual migration strategy outlined in PRD-048:

  • New CEX data sources route through Python daemon
  • TS tool layer calls Python via gRPC
  • Existing TS providers preserved but marked deprecated
  • Exchange adapter implemented once (Python side only)

Verdict

Clean implementation with proper error handling, good test coverage, and correct architecture alignment. The suggestions above are non-blocking improvements. APPROVE.

@ChoKhoOu ChoKhoOu merged commit c663b4a into main Feb 25, 2026
4 of 7 checks passed
@ChoKhoOu ChoKhoOu deleted the refactor/unify-data-flow branch February 25, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/daemon Python NautilusTrader daemon area/grpc gRPC / ConnectRPC layer area/tools Agent tools type/tests Test-related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant