Qt dev 2.0 #232
Open
Qt dev 2.0 #232
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refactored history data retrieval logic to improve handling of start/end dates and row counts, ensuring correct offsets and time adjustments. Updated get_history_data_packages to accept optional start/end parameters. Enhanced unit tests for history panel to cover more scenarios, including index data, row count limits, and error handling for empty results. Removed unnecessary debug statements and clarified docstrings.
Changed DataType.id to use default frequency instead of asset type for merging data with the same name but different asset types. Updated get_history_data_from_source and get_history_panel to use id for merging and retrieving data. Adjusted related tests to check for new id format and improved test coverage for cases with different frequencies and asset types.
Updated DataSource.read_table_data to accept shares as a list or string for more flexible data selection. Enhanced test_historypanel.py with additional assertions, improved test coverage for multiple shares and index data, and added debug print statements for better traceability.
Updated DataType.id to use asset type instead of frequency, and refactored history data acquisition to use name-based keys for merging. Added a check to raise ValueError when duplicate frequencies are present for the same data type with combine_asset_types=True. Adjusted related tests to match new ID conventions and added a test for expected ValueError on freq conflicts.
Introduces a reset() method to the Operator class, allowing the object's runtime state and cached data to be cleared for restarting signal generation. The method is currently not implemented and raises NotImplementedError.
Deleted the parse_backtest_data_package function from config_parser.py as it is no longer needed. This helps clean up the codebase and improve maintainability.
Replaces get_backtest_data_package with check_and_prepare_backtest_data, and introduces separate functions for preparing trade prices and benchmark data. Improves parameter validation in get_history_data, removes deprecated check_and_prepare_hist_data, and updates run_mode_1 to use the new data preparation workflow for clarity and modularity.
Improved docstrings in history.py and _arg_validators.py for clarity and accuracy. Added type hints to _infer_symbol_market in qt_code.py and updated its docstring to better describe parameters and return values. Marked some parameters in _arg_validators.py for deprecation with TODO comments.
Introduces comprehensive unittest cases for check_and_prepare_backtest_data, check_and_prepare_trade_prices, and check_and_prepare_benchmark_data functions. Tests cover normal and edge cases, data structure validation, operator frequency handling, and data source integration to ensure robustness of qt run process.
Added TODO comments to deprecate 'multiple' options in argument validators in _arg_validators.py. Updated and clarified docstrings for data preparation functions in core.py to better describe their purpose and usage. Minor formatting fix in test_qt.py.
Refactored TestStrategy initialization in test_qt_run.py to use explicit parameters and DataType objects. Updated DataSource instantiation to use 'file_loc' instead of 'file_path' and removed 'file_type'. Fixed Operator __repr__ test expectations in test_operator_and_strategy.py. Removed unused import in test_config_parser.py. Minor comment formatting in visual.py.
Replaced BaseStrategy with GeneralStg for test strategy classes and updated their initialization to match the new base class requirements. Removed duplicate DummyStrategy and related legacy code, and improved test data access to use data type IDs instead of share codes.
Added 'forward', 'f', and 'accu' as valid values for the 'backtest_price_adj' argument. Updated the help text to clarify the meaning of each option for both stocks and funds.
Introduces the available_time property to DataType for determining when data becomes available during the day, and adds the unsymbolized property to indicate reference data types. Also updates reference data acquisition logic to use dtype_id instead of name for consistency.
Improved `get_history_data_packages` in `qteasy/history.py` to handle symbolized and unsymbolized data types separately, apply correct time offsets to DatetimeIndex, and validate shares input. Updated and expanded unit tests in `tests/test_qt_run.py` to cover more scenarios, including mixed frequency strategies, reference data, time offset checks, and error handling for edge cases.
Updated check_and_prepare_trade_prices to use Operator group schedules for determining price data ranges and frequency, and to support price adjustment type. Refactored logic to handle multiple trading groups and improved price data selection based on timing and asset type. Updated related function calls to match new signature.
Simplified the merging and forward-filling of trade prices for multiple trading groups. The combined trade prices are now always aligned to the union of all group indices and forward-filled, ensuring consistency across all trading time points.
Enhanced the check_and_prepare_trade_prices function to better handle cases with both equity and fund shares. Added validation for price_adj, refined asset type detection, and improved logic for selecting and merging price data for different asset types.
Added comprehensive test cases for check_and_prepare_trade_prices, covering various asset combinations, operator timing/frequency, adjustment types, and error handling. Improved test data generation for stocks, indices, and funds, and ensured proper cleanup after tests.
Replaces force_match_freq and force_match_asset_type parameters with allow_ignore_freq, allow_ignore_asset_type, and allow_ignore_adj in infer_data_types and updates all usages accordingly. Improves clarity and flexibility in data type inference logic and updates related tests and function calls.
Enhanced trade price preparation in core.py to better handle multiple data types and missing columns. Updated history.py to clarify frequency matching and data type selection. Changed test cases to use allow_ignore_freq instead of force_match_freq. Fixed RuntimeError message handling in qt_operator.py. Adjusted index search in test_qt_run.py for type consistency and improved test output.
Enhanced get_reference_data_from_source to trim DataFrames by row_count and remove empty DataFrames. Updated and expanded unit tests for data type inference and reference data acquisition, including tests for allow_ignore_adj parameter.
Added a TODO in get_history_panel to refactor the data_types parameter to data_type_names, improving internal matching logic. Also removed a redundant TODO comment from the related test to clean up code.
Updated check_and_prepare_benchmark_data to infer start/end dates and frequency from Operator, and to select data type based on benchmark symbol. Improved docstrings and parameter names for clarity. Added TODOs for future enhancements in evaluation and reporting functions.
Marked the 'op_data_freq' property in Operator as deprecated. Updated test assertions in test_historypanel.py to check for timestamps with time component in DataFrame index.
Refactored TestCheckAndPrepareBenchmarkDataWithoutMock to remove inline test data creation and update parameter names for check_and_prepare_benchmark_data. Improved test cleanup by dropping additional tables and removing test directories. Updated strategy usage and operator instantiation for clarity and consistency.
Added comprehensive test cases for check_and_prepare_benchmark_data, covering stocks, indices, ETFs, and mutual funds with both daily and hourly frequencies. Tests now validate correct data integration, handling of missing and nonexistent symbols, and operator schedule edge cases. Randomized test data generation for all asset types is included to ensure robustness.
Introduces a new unit test to verify that check_and_prepare_benchmark_data returns a valid DataFrame for daily benchmark stock input, including checks for index type, columns, and absence of NaN values.
Replaced usage of group_schedules with group_timing_table for benchmark data preparation. Improved handling of empty DataFrames, added frequency-based time adjustment, and aligned benchmark data indices with operator timing table, including forward fill for missing values.
Tests now expect ValueError or RuntimeError to be raised when benchmark symbols are missing or operator schedules are out of range, instead of returning empty DataFrames. Added print statements for debugging index mismatches.
Import TIME_FREQ_STRINGS and add validation for run_freq (ensure string, extract main freq before '-' and verify against allowed TIME_FREQ_STRINGS). Simplify strategy addition by replacing kwargs popping with explicit type checks for run_freq/run_timing, remove direct assignment of strategy._group_id, and tweak an error message wording. These changes tighten input validation and simplify group/strategy matching in qteasy/qt_operator.py.
Refactor tests to stop setting strategy run_freq/run_timing directly and instead delegate those attributes to the Operator/Group. Updated tests in tests/test_operator_and_strategy.py, tests/test_strategy.py and tests/test_qt.py to add strategies through Operator.add_strategy or use Operator.set_parameter, removed redundant run_freq/run_timing assignments in custom strategy classes, and changed an expectation from ValueError to TypeError for invalid run_freq type. Also added a test asserting AttributeError when accessing run_freq/run_timing on a strategy not added to an Operator. Minor cleanup: removed explicit None parameters and adjusted test_qt to set a custom run_freq via Operator.
Streamline BaseStrategy internals and improve info/display behavior. Added str_to_list import and removed deprecated run_freq/run_timing fields and comments. Share tracking was refactored: _share_count initialization removed, update_shares now uses a keyword-only signature, accepts share_names (string or list), converts strings to lists, validates inputs, and generates placeholder names when only share_count is provided; share names are stored in _share_names. Adjusted group_id to return group.name or None, tightened extra_info type on info(), and updated printed parameter/data-type formatting (par_names = par_values, and dtype shown as "type x window"). Also applied a small fix to RuleIterator multi_pars access.
Make update_shares accept share_count as a positional parameter (remove forced keyword-only). Add a TypeError check to ensure share_count is an int and keep the existing ValueError for non-positive counts. share_names still takes precedence when provided.
Extend data-type update capabilities to support freq and asset_type changes across Operator and BaseStrategy. BaseStrategy.update_data_types now accepts freq and asset_type (scalar/list/tuple/dict) and replaces affected DataType entries while preserving _data_ids order and migrating ULC/WL values; conflicts merge into existing entries. Operator.set_parameter/add_strategy signatures and docs updated to pass freq/asset_type through to strategies. Also adjust strategy.update_shares call site to use the share_names keyword. Added unit tests covering single/dict updates, error conditions, and Operator integration.
Implement __repr__ for StgData to provide a concise, human-readable representation including name, freq, asset_type, window_length and use_latest_data_cycle (ULC). This aids debugging and logging by showing key attributes at a glance.
Prepare Broker for a v2.0 refactor by adding a set of abstract methods that define the common broker interface: log_in_broker, log_out_broker, get_positions, get_orders, get_results, submit_order, and withdraw_order (all raising NotImplementedError where appropriate). Bump the TODO note to v2.0. Also add a bool return type hint to _verify_trade_result and expand its docstring to document the returned boolean. These changes lay groundwork for standardizing broker subclasses (e.g., simulators, exchange adapters).
Unify parameter name data_type_ids -> data_type_id in Operator signatures, docstrings and update_data_types call. Add error handling when adding a strategy: wrap set_parameter in try/except to call remove_strategy(stg_id) and raise a RuntimeError if parameter setting fails, ensuring no partially-configured strategy remains. Update related comment to reflect rollback behavior. (file: qteasy/qt_operator.py)
Fix BaseStrategy share handling: share_count property now safely returns 0 if _share_names is None and otherwise returns len(_share_names). update_shares signature makes share_count optional (share_count: int = None) and the method now raises a ValueError if neither share_names nor share_count is provided. Keeps/improves existing type checks for share_names and share_count and updates docstring to mark share_count as optional. This prevents errors when the strategy hasn't been initialized with share names and allows updating by names only.
Fixes and enhancements to Operator.set_parameter and related behaviors:
- Correct KeyError message when indexing strategies to show the actual provided index.
- Mark data_type_id parameter as optional in the docstring.
- Differentiate par_values formats for RuleIterator strategies: treat {param_name: value} as kwargs (passed via **kwargs) and treat dicts with stock_ids/'default' as multi_par mappings; validate presence of at least one non-default stock_id for multi_par and raise a ValueError if only 'default' is provided.
- When changing run_freq/run_timing for a strategy that is the sole member of its group, attempt to merge it into an existing group with the same timing/frequency instead of always mutating the original group; otherwise update in place.
- Add comprehensive unit tests (tests/test_operator_and_strategy.py) that cover stg_id validation, pars input, opt_tag, data_types and window settings, par_values (tuples, kwargs, multi_par cases and error conditions), par_range validation, run_freq/run_timing moves and group merging, custom attribute setting, and combined parameter updates.
These changes improve robustness for parameter setting, clarify multi-stock parameter handling, prevent accidental group duplication, and increase test coverage for edge cases.
Append a final newline character to the end of tests/test_channels.py, tests/test_datatypes.py, and tests/test_fast_experiments.py to remove 'No newline at end of file' warnings and satisfy POSIX/end-of-file conventions.
Update examples and package exports to match new strategy/parameter APIs and runtime helpers. Changes include: - Use StgData/update_data_types for defining data/window settings instead of setting run_freq/data_freq/window_length directly (live_daily, live_rolling). - Switch Operator usage to add_strategy with per-strategy run/freq/window params and simplify Operator construction (live_example). - Migrate custom strategies to RuleIterator signatures and use self.get_data for price access; simplify base_grid rounding logic (live_grid, live_grid_multi). - Replace legacy parameter definitions with qt.Parameter lists and par_values/allow_multi_par APIs; provide par_values dict for multi-stock grids (live_grid, live_grid_multi). - Adjust operator run configuration to use run_freq/run_timing and call qt.run(op) instead of op.run() across examples. - Move delete_account import location and minor code cleanups (live_grid_multi). - Export Parameter from qteasy and add it to __all__ so examples can import it. These updates align examples with the new qteasy Parameter/StgData/Operator interfaces and improve multi-asset grid handling.
Import StgData and switch SelectingNDayRateChange to use StgData('close', freq='d', asset_type='ANY', window_length=150'), removing the separate window_length kwarg so the strategy data carries its required window. Also make Operator's display use data_freq_name.get(group.run_freq, group.run_freq) to avoid KeyError when a run_freq is not present in the mapping, falling back to the raw frequency string.
Extend TestOperatorSetParameter with unit tests covering multi_par handling: ensure dict key order does not affect parsing (shares order wins), multiple unspecified shares use the 'default' value, setting multi_par before set_shares raises, value-length mismatches raise, and tuple/list par_values are treated as regular positional params (not multi_par). Also clarify docstring about multi_par length matching share_count.
Import Parameter and update Operator.set_parameter type annotations to accept Parameter instances, lists and dicts for various parameters (pars, window_length, use_latest_data_cycle, par_values). This broadens accepted input shapes (including dict mappings) for more flexible parameter configuration. Also apply a minor formatting tweak in a test assertion message.
Set default run_freq='d' and run_timing='close' in Group.__init__ and add validation to ensure both are non-None strings. Prevent clearing a group's run_freq/run_timing when removing members or clearing the group. In Operator, normalize run_freq/run_timing defaults when None and tighten type checks (raise ValueError for non-strings) so Group remains the single source of truth for scheduling. In BaseStrategy, stop popping deprecated run_freq/run_timing from kwargs so legacy keys are preserved for upstream handling. These changes centralize scheduling config in Group and harden input validation.
Update tests to require Group to be constructed with valid run_freq and run_timing and to propagate those values to member strategies. Tests now assert Group.run_freq/run_timing are set at creation, prohibit missing/None values, and verify strategies inherit/track group frequency/timing (including when moved between groups). Adjust related operator and QT tests to construct Group instances with run_freq/run_timing instead of mutating internals.
Remove deprecated run_freq/run_timing parameters and update caller code accordingly. - Deleted run_freq attributes from several built-in strategy classes in qteasy/built_in.py. - Simplified Group.add_strategy signature to remove run_freq/run_timing and removed related validation logic. - Updated Operator to stop passing run_freq/run_timing when adding strategies and to handle group changes using Group's delegated run settings. - Changed input validation errors for run_freq/run_timing to raise TypeError instead of ValueError. - Added traceback printing when strategy addition fails to aid debugging. - Removed the unused **kwargs parameter from BaseStrategy.__init__ and cleaned up deprecated run_freq/run_timing handling. - Minor formatting and whitespace cleanups in qteasy/qt_operator.py (dict comprehensions, alignment) and a newline fix for raising NotImplementedError.
Adapt tests to recent API updates: remove explicit run_freq/run_timing from many strategy constructors (passing freq/timing via add_strategy/group instead), fix broken GenStg instantiations, and allow adding strategies with differing freq/timing in Group tests. Adjust test assertions and parameter names: change set_parameter arg name from data_type_ids to data_type_id, update some dtype_id expectations (close_E_d -> close_E_h), and add a check for update_data_types window_length. Minor formatting fixes (newline at EOF).
Document that run_freq/run_timing are managed by Group and update usage notes in BaseStrategy. Change RuleIterator.update_par_values to treat multi_par only when the first arg is a dict (tuple/list must not be interpreted as multi_par). Rewrite _update_multi_pars to accept only dicts, require share_count>0, resolve per-share parameter tuples in the order of share_names (with optional 'default' fallback), validate types and tuple lengths against par_count, and store multi_pars as a tuple aligned to share_names. Update tests to reflect the new behavior and assumptions (ordering, errors before set_shares, and tuple/list not being treated as multi_par).
Ignore the .cursor file at the repository root by adding '/.cursor' to .gitignore to prevent it from being accidentally committed.
Update the docstring for Group.add_strategy to state that a strategy's run_freq and run_timing are delegated from the group and that the group is the single source of truth. Removed redundant ValueError entries about mismatched run frequency/timing. No functional code changes.
Make strategy timing/frequency explicit in tests and fix delivery call usage. Removed hardcoded run_timing/run_freq from several strategy class definitions and set them via Operator.add_strategy / Operator.set_parameter in tests (TestFastExperiments, TestQT, TestQTRun) so groups/strategies are configured explicitly. Updated Operator construction pattern in TestQT to use add_strategy and preserve per-strategy run_timing/run_freq. Added stg_rule.update_shares call in TestOperatorAndStrategy. Fixed process_account_delivery calls in TestTradingUtilFuncs to unpack delivery_config via **delivery_config. These changes clarify test configuration and fix incorrect function argument usage.
Improve handling and validation of per-share parameters in RuleIterator and update tests accordingly. - RuleIterator: iterate share_names in reverse order, require explicit per-share par_values or a default (raise KeyError if missing), enforce tuple/list type and correct length with clearer English error messages, and call update_par_values(*par_tuple) to validate/apply the first par_tuple to strategy parameters. - Tests: adjust tests to assert that par_values follow the latest multi_par entry, add multiple assertRaises cases for invalid multi_pars (wrong type, missing keys, wrong length, out-of-range or wrong dtype), and adapt a strategy test to use update_shares plus keyed multi_pars input. These changes tighten validation of multi_pars input and make failures explicit in tests.
Fix a loop that previously iterated self.share_names in reverse (using [::-1]). The iteration now uses the natural order of self.share_names so parsing and result assembly follow the intended share_names sequence when selecting values from multi_pars, preventing incorrect ordering-related behavior.
Change get_account_positions to return a single reusable empty canonical DataFrame and guard against empty/malformed positions (missing 'symbol'). In Trader, iterate the daily schedule in reverse to avoid index shifts when popping scheduled tasks and normalize task tuple handling. Add many unit tests and test fixtures (helpers, account/order/result setup) to exercises Trader initialization, properties, task handling, whitelist behavior, account/orders/logging, info APIs, boundary cases, and asset pool details; also adjust test imports and a few existing test setups.
Introduce _write_minimal_stock_daily helper to write minimal daily bar data (ts_code, trade_date, open, high, low, close, pre_close, change, pct_chg, vol, amount) covering a date range with >=30 rows per symbol to tests/test_trader.py. Update test_task_run_strategy_step_index_zero to set force_current_date, populate stock_daily via the new helper, and inject a live_price DataFrame so the strategy run no longer depends on external live price fetches. Remove an unused commented broker config block and lightly reformat a long docstring for readability. These changes make the run_strategy test deterministic and self-contained.
When real-time data download returns empty, create a placeholder live_price DataFrame if no valid live_price exists. The code derives symbol list from asset_pool (supports string via utilfuncs.str_to_list or iterable), builds a DataFrame indexed by symbols with price set to NaN, and assigns it to self.live_price to avoid downstream failures when live prices are unavailable.
Update unit tests to be deterministic across market states and reflect multi-share parameter behavior. Trader tests now check ts.status and broker.status conditionally based on is_market_open instead of assuming a fixed initial state. Strategy test expectations updated to the last stock's parameters after _update_multi_pars. QT tests: remove redundant run_timing default from StgSelClose, set shares on operators in the loop, and reorder adding strategies so group run_freq/run_timing are applied correctly.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
qteasy 2.0
qteasy 2.0 contains multiple significant improvements and new features:
New Features
with some of above modifications unfortunately qteasy 2.0 is not compatible with qteasy 1.X anymore, but the changes in the previous trading strategies is limited. warning information will be given in qteasy version 1.5 and further, however, no significant functional improvement will be done after 1.4.9, it is strongly recommended for all users to switch to qteasy 2.0, since qteasy 2.0 is far more superior, and all further improvements will only be implemented on qteasy 2.0, including:
Further improvements on 2.0: