Recurring stream retrieval misses liability streams, newly created merchants, and pending items — switch to aggregated query

[krisrowe]

Problem

The SDK's recurring stream retrieval uses Web_GetUpcomingRecurringTransactionItems, which silently excludes significant categories of streams:

• Credit report liability streams — credit cards, mortgages, HELOCs sourced from credit bureau data (see Support credit report liability streams in list_recurring #2)
• Newly created merchant streams — merchants flagged as recurring but whose transactions have isRecurring: false (e.g., after renaming transactions to a new merchant)
• Pending streams — detected by Monarch but not yet approved
• Inactive/stale streams — exist in the catalog but have no scheduled occurrences

The raw catalog query returns ~100 streams while the current retrieval returns ~54 — nearly half are invisible.

Monarch UI tabs and which queries they use

Analysis of the Monarch Recurring UI's API traffic across multiple tabs:

Calendar tab

• Common_GetRecurringStreams (includeLiabilities: true) — full catalog, no date range, one entry per stream
• Common_GetAggregatedRecurringItems (single month, e.g., 2026-04-01 to 2026-04-30) — payment status grouped by complete/upcoming

Monthly tab

• Common_GetAggregatedRecurringItems (single month, 2026-03-01 to 2026-03-31) — 55 complete + 24 upcoming = 79 items for March
• Common_GetSpinwheelCreditReport — credit report data (40 liability accounts)
• RecurringMerchantSearch — merchant search index (93 streams)

All tab

• Common_GetAllRecurringTransactionItems (includeLiabilities: true, includePending: true, no date range) — 108 streams (full catalog including liabilities)
• RecurringMerchantSearch — same merchant search (93 streams)

Not observed in any captured tab

• Web_GetUpcomingRecurringTransactionItems — this is what our SDK uses. The "Web_" prefix indicates it IS a Monarch web UI query, likely used by a dashboard widget or view we didn't capture. But it is NOT used by the Calendar, Monthly, or All tabs under default conditions. It may appear under specific filter scenarios, or it may simply be an available API endpoint regardless of current UI usage.

Stream counts across queries

Query	Streams	Notes
Web_GetUpcomingRecurringTransactionItems (SDK today)	~54	Missing liabilities, new merchants, pending
RecurringMerchantSearch	93	Merchant-based streams only (no liabilities)
Common_GetAggregatedRecurringItems (March)	79 items	Single month, includes liabilities, has payment status
Common_GetAllRecurringTransactionItems (All tab)	108	Full catalog, includes liabilities, no date range

Account association accuracy

The aggregated query returns the correct account for each stream, matching what the UI shows. The items query (Web_GetUpcomingRecurringTransactionItems) derives account from isRecurring: true transactions, which can be wrong when transactions have been moved between merchants. Confirmed: State Farm (Auto) showed on the wrong account in list_recurring but correct account in the aggregated query and Monarch UI.

Five Monarch GraphQL queries for recurring data

Web_GetUpcomingRecurringTransactionItems (what SDK uses today)

Returns one row per expected payment per stream within a date range. For example, a monthly mortgage queried over 12 months produces 12 rows — one per month. Each row has:

• date — when the payment was expected
• transactionId — non-null if Monarch found a matching transaction (i.e., it was paid)
• isPast — whether the date has passed
• amount — actual payment amount (if paid)
• category, account — for the payment

Unique strength: Proven to work with multi-month ranges (trailing 12 months). Returns all streams' expected payments in a single API call (~500+ rows), enabling last_paid_date derivation across months.

Weaknesses: No liability streams, no isLate/isCompleted flags, no minimum payment amounts. Account association is derived from transaction linking and can be wrong.

Common_GetRecurringStreams (catalog — used by Calendar tab)

Returns one entry per stream regardless of date range. Includes every stream type: merchant-based, credit report liabilities, pending, inactive. Each entry has:

• frequency, amount, baseDate, isActive, reviewStatus, recurringType
• creditReportLiabilityAccount with account link, balance, status

Unique strength: The only source for streams that have no scheduled occurrences (newly created, inactive). No date range means no gaps.

Does not include: any payment status or verification data.

Common_GetAggregatedRecurringItems (used by Calendar + Monthly tabs)

Returns one row per expected payment in a date range, grouped by status (complete/upcoming):

• Everything Web_GetUpcomingRecurringTransactionItems has (date, transactionId, isPast, amount, category, account)
• Plus: isLate, isCompleted, markedPaidAt
• Plus for liabilities: liabilityStatement with minimumPaymentAmount, paymentsInformation (status: paid/unpaid/partially_paid, remainingBalance, actual payment transactions with amounts and dates)
• Correct account associations (matches UI, unlike items query)

Unique strength: Richest data source. Only source for liability payment details, isLate/isCompleted flags, and minimum payment amounts.

Unknown: Whether it supports multi-month date ranges. UI only calls it with single-month ranges.

Common_GetAllRecurringTransactionItems (used by All tab)

Uses the same recurringTransactionStreams endpoint as Common_GetRecurringStreams but with different parameters:

• includeLiabilities: true, includePending: true (from variables, not hardcoded)
• No date range — returns the full catalog
• 108 streams in test data

Returns per-stream: id, frequency, amount, isApproximate, isActive, reviewStatus, recurringType, baseDate, merchant, creditReportLiabilityAccount, plus nextForecastedTransaction (date + amount).

Difference from Common_GetRecurringStreams: May include nextForecastedTransaction and return slightly different fields. Needs comparison to determine if they're interchangeable or complementary.

RecurringMerchantSearch (used by Monthly + All tabs)

Returns 93 streams — merchant-based only (no liabilities). Powers the "Find recurring merchants in your accounts" feature in the Monarch UI, which triggers a re-scan of transaction history to detect new recurring patterns.

UI behavior: Can only be triggered once every 60 minutes in the UI. Unknown whether this rate limit applies to direct API calls. The query appears to be read-only (checking scan status), but the act of calling it may trigger a server-side scan.

Observed impact: After renaming transactions to new merchants and triggering this feature, Monarch did NOT reassociate existing streams with correct accounts. It may only detect new recurring patterns, not fix existing stream-to-account associations.

Potential skill relevance: After the agent renames transactions and flags merchants as recurring, it may need to advise the user to trigger this feature (or trigger it via API if possible) to help Monarch detect the new patterns. Needs investigation: is there a mutation to trigger the scan, and does the 60-minute rate limit apply to API consumers?

Liability Account Mapping UI (/recurring/map-liabilities)

The Monarch UI has a "Manage your synced accounts" option under Recurring that navigates to a "Link to your accounts in Monarch" page. This page presents a table with all credit accounts from the user's credit report:

Column	Description
Credit report account	Account name from credit bureau data
Balance	Current balance
Payment due date	Editable/selectable
Monarch Account	Editable/selectable — must be set to either a linked Monarch account or "Ignore this account"

This is how liability streams get associated with Monarch accounts. Each row represents a credit report account that Monarch discovered via Spinwheel. The user maps each one to a Monarch account (or ignores it), which creates the link that the aggregated query uses for payment verification.

Relevance: If a liability stream shows no account association or the wrong account, this mapping page is where the user fixes it. The bills-agent skill may need to direct the user here when liability streams aren't linked correctly.

Documentation deliverable

All analysis captured in this issue — query comparisons, UI tab mappings, behavior, liability stream mechanics, account association accuracy — must be transferred to (for implementation context) and (for user-facing tool documentation) as part of the implementation work. A coding agent working on this repo will use those files as context and needs this research available without reading GitHub issues.

How last_paid_date is derived today

last_paid_date is not a field returned by any Monarch API. It is computed in collapse_to_streams():

1. The SDK calls Web_GetUpcomingRecurringTransactionItems with a trailing 12-month date range
2. This returns all streams' expected payments in one call — multiple rows per stream, one per month
3. Each row has a transactionId that is non-null if Monarch matched a real transaction to that expected payment
4. The code scans all rows for a given stream and picks the latest date where transactionId is not None
5. That date becomes last_paid_date

This derivation depends on the query returning multiple months of data per stream in a single call.

isRecurring transaction flag behavior

When Monarch links a transaction to a recurring stream occurrence, the transaction gets isRecurring: true. Observed behavior:

• Updating a transaction's merchant via update_transaction resets isRecurring to false in most cases
• Fresh transactions synced from the bank that match a recurring merchant get isRecurring: true automatically
• Exception observed: A transaction that was moved from one merchant to another merchant that ALSO had an active recurring stream on the same account retained isRecurring: true
• Mixed flags on same merchant: Same merchant with consistent $2,100 monthly payments had 5 out of 6 transactions as isRecurring: true and 1 as false — Monarch's per-transaction matching is not 100% deterministic

This flag affects which query returns what — Web_GetUpcomingRecurringTransactionItems only links transactions with isRecurring: true to stream occurrences. The aggregated query appears to use the authoritative stream definition instead, which is why it shows correct account associations even when transaction flags are wrong.

The likely outcome is a combination of sources

No single query provides everything. The final implementation will likely combine multiple sources, chosen based on experimentation results:

Need	Best source(s)	Why
"Is this month's bill paid?"	Common_GetAggregatedRecurringItems (current month)	Has isCompleted, isLate, and liability payment status
"When was this last paid?"	Web_GetUpcomingRecurringTransactionItems (trailing 12mo)	Proven multi-month support, enables last_paid_date scan
"What streams exist?" (complete list)	Common_GetAllRecurringTransactionItems or Common_GetRecurringStreams	Catches streams with no scheduled occurrences
"What's the minimum CC payment?"	Common_GetAggregatedRecurringItems	Only source with liabilityStatement
"Is this stream active/pending?"	Common_GetRecurringStreams / Common_GetAllRecurringTransactionItems	Has isActive, reviewStatus

Possible combination strategies:

• Aggregated + Catalog: Aggregated for current month payment status and liability data, catalog for complete stream list. last_paid_date derived from aggregated if multi-month works, otherwise from transaction search.
• Aggregated + Items + Catalog: All three — aggregated for liability/status data, items for trailing 12-month last_paid_date, catalog for completeness. Most complete but most API calls.
• Aggregated + Catalog + transaction fallback: Aggregated for current month, catalog for completeness, list_transactions by merchant_id for last_paid_date on streams where needed. Avoids the items query entirely but adds per-stream transaction lookups.

The right combination depends on experimentation results (see below).

Open questions requiring experimentation

Before committing to an implementation, the following must be tested against the live Monarch API:

Multi-month range support for Common_GetAggregatedRecurringItems

The Monarch UI only calls this query with single-month ranges. Can it accept a multi-month or trailing 12-month range?

• If yes: it may replace Web_GetUpcomingRecurringTransactionItems entirely (aggregated + catalog = complete solution)
• If no: the items query remains necessary for last_paid_date derivation, and we use a combination

Performance with wider ranges

Even if multi-month ranges are accepted, do they return excessive data? The single-month response is already ~94KB with ~79 items. A 12-month range could be significantly larger.

Common_GetAllRecurringTransactionItems vs Common_GetRecurringStreams

Both return the full catalog without a date range. Are they interchangeable? Does one return fields the other doesn't (e.g., nextForecastedTransaction)? Can we use just one?

Completeness comparison

Do all streams returned by Web_GetUpcomingRecurringTransactionItems also appear in Common_GetAggregatedRecurringItems for the same date range? Are there any streams one returns that the other does not?

Interface considerations

The tool interface should allow callers to choose the right balance of completeness vs performance:

• Default: current month payment status (fast, answers "is everything paid this month?")
• Optional: include historical months for last_paid_date (may require additional API calls)
• Optional: include/exclude liability streams
• Optional: include/exclude pending/inactive streams

The implementation should use the most efficient combination of underlying sources to satisfy the caller's request, without exposing which queries are used under the hood.

Existing code

Queries already defined in queries.py but not fully wired:

• Common_GetRecurringStreams — defined at line 380, used only internally in _find_merchant_for_stream()
• Common_GetAggregatedRecurringItems — defined at line 420, not used anywhere
• Common_GetAllRecurringTransactionItems — need to verify if captured in queries.py

Recommended approach

1. Experiment — test Common_GetAggregatedRecurringItems with multi-month ranges; compare Common_GetAllRecurringTransactionItems vs Common_GetRecurringStreams; test completeness vs Web_GetUpcomingRecurringTransactionItems
2. Design — based on results, choose the right combination of sources for each use case
3. Implement — wire up the chosen combination behind a clean interface, keeping all queries available internally
4. Test — verify no streams are lost compared to current behavior; verify last_paid_date is accurate or adequately replaced; verify account associations are correct
5. Document — update tool descriptions to reflect new capabilities and interface options

Related

• Support credit report liability streams in list_recurring #2 — Credit report liability stream acceptance criteria (delivered as part of this change)
• Merchant logo management and split-merchant workflow documentation #3 — Parent tracking issue
• Add merchant management tools (get_merchant, list_merchants, update_merchant) #4 — Merchant management tools (cross-reference in tool docs)
• Add balance history download and upload tools #7 — Balance history download/upload tools