Skip to content

better tokenpool downloading#56

Open
GelatoGenesis wants to merge 1 commit intomainfrom
b-17733901733
Open

better tokenpool downloading#56
GelatoGenesis wants to merge 1 commit intomainfrom
b-17733901733

Conversation

@GelatoGenesis
Copy link
Copy Markdown
Collaborator

@GelatoGenesis GelatoGenesis commented Mar 13, 2026

Summary by CodeRabbit

  • New Features

    • Added configurable timeout settings for Seize metadata and Arweave download operations via environment variables.
    • Implemented fallback gateway support for improved resilience during data retrieval.
    • Enhanced contextual logging for better operation tracking and diagnostics.
  • Bug Fixes

    • Improved error handling with timeout-aware recovery and fallback mechanisms.
  • Tests

    • Added tests for timeout scenarios and fallback behavior.

Signed-off-by: GelatoGenesis <tarmokalling@gmail.com>
@sonarqubecloud
Copy link
Copy Markdown

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 13, 2026

📝 Walkthrough

Walkthrough

This pull request introduces timeout configuration for Seize metadata and Arweave downloads, adds an execution context service for managing contextual logging across operations, implements a Seize API patching mechanism with timeout handling and multi-gateway fallback logic, and integrates these components throughout the allowlist and token pool layers.

Changes

Cohort / File(s) Summary
Configuration
README.md
Added environment variables ALLOWLIST_SEIZE_METADATA_TIMEOUT_MS and ALLOWLIST_ARWEAVE_DOWNLOAD_TIMEOUT_MS to document new timeout configuration options.
Execution Context Service
src/allowlist-lib/allowlist-lib-execution-context.service.ts
Introduced AllowlistLibExecutionContextService using AsyncLocalStorage to manage execution context (tokenPoolId, contract, blockNo, consolidateBlockNo) with methods to run code within context and generate context-aware log prefixes.
Log Listener Integration
src/allowlist-lib/allowlist-lib-log-listener.service.ts, src/allowlist-lib/allowlist-lib-log-listener.service.spec.ts
Integrated AllowlistLibExecutionContextService into AllowlistLibLogListener; all log methods now prefix messages with execution context information via new withContext helper; added test verifying context-prefixed log output.
Seize API Timeout Patch
src/allowlist-lib/allowlist-lib-seize-timeout-patch.ts, src/allowlist-lib/allowlist-lib-seize-timeout-patch.spec.ts
Implemented timeout-aware Seize API patching with parseTimeoutMs utility, patchAllowlistCreatorSeizeApi function supporting configurable metadata and download timeouts, CSV parsing with fallback across Arweave gateways, and comprehensive error handling; added extensive tests covering timeout scenarios and fallback flows.
Module Integration
src/allowlist-lib/allowlist-lib.module.ts
Registered AllowlistLibExecutionContextService as provider, exported it from module, instantiated loggerFactory, and invoked patchAllowlistCreatorSeizeApi with timeout configuration derived from environment variables (defaults: 10000ms metadata, 30000ms downloads).
Token Pool Layer Integration
src/token-pool/token-pool-downloader.service.ts, src/token-pool/token-pool-downloader.service.spec.ts
Integrated AllowlistLibExecutionContextService into TokenPoolDownloaderService; wrapped allowlistCreator.execute within execution context containing pool metadata; added toTokenPoolExecutionError helper for standardized error formatting; added test verifying timeout error propagation and state transitions.

Sequence Diagram(s)

sequenceDiagram
    participant Client as TokenPool<br/>Downloader
    participant ExecCtx as Execution<br/>Context
    participant SeizeAPI as Seize<br/>API
    participant Metadata as Metadata<br/>Fetch
    participant Gateway as Arweave<br/>Gateway
    participant CSV as CSV<br/>Parser

    Client->>ExecCtx: run(context, callback)
    ExecCtx->>SeizeAPI: getDataForBlock()
    SeizeAPI->>Metadata: fetchJson(metadata)
    Note over Metadata: Apply timeout<br/>(seizeMetadataTimeoutMs)
    
    alt Metadata Success
        Metadata-->>SeizeAPI: uploads page
        SeizeAPI->>SeizeAPI: select closest TDH URL
        SeizeAPI->>Gateway: downloadAndParseCsvWithFallback()
        Note over Gateway: Apply timeout<br/>(arweaveDownloadTimeoutMs)
        
        alt Gateway Success
            Gateway->>CSV: fetch & parse CSV
            CSV-->>SeizeAPI: parsed records
            SeizeAPI-->>Client: data
        else Gateway Timeout
            Gateway-->>Gateway: log timeout, try fallback
            Gateway->>Gateway: next gateway
            Gateway->>CSV: retry fetch & parse
            CSV-->>SeizeAPI: parsed records
            SeizeAPI-->>Client: data
        end
    else Metadata Timeout
        Metadata-->>SeizeAPI: timeout error
        SeizeAPI-->>Client: error with context
    end
    
    Client->>ExecCtx: error handled with log prefix
Loading

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 Hop hop, through timeouts we go!
Seize the data, watch it flow—
Fallback gateways, context true,
Timeouts handled, fresh and new!
One pool's journey, logs aligned,
A feature crystalline, refined.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Title check ❓ Inconclusive The title 'better tokenpool downloading' is vague and generic, using non-descriptive language that doesn't convey the specific changes made in this PR. Consider a more specific title that highlights the main changes, such as 'Add execution context and timeout handling for tokenpool downloader' or 'Implement Seize API timeout patches and contextual logging'.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch b-17733901733
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (5)
src/allowlist-lib/allowlist-lib-seize-timeout-patch.ts (2)

7-16: Consider using a Set for content-type lookup.

Set.has() is more efficient than Array.includes() for membership checks, especially as the list grows.

Suggested fix
-const CSV_CONTENT_TYPES = [
+const CSV_CONTENT_TYPES = new Set([
   'text/csv',
   'application/csv',
   'application/x-csv',
   'text/x-csv',
   'text/comma-separated-values',
   'application/vnd.ms-excel',
   'text/plain',
   'application/octet-stream',
-];
+]);

Then update the check:

-  return CSV_CONTENT_TYPES.includes(normalized);
+  return CSV_CONTENT_TYPES.has(normalized);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/allowlist-lib/allowlist-lib-seize-timeout-patch.ts` around lines 7 - 16,
Replace the CSV_CONTENT_TYPES array with a Set to optimize membership checks:
change the constant named CSV_CONTENT_TYPES to be a Set of the same strings and
update any lookup logic to use CSV_CONTENT_TYPES.has(contentType) instead of
Array.includes; ensure usages referenced in this file or elsewhere (calls
checking CSV_CONTENT_TYPES) are updated to call .has on the Set.

229-234: Prefer findLast() over filter().at(-1).

findLast() is more efficient as it stops at the first match from the end, avoiding the intermediate array allocation.

Suggested fix
 function getClosestTdh(apiResponseData: SeizeUploadsPage, blockId: number) {
   return [...apiResponseData.data]
     .sort((a, b) => a.block - b.block)
-    .filter((item) => item.block <= blockId)
-    .at(-1)?.url;
+    .findLast((item) => item.block <= blockId)?.url;
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/allowlist-lib/allowlist-lib-seize-timeout-patch.ts` around lines 229 -
234, The getClosestTdh function currently sorts a copied array then uses
filter().at(-1), which allocates an intermediate array; replace the
.filter((item) => item.block <= blockId).at(-1) chain with .findLast(item =>
item.block <= blockId) on the sorted copy to stop at the first match from the
end and avoid the extra allocation (i.e. keep the
[...apiResponseData.data].sort((a,b)=>a.block-b.block) but call .findLast(...)
and then ?.url).
src/token-pool/token-pool-downloader.service.ts (1)

365-365: Use this.logger.error instead of console.error for consistency.

The rest of the service uses the NestJS Logger instance (this.logger), but this line uses console.error. Using the class logger maintains consistent formatting and log routing.

Suggested fix
-      console.error(error.message, e);
+      this.logger.error(error.message, e);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/token-pool/token-pool-downloader.service.ts` at line 365, Replace the
direct console.error call with the class Logger: in the
TokenPoolDownloaderService method where you currently call
console.error(error.message, e) (search for console.error in
token-pool-downloader.service or inside methods like downloadTokens /
handleDownloadError), change it to this.logger.error with a descriptive message
and include the error object/stack (e.g., this.logger.error(`Failed to ...:
${error.message}`, e?.stack || e)) so logs use the NestJS logger instance and
preserve error details for debugging.
src/allowlist-lib/allowlist-lib-seize-timeout-patch.spec.ts (1)

21-21: Consider using no-op arrow functions to silence ESLint.

ESLint flags the empty method bodies. Using arrow function syntax with explicit undefined can clarify intent.

Suggested fix
-  debug = (): void => {};
+  debug = (): void => undefined;

-  warn = (): void => {};
+  warn = (): void => undefined;

Also applies to: 31-31

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/allowlist-lib/allowlist-lib-seize-timeout-patch.spec.ts` at line 21,
Replace the empty method bodies used as no-ops (e.g. the "debug = (): void =>
{}" occurrence and the other similar no-op in this spec) with explicit arrow
no-ops that return undefined (e.g. "debug = (): void => undefined;") so ESLint
recognizes intent; update both occurrences of the no-op assignment in this file
to return undefined instead of using an empty block.
src/allowlist-lib/allowlist-lib-execution-context.service.ts (1)

2-2: Use node: prefix for built-in modules.

The node: prefix is the recommended way to import Node.js built-in modules, providing clearer intent and avoiding potential conflicts with npm packages.

Suggested fix
-import { AsyncLocalStorage } from 'async_hooks';
+import { AsyncLocalStorage } from 'node:async_hooks';
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/allowlist-lib/allowlist-lib-execution-context.service.ts` at line 2,
Replace the plain import of the built-in AsyncLocalStorage module with the
Node-prefixed specifier: change the import source used for AsyncLocalStorage to
'node:async_hooks' (i.e., update the import statement that references
AsyncLocalStorage) so the code explicitly uses the Node.js built-in module.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/allowlist-lib/allowlist-lib-execution-context.service.ts`:
- Line 2: Replace the plain import of the built-in AsyncLocalStorage module with
the Node-prefixed specifier: change the import source used for AsyncLocalStorage
to 'node:async_hooks' (i.e., update the import statement that references
AsyncLocalStorage) so the code explicitly uses the Node.js built-in module.

In `@src/allowlist-lib/allowlist-lib-seize-timeout-patch.spec.ts`:
- Line 21: Replace the empty method bodies used as no-ops (e.g. the "debug = ():
void => {}" occurrence and the other similar no-op in this spec) with explicit
arrow no-ops that return undefined (e.g. "debug = (): void => undefined;") so
ESLint recognizes intent; update both occurrences of the no-op assignment in
this file to return undefined instead of using an empty block.

In `@src/allowlist-lib/allowlist-lib-seize-timeout-patch.ts`:
- Around line 7-16: Replace the CSV_CONTENT_TYPES array with a Set to optimize
membership checks: change the constant named CSV_CONTENT_TYPES to be a Set of
the same strings and update any lookup logic to use
CSV_CONTENT_TYPES.has(contentType) instead of Array.includes; ensure usages
referenced in this file or elsewhere (calls checking CSV_CONTENT_TYPES) are
updated to call .has on the Set.
- Around line 229-234: The getClosestTdh function currently sorts a copied array
then uses filter().at(-1), which allocates an intermediate array; replace the
.filter((item) => item.block <= blockId).at(-1) chain with .findLast(item =>
item.block <= blockId) on the sorted copy to stop at the first match from the
end and avoid the extra allocation (i.e. keep the
[...apiResponseData.data].sort((a,b)=>a.block-b.block) but call .findLast(...)
and then ?.url).

In `@src/token-pool/token-pool-downloader.service.ts`:
- Line 365: Replace the direct console.error call with the class Logger: in the
TokenPoolDownloaderService method where you currently call
console.error(error.message, e) (search for console.error in
token-pool-downloader.service or inside methods like downloadTokens /
handleDownloadError), change it to this.logger.error with a descriptive message
and include the error object/stack (e.g., this.logger.error(`Failed to ...:
${error.message}`, e?.stack || e)) so logs use the NestJS logger instance and
preserve error details for debugging.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 812d29db-cc9e-46ac-be0d-ec1a6a516334

📥 Commits

Reviewing files that changed from the base of the PR and between 4601d1a and ddf5ba7.

📒 Files selected for processing (9)
  • README.md
  • src/allowlist-lib/allowlist-lib-execution-context.service.ts
  • src/allowlist-lib/allowlist-lib-log-listener.service.spec.ts
  • src/allowlist-lib/allowlist-lib-log-listener.service.ts
  • src/allowlist-lib/allowlist-lib-seize-timeout-patch.spec.ts
  • src/allowlist-lib/allowlist-lib-seize-timeout-patch.ts
  • src/allowlist-lib/allowlist-lib.module.ts
  • src/token-pool/token-pool-downloader.service.spec.ts
  • src/token-pool/token-pool-downloader.service.ts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant