Inefficient HTTP client creation - new client per request defeats connection pooling

[user1303836]

Problem

Multiple adapters and services create a new httpx.AsyncClient() for each request instead of reusing a shared client. This defeats HTTP connection pooling and increases latency.

Affected Locations:

File	Pattern
src/intelstream/adapters/rss.py	Creates client in fetch_latest()
src/intelstream/adapters/arxiv.py	Creates client in multiple methods
src/intelstream/adapters/page.py	Creates client in fetch_latest()
src/intelstream/adapters/substack.py	Creates client per request
src/intelstream/adapters/strategies/*.py	Each strategy creates its own client
src/intelstream/services/content_extractor.py	Creates client when none provided

Example Pattern

# Current - inefficient
async def fetch_latest(self, identifier: str, feed_url: str | None) -> list[ContentItem]:
    async with httpx.AsyncClient() as client:  # New connection pool each time
        response = await client.get(url)

Impact

• Performance: Each request does fresh DNS lookup and TCP handshake
• Latency: Connection reuse would save ~50-100ms per request
• Resource waste: Repeated client creation/teardown
• Rate limiting risk: New connections may trigger rate limits on some servers

Proposed Solution

1. Accept injected client with proper lifecycle:

class RSSAdapter:
    def __init__(self, http_client: httpx.AsyncClient | None = None):
        self._http_client = http_client
        self._owns_client = http_client is None
    
    async def fetch_latest(self, ...) -> list[ContentItem]:
        client = self._http_client or httpx.AsyncClient(timeout=30.0)
        try:
            # use client
        finally:
            if self._owns_client:
                await client.aclose()

2. Create shared client in pipeline:

class ContentPipeline:
    async def __aenter__(self):
        self._http_client = httpx.AsyncClient(timeout=30.0)
        return self
    
    async def __aexit__(self, ...):
        await self._http_client.aclose()

3. Pass client to all adapters from a central location

Labels

enhancement, performance