refactor: extract base HTTP resource from repetitive requests.Session wrappers

[cbini]

Problem

10 active ConfigurableResource classes independently implement the same requests.Session wrapper pattern, duplicating session lifecycle, URL construction, request wrapping, error handling, and auth setup. This makes it hard to add cross-cutting improvements (retries, rate limiting, metrics) and increases the cost of adding new API integrations.

Approach

Hook-based single-inheritance base class (BaseHTTPResource) in src/teamster/libraries/http/resources.py. Each subclass overrides only what it needs. The base provides built-in retry via tenacity, standardized error handling with rate-limit awareness, and pagination helpers for three common patterns.

Active resources (10)

Resource	Auth	Pagination	Tier
SmartRecruitersResource	X-SmartToken header	None	1
PowerSchoolEnrollmentResource	Basic auth	page	1
OvergradResource	ApiKey header	offset	1
KnowBe4Resource	Bearer header	page	1
ZendeskResource	HTTPBasicAuth	cursor	2
FinalsiteResource	JWT bearer	cursor	2
CoupaResource	OAuth2Session	offset	2
GrowResource	OAuth2 token fetch	offset	2
AdpWorkforceNowResource	OAuth2Session + mTLS	offset	3
DeansListResource	API key per school	offset	3

Dead resources (4, preserved untouched)

AdpWorkforceManagerResource, MClassResource, DibelsDataSystemResource, CouchdropResource — no active consumers.

Base class design

Extension hooks

Hook	Default behavior	Override example
_setup_session()	No-op (subclass sets auth + _base_url)	Every subclass
_prepare_request(method, url, kwargs)	Pass-through	DeansList injects school_id
_handle_error(response, error)	429: sleep + retry; 401: re-auth + retry; 5xx: retry; other: raise	Zendesk custom rate-limit headers
_get_retry_after(response)	Parses Retry-After / X-RateLimit-Reset headers	ADP WFN computed delay
_reauthenticate()	Re-raises (no re-auth)	OAuth2 token refresh flows

Built-in retry (tenacity)

• stop_after_attempt(3)
• wait_exponential_jitter(initial=1, max=60)
• retry_if_exception_type(HTTPError)

Pagination helpers

All return Iterator[list[dict]] with fetch_page and extract_records callbacks.

• _paginate_cursor — cursor-based (Zendesk, Finalsite)
• _paginate_offset — offset/limit (Overgrad, Grow, Coupa, DeansList, ADP WFN)
• _paginate_page — page number (KnowBe4, PowerSchool)

Migration tiers

Tier 1 — Direct drop-in (4 resources)

SmartRecruitersResource, PowerSchoolEnrollmentResource, OvergradResource, KnowBe4Resource

Override _setup_session() only, optionally _get_url().

Tier 2 — Additional hooks (4 resources)

ZendeskResource, FinalsiteResource, CoupaResource, GrowResource

Uses _get_retry_after override, OAuth2Session, or non-trivial _setup_session.

Tier 3 — Complex (2 resources)

AdpWorkforceNowResource, DeansListResource

OAuth2 + mTLS, _prepare_request override, custom Avro writing.

Testing

• Base class unit tests — mocked request pipeline, retry, error handling, pagination helpers
• Subclass mocked tests — _setup_session, _get_url, override behavior per resource
• Pagination contract tests — multi-page sequences with realistic response shapes, edge cases
• Integration validation — dagster definitions validate after each tier

Design spec

docs/superpowers/specs/2026-03-25-base-http-resource-design.md

[cbini]

Implementation Plan

Base Class Design

Create BaseHTTPResource(ConfigurableResource) in src/teamster/core/http.py providing the shared boilerplate all 14 resources duplicate:

Base class provides	Subclass implements
_session: Session (PrivateAttr)	Config fields (credentials, subdomain, etc.)
_base_url: str (PrivateAttr)	_setup_session() — auth logic
_log: DagsterLogManager (PrivateAttr)	_get_url() override (when URL structure differs)
setup_for_execution() — assigns _log, calls _setup_session()	_handle_error() override (429 retry, 401 re-auth)
_get_url(path, *args) — joins base + segments	list() / pagination methods
_request(method, url, **kwargs) — request + raise_for_status + _handle_error hook	Domain-specific methods
get(), post(), put(), delete() convenience methods

Key design decisions:

• _setup_session() is separated from setup_for_execution() so the base always handles _log assignment
• _handle_error(response, error) hook lets subclasses implement retry (Finalsite 429) or re-auth (AdpWfm 401) without overriding all of _request
• Pagination stays in subclasses — every list() has completely different logic (cursor, offset, page)
• OAuth2Session users (Coupa, AdpWfn) override _session in _setup_session() since OAuth2Session extends Session

Migration Tiers

Tier 1 — Direct drop-in (minimal override beyond _setup_session):

1. SmartRecruitersResource — simplest case, header auth
2. DibelsDataSystemResource — standard pattern, form login isolated in setup
3. PowerSchoolEnrollmentResource — basic auth, versioned URL
4. OvergradResource — header auth with timeout

Tier 2 — Minor _get_url or _request override:
5. KnowBe4Resource — api_version in URL
6. MClassResource — form login calls self.get() during setup
7. ZendeskResource — BasicAuth, rate limit helpers stay in subclass
8. CouchdropResource — changes _base_url after auth in _setup_session
9. FinalsiteResource — JWT auth, override _handle_error for 429 retry

Tier 3 — OAuth2 / custom session:
10. CoupaResource — OAuth2Session
11. GrowResource — OAuth2 token fetch, plain Session for requests, get() returns dict
12. AdpWorkforceNowResource — OAuth2Session + mTLS + rate limiting

Tier 4 — Significant behavioral override:
13. AdpWorkforceManagerResource — tenacity retry + 401 re-auth, overrides _request entirely
14. DeansListResource — non-standard _request signature (school_id, params injection); may only inherit boilerplate or be deferred

Edge Cases

Resource	Edge case	Mitigation
Couchdrop	Changes _base_url after auth POST	Works naturally — _setup_session reassigns _base_url after calling self.post()
AdpWfm	@retry decorator + 401 re-auth	Override _request entirely in subclass
MClass / Dibels	Call self.get() during _setup_session()	Works because _session is created via default_factory before setup runs
Coupa / AdpWfn	OAuth2Session instead of Session	Assign in _setup_session(); OAuth2Session extends Session
DeansListResource	_request(method, url, school_id, params, **kwargs)	Override _request; base get()/post() not used — subclass calls _request directly

Implementation Sequence

1. Create BaseHTTPResource in src/teamster/core/http.py
2. Migrate Tier 1 (4 resources) — validate definitions after each
3. Migrate Tier 2 (5 resources) — Finalsite introduces _handle_error pattern
4. Migrate Tier 3 (3 resources) — OAuth2 session handling
5. Migrate Tier 4 (2 resources) — evaluate DeansListResource cost/benefit; may defer
6. Run full test suite + trunk check

Risk Assessment

• Low risk: Tiers 1-2 — near-identical refactors
• Medium risk: Tier 3 — verify OAuth2Session assigned to Session-typed PrivateAttr works with Pydantic (runtime: yes; pyright may need a type annotation adjustment)
• Higher risk: Couchdrop (HTTP during setup) and AdpWfm (retry decorator) — need careful testing
• Lowest value: DeansListResource overrides nearly everything; may not be worth migrating