[py] Add APIRequestContext for HTTP requests with browser cookie sync#17226
Closed
mayank-at-sauce wants to merge 1 commit intoSeleniumHQ:trunkfrom
Closed
[py] Add APIRequestContext for HTTP requests with browser cookie sync#17226mayank-at-sauce wants to merge 1 commit intoSeleniumHQ:trunkfrom
mayank-at-sauce wants to merge 1 commit intoSeleniumHQ:trunkfrom
Conversation
Add a `driver.request` API that lets Selenium users make HTTP requests with automatic browser cookie synchronization — bridging the biggest feature gap with Playwright's APIRequestContext. Key features: - Bidirectional cookie sync: browser cookies sent with API requests, API response cookies synced back to the browser - Isolated contexts via `new_context()` with separate cookie jars - Auth state persistence via `storage_state()` save/load to JSON - All HTTP methods: GET, POST, PUT, PATCH, DELETE, HEAD, plus `fetch()` - Common kwargs: data, form, json_data, headers, params, timeout, max_redirects, fail_on_status_code - Pure Python HTTP client using urllib3 (already a Selenium dep), no BiDi/WebSocket dependency - Resource cleanup on `driver.quit()` Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Contributor
Review Summary by QodoAdd APIRequestContext for HTTP requests with browser cookie sync
WalkthroughsDescription• Adds driver.request API for HTTP requests with automatic browser cookie synchronization • Implements bidirectional cookie sync between browser and API requests • Supports isolated contexts with separate cookie jars via new_context() • Includes auth state persistence via storage_state() save/load to JSON • Provides all HTTP methods (GET, POST, PUT, PATCH, DELETE, HEAD, fetch) with common kwargs File Changes1. py/selenium/webdriver/common/api_request_context.py
|
Contributor
Code Review by Qodo
1. driver.request lacks parity note
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+1323
to
+1339
| @property | ||
| def request(self) -> APIRequestContext: | ||
| """Returns an APIRequestContext for making HTTP requests with browser cookie sync. | ||
| Returns: | ||
| An APIRequestContext instance bound to this driver. | ||
| Examples: | ||
| ``` | ||
| response = driver.request.get("https://api.example.com/data") | ||
| assert response.ok | ||
| data = response.json() | ||
| ``` | ||
| """ | ||
| if self._request is None: | ||
| self._request = APIRequestContext(self) | ||
| return self._request |
Contributor
There was a problem hiding this comment.
1. driver.request lacks parity note 📘 Rule violation ⛯ Reliability
A new user-visible WebDriver.request API is introduced without any in-code reference to cross-binding parity verification or a tracked follow-up for other Selenium bindings. This risks inconsistent behavior across language bindings and makes parity work easy to miss.
Agent Prompt
## Issue description
A new user-visible API (`WebDriver.request`) is added, but there is no evidence in the change itself that cross-binding parity was assessed or that parity follow-up work is tracked.
## Issue Context
Compliance requires user-visible behavior changes to be compared against at least one other Selenium binding and/or have a parity follow-up tracked.
## Fix Focus Areas
- py/selenium/webdriver/remote/webdriver.py[1323-1339]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+504
to
+535
| if storage_state is not None: | ||
| if isinstance(storage_state, (str, pathlib.Path)): | ||
| with open(storage_state) as f: | ||
| state = json.load(f) | ||
| else: | ||
| state = storage_state | ||
| cookies = list(state.get("cookies", [])) | ||
|
|
||
| return _IsolatedAPIRequestContext( | ||
| base_url=base_url, | ||
| extra_headers=extra_headers, | ||
| cookies=cookies, | ||
| timeout=self._timeout, | ||
| max_redirects=self._max_redirects, | ||
| fail_on_status_code=fail_on_status_code, | ||
| ) | ||
|
|
||
| def storage_state(self, path: str | pathlib.Path | None = None) -> dict[str, Any]: | ||
| """Export the current browser cookies as a storage state dict. | ||
|
|
||
| Args: | ||
| path: Optional file path to save the storage state as JSON. | ||
|
|
||
| Returns: | ||
| A dict with a "cookies" key containing the browser cookies. | ||
| """ | ||
| cookies = self._driver.get_cookies() | ||
| state: dict[str, Any] = {"cookies": cookies} | ||
| if path is not None: | ||
| with open(path, "w") as f: | ||
| json.dump(state, f, indent=2) | ||
| return state |
Contributor
There was a problem hiding this comment.
2. storage_state path unvalidated 📘 Rule violation ⛯ Reliability
User-provided file paths for storage_state import/export are used directly without fail-fast validation or actionable errors (e.g., missing file, invalid JSON, unwritable path). This can produce unclear downstream exceptions and makes troubleshooting harder.
Agent Prompt
## Issue description
`storage_state` load/save uses user-provided paths without validation and can raise low-level exceptions (e.g., `FileNotFoundError`, `PermissionError`, `JSONDecodeError`) that are not actionable.
## Issue Context
Compliance requires fail-fast validation with clear error messages for required/user-provided inputs.
## Fix Focus Areas
- py/selenium/webdriver/common/api_request_context.py[504-511]
- py/selenium/webdriver/common/api_request_context.py[530-535]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+117
to
+125
| # Domain matching | ||
| cookie_domain = cookie.get("domain", "") | ||
| if not cookie_domain: | ||
| # No domain set — treat as host-only, match any host | ||
| pass | ||
| elif cookie_domain.startswith("."): | ||
| # .example.com matches example.com and sub.example.com | ||
| if not (hostname == cookie_domain[1:] or hostname.endswith(cookie_domain)): | ||
| return False |
Contributor
There was a problem hiding this comment.
3. Host-only cookie overmatch 🐞 Bug ⛨ Security
_cookie_matches treats cookies with missing/empty 'domain' as matching any hostname, so such cookies can be attached to requests for unrelated domains. This violates the intended “host-only cookie” behavior and can leak session cookies outside their origin host via driver.request.
Agent Prompt
### Issue description
`_cookie_matches()` currently treats cookies with missing/empty `domain` as matching **any** request hostname, which can cause host-only cookies to be sent to unrelated domains via `driver.request`.
### Issue Context
Host-only cookies (no Domain attribute) must only match the origin host, not arbitrary hosts. In the WebDriver cookie dict, `domain` is optional, so this case must be handled safely.
### Fix Focus Areas
- py/selenium/webdriver/common/api_request_context.py[100-141]
- py/selenium/webdriver/common/api_request_context.py[537-545]
### Implementation notes
- Prefer changing `_cookie_matches(cookie, url, default_host=None)` so that if `cookie` has no domain, it only matches when `hostname == default_host`.
- In `APIRequestContext._get_cookies_for_request`, compute `default_host` from `urlparse(self._driver.current_url).hostname` (guard exceptions/empty current_url). If unavailable, skip cookies missing `domain`.
- Consider normalizing any cookie dicts returned by the driver by filling in missing `domain` with the current host before matching.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+593
to
+612
| def _get_cookies_for_request(self, url: str) -> list[dict]: | ||
| """Get matching cookies from the internal jar.""" | ||
| return [c for c in self._cookies if _cookie_matches(c, url)] | ||
|
|
||
| def _handle_response_cookies(self, set_cookie_headers: list[str], url: str) -> None: | ||
| """Store Set-Cookie headers in the internal jar.""" | ||
| parsed_url = urllib.parse.urlparse(url) | ||
| for sc_header in set_cookie_headers: | ||
| cookie = _parse_set_cookie(sc_header) | ||
| if not cookie.get("name"): | ||
| continue | ||
| cookie.setdefault("domain", parsed_url.hostname or "") | ||
| cookie.setdefault("path", "/") | ||
| # Cookies are unique by (name, domain, path) | ||
| key = (cookie["name"], cookie.get("domain", ""), cookie.get("path", "/")) | ||
| self._cookies = [ | ||
| c for c in self._cookies | ||
| if (c.get("name"), c.get("domain", ""), c.get("path", "/")) != key | ||
| ] | ||
| self._cookies.append(cookie) |
Contributor
There was a problem hiding this comment.
4. Expired cookies still sent 🐞 Bug ✓ Correctness
The isolated cookie jar never removes or filters cookies by 'expiry', so cookies set with Max-Age=0/negative (deletion) or past Expires remain stored and will still be sent on subsequent requests. This breaks cookie lifecycle semantics and can keep stale authentication state alive in isolated contexts.
Agent Prompt
### Issue description
Expired/deleted cookies are never filtered or removed, so isolated contexts can keep sending cookies whose `expiry` is in the past (e.g., `Max-Age=0`).
### Issue Context
`_parse_set_cookie` already computes `expiry`, but `_cookie_matches` ignores it and `_IsolatedAPIRequestContext` always stores the cookie.
### Fix Focus Areas
- py/selenium/webdriver/common/api_request_context.py[100-141]
- py/selenium/webdriver/common/api_request_context.py[593-612]
- py/selenium/webdriver/common/api_request_context.py[144-201]
### Implementation notes
- Add an expiry check to `_cookie_matches`: if `expiry` exists and `expiry <= int(time.time())`, return `False`.
- In `_IsolatedAPIRequestContext._handle_response_cookies`, if parsed cookie has `expiry <= now`, remove any existing cookie with the same (name, domain, path) key and **do not append**.
- Consider doing the same for `APIRequestContext` browser-sync path where feasible (e.g., attempt `driver.delete_cookie(cookie['name'])` when `expiry` is in the past), but at minimum ensure isolated contexts behave correctly.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
mayank-at-sauce
deleted the
mayankbhandari/feature/add-apirequestcontext
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
driver.requestAPI that lets Selenium users make HTTP requests with automatic browser cookie synchronization — bridging the biggest feature gap with Playwright'sAPIRequestContextMotivation
Selenium currently has no way to make HTTP API calls that share authentication/cookie state with the browser session. Common real-world scenarios like API-based login, test data seeding, and hybrid API+UI testing require manual cookie extraction and format conversion. This PR provides a clean, built-in, first-class API.
Key features
driver.get_cookies()), API responseSet-Cookieheaders synced back to browser (driver.add_cookie())get(),post(),put(),patch(),delete(),head(),fetch()data,form,json_data,headers,params,timeout,max_redirects,fail_on_status_codedriver.request.new_context()creates a separate cookie jar that doesn't affect the browserstorage_state(path="auth.json")saves cookies to disk,new_context(storage_state="auth.json")restores themdriver.quit()Usage examples
Files changed (5)
py/selenium/webdriver/common/api_request_context.pyAPIResponse,APIRequestContext,_IsolatedAPIRequestContext, cookie matching, Set-Cookie parsingpy/selenium/webdriver/remote/webdriver.py_requestinstance var, lazyrequestproperty, cleanup inquit()py/test/selenium/webdriver/common/webserver.pyecho_headers,echo_json,set_cookie,echo_body,do_HEADpy/test/selenium/webdriver/common/api_request_context_tests.pypy/test/unit/selenium/webdriver/common/api_request_context_tests.pyTest plan
bazel test //py:unitandbazel test //py:test-chrome