Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jan 5, 2026

The extraction loop makes 3 additional API calls (commits, reviewers, comments) per PR, creating an N+1 query pattern. For 250 PRs: 250 + (250×3) = 1000 API calls.

Changes

  • New environment variables to control data fetching (defaults maintain current behavior):

    • FETCH_COMMITS - fetch commit data per PR
    • FETCH_REVIEWERS - fetch reviewer data per PR
    • FETCH_COMMENTS - fetch comment data per PR
  • Conditional fetching in extract_pull_requests():

    • Skip API calls when disabled
    • Provide empty data structures ([] or {"users": [], "teams": []}) to maintain schema compatibility
  • Performance impact: Disabling all extra fetching reduces calls from 1000 to 250 (75% reduction) for 250 PRs

Example

# Fetch only PR metadata, skip extras
FETCH_COMMITS=false FETCH_REVIEWERS=false FETCH_COMMENTS=false python3 main.py

# Fetch PRs and reviewers only
FETCH_COMMITS=false FETCH_COMMENTS=false python3 main.py

Other fixes

  • Corrected extract_reviewers() return type annotation (dict not list[dict])
  • Fixed GITHUB_REPOS env var handling to check for None before calling .split()

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 3 commits January 5, 2026 21:59
Co-authored-by: dklawren <826315+dklawren@users.noreply.github.com>
Co-authored-by: dklawren <826315+dklawren@users.noreply.github.com>
Co-authored-by: dklawren <826315+dklawren@users.noreply.github.com>
Copilot AI changed the title [WIP] Update GitHub data export script for BigQuery Add optional data fetching to reduce API calls in N+1 query pattern Jan 5, 2026
Copilot AI requested a review from dklawren January 5, 2026 22:06
@dklawren
Copy link
Contributor

dklawren commented Jan 5, 2026

We need to get the data every time the script is ran so no need for parameters to turn on or off each API call.

@dklawren dklawren closed this Jan 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants