🐛 [firestore-bigquery-export] Extension stopped streaming new data to BigQuery

[bgergen]

[READ] Step 1: Are you in the right place?

Sent here by GCP support. They said that since the issue is with a Firebase extension, we should file it here as a bug.

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.2.6
• Configuration values (redact info where appropriate):
  • cloud_functions_location: us-east4
  • bq_dataset_location: us
  • firestore_instance_location: nam5

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

What happened? How can we make the problem occur?
Unknown. The extension stopped streaming new data to BigQuery on 10/23. No changes were made to the extension during this time, and I did not see any error messages in the logs.

Expected result

The extension should stream new data to the designated BigQuery table as appropriate.

Actual result

New data in Firestore is not being streamed to BigQuery. We have been running the import script on a regular basis in order to continuously backfill our data.

[CorieW]

Hi @bgergen. Thanks for creating this issue! We'll review it as soon as possible.

[cabljac]

Hi @bgergen, were there any other configuration values you can provide to help us try and reproduce this?

Do you see any error logs in the deployed function?

[bgergen]

@cabljac What sorts of config values might be helpful? The only thing in our config that might be a bit out of the ordinary as far as I can tell is that we have an IAM Deny policy set up in our GCP project to restrict access to our BigQuery table, but the ext-firestore-bigquery-export service account is included in our list of exception principals. I wasn't able to find any error logs at the time it stopped streaming, or at any time since, which has made this particularly baffling.

[CorieW]

Hi @bgergen. Could you share all of the extension's config values (redact any private info), that way we can best try to reproduce this issue?

Also, roughly how long did you have the extension operating before you experienced this issue?