Gather API Gateway

Prerequisites

• Node.js 18+
• Access to a MongoDB deployment (Atlas or self-hosted)
• Firebase project with service account credentials that can access Cloud Storage and the Realtime Database

Environment Variables

Create a .env file (or export the variables in your shell) with the following keys:

Variable	Description
MONGODB_URI	MongoDB connection string
DATABASE_NAME	MongoDB database name
COLLECTION_NAME	Collection that stores survey documents
FIREBASE_STORAGE_BUCKET	Firebase Storage bucket used for export archives and status files (e.g. your-project.appspot.com)
GOOGLE_APPLICATION_CREDENTIALS (optional)	Absolute path to your Firebase service-account JSON file
API_KEYS (optional)	Comma-separated list of shared secrets required in the x-api-key header. When omitted, the API key guard is disabled.
API_KEY_HEADER (optional)	Override the header name checked for API keys. Defaults to x-api-key.
RATE_LIMIT_WINDOW_MS (optional)	Request throttling window in milliseconds. Defaults to 900000 (15 minutes).
RATE_LIMIT_MAX (optional)	Maximum number of requests per IP within the window. Defaults to 100.
TRUST_PROXY (optional)	Set to true when deploying behind a reverse proxy/load balancer so rate limiting uses the forwarded client IP.

Firebase Credentials

1. In the Firebase console, create a service account key with access to Cloud Storage (read/write) for the project bucket.
2. Download the JSON key file and place it in the project root as GOOGLE_APPLICATION_CREDENTIALS.json.
3. You can optionally set GOOGLE_APPLICATION_CREDENTIALS (or FIREBASE_CREDENTIALS_PATH) to the absolute path of the JSON file. If unset, the app automatically falls back to the root GOOGLE_APPLICATION_CREDENTIALS.json file when it exists.
4. When running in deployed environments, configure the same variables via your host's secret manager or environment configuration.

Export job status tracking

• Export runs are asynchronous: the API responds immediately with a jobId and a status endpoint.
• Job metadata is stored next to the archive as exports/{jobId}.json in the configured Storage bucket.
• Poll GET /api/export/status/{jobId}. The server reads the companion JSON file and, when the job is complete, includes a short-lived signed URL for exports/{jobId}.zip.
• Each archive contains metadata.json with survey summaries, counts, and timing details alongside the requested data file: data.json for the default export or data.csv when the CSV format is selected. The CSV export produces a flattened table and adds dynamic columns for new keys detected across observations.
• If you cancel or delete an export, remove both the .zip and .json files from Storage to clean up.

Security hardening

• Rate limiting: Every request passes through express-rate-limit with sane defaults (100 requests per 15 minutes). Tune via RATE_LIMIT_WINDOW_MS and RATE_LIMIT_MAX.
• API keys: Provide a comma-separated list in API_KEYS and supply one of the values in the x-api-key header on each request. Change the header name with API_KEY_HEADER. If API_KEYS is blank, the guard stays disabled—for staging environments, use separate keys or IP allow lists as needed.
• Reverse proxies: When deploying behind a load balancer (e.g., Cloud Run, App Engine, Nginx), set TRUST_PROXY=true so rate limiting measures the originating client IP instead of the proxy.
• Health checks: The /health endpoint remains unauthenticated for platform probes. Adjust the exempt route list in src/middleware/security.js if you need stricter controls.

API endpoints

GET /health

• Purpose: Lightweight readiness probe.
• Authentication: Not required.
• Response:

{
	"status": "ok"
}

POST /api/mongodb/:action

• Purpose: Read-only pass-through to MongoDB for UI search/browse flows.
• Authentication: API key header required when API_KEYS is configured.
• Allowed actions: find, findOne, aggregate.
• Request body (example: typeahead by survey name):

{
	"pipeline": [
		{ "match": { "surveyName": { "$regex": "^air", "$options": "i" } } },
		{ "group": { "_id": "$surveyName", "surveyIds": { "$addToSet": "$_id" }, "collections": { "$addToSet": "$collections" }, "startTime": { "$min": "$startTime" }, "stopTime": { "$max": "$stopTime" } } },
		{ "project": { "_id": 0, "surveyName": "$_id", "surveyIds": 1, "collections": 1, "startTime": 1, "stopTime": 1 } },
		{ "sort": { "surveyName": 1 } },
		{ "limit": 20 }
	]
}

• Response:

{
	"documents": [
		{
			"surveyName": "Air Quality",
			"surveyIds": ["66eab5a9f1..."],
			"collections": [["Field Notes", "Photos"]],
			"startTime": 1704067200000,
			"stopTime": 1704499200000
		}
	]
}

ℹ️ The proxy automatically converts _id strings into ObjectIds and blocks non-read actions.

POST /api/export/surveys

• Purpose: Kick off asynchronous export jobs for the selected surveys.
• Authentication: API key header required when enabled.
• Request body:

{
	"surveyIds": ["66eab5a9f17d4c31f5e7a4b2", "66eab5a9f17d4c31f5e7a4b3"],
	"includeMedia": true,
	"format": "csv"
}

• includeMedia (optional, default false): when true, Firebase media URLs referenced in the observations are downloaded into a media/ folder within the archive and sibling *_path fields point to those files.

• format (optional, default json): set to csv to receive a flattened data.csv instead of data.json. Leave as json to keep the JSON file.

• Response:

{
	"jobId": "1c2b3a4d-5678-90ef-1234-abcdef012345",
	"status": "processing",
	"statusCheckEndpoint": "/api/export/status/1c2b3a4d-5678-90ef-1234-abcdef012345"
}

GET /api/export/status/:jobId

• Purpose: Poll the job state and collect the download URL when complete.
• Authentication: API key header required when enabled.
• Response (complete job):

{
	"jobId": "1c2b3a4d-5678-90ef-1234-abcdef012345",
	"status": "complete",
	"progress": 100,
	"displayName": "Air-Quality-2024-09-29.zip",
	"fileSize": 1234567,
	"observationCount": 245,
	"aggregatedSurveyCount": 2,
	"downloadUrl": "https://storage.googleapis.com/..."
}

• Response (error):

{
	"jobId": "1c2b3a4d-5678-90ef-1234-abcdef012345",
	"status": "error",
	"error": "Export failed: <details>"
}

Setup

npm install

Local Development

npm run dev

Testing

npm test

Logging

• The app uses a lightweight logger that defaults to debug level in development and info in production.
• Override the level by setting LOG_LEVEL to one of debug, info, warn, or error.
• Example: LOG_LEVEL=debug npm run dev enables the verbose messages that help trace media exports locally while keeping production output tidy.