Federator next steps

[Stephen-ONeil]

Creating this issue for discussion/to get the ball rolling on defining the next steps for the PT-to-PHAC federation process. Unlike the PT-to-PT side of things, we do anticipate continued technical work on the PT-to-PHAC side. Specifically, the federator service should be expected to move towards production, and the requirements should be expected to change/be clarified as we work with PTs to figure out what their aggregation (or equivalent) APIs can/will look like.

A lot of these can be broken out in to their own issues/tasks after discussion/initial investigation. Feel free to check off discussed items/add links to related issues if they've been converted in to specific tasks. Maybe cross out discarded thoughts.

Things we'll need to make assumptions about for now and seek clarity on as we can

• data classification
  • what classification will each province place on their aggregated values and what's the federal equivalent? We can probably assume based on the current reports IDVP receives, though it's possible timeliness of the proposed direct API access could change the classification?
  • what impacts on architectural decisions would classifications above "unclassified" apply? We'll probably aim for a higher bar by default, so maybe it won't be impactful
• authentication
  • what PT facing (external, system-to-system) authentication mechanism(s) will be used?
    • we could make recommendations, but ultimately it will be up to their security requirements and tech choices
    • might need to be able to handle multiple mechanisms to be applied on a PT-by-PT basis
    • current federator and mock-aggregators have the federator signing JWTs with its private key and the aggregators verifying the federator's identity by decrypting them with the public key. Reasonable choice, but not guaranteed
  • what PHAC facing (internal, system-to-system but also possibly system-to-user) authentication mechanism(s) will be used?
    • will IDVP data scientists/analysts require user level credentials for direct API data access? Would they use that if we provided it or would they have an intermediary data-extraction system (STARVAC?)
      • if user level access is in the cards, assume we work it out via PHAC's Entra ID
      • PHAC system to federator authentication is a guaranteed requirement. Per consumer API keys should be sufficient
• rate limiting
  • the federator should enforce it for consumers (PHAC systems/users)
  • maybe assume it from providers (PT aggregators), consider per-PT caching inside the federator
• data validation
  • currently applies strict validation to the data received from the mock-aggregators. Ideally all the PTs send consistent data to us but we could also consider having the ability to define expected formats per-PT in case we need to accept various formats and take responsibility for standardizing them within the federator
• PHAC facing endpoints
  • what does IDVP expect/want?
  • one endpoint for all aggregated records across time, from all provinces, might be acceptable as the data will never get too large. Easiest to start with and for them to immediately consume
  • time bounded queries?
  • pagination?
  • GraphQL?
• our source code
  • if PT to PT is parked and PT to PHAC moves forward on its own, may be worth considering moving it to its own GitHub repo
  • PTs may be sensitive about our code staying open source once it starts interacting with their actual APIs (our code and test cases may reveal specifics of their APIs/the data shared that they should at least have a say in us releasing, even indirectly). If we move to a new repo at some point, consider keeping it private by default until this has been discussed with PTs (... and keep in mind all the features we'll lose if we have to go private at some point)
• failure handling
  • what will we do if one PT doesn't respond but the rest do? IDVP may have an opinion

Unambiguous items

• review of express JS best practices for performance and security
• structured logging, consistent logging practices