Risk: Service discovery between webapp/service in AWS #29
Description
Overview
The webapp needs to be able to talk to the service and currently this is done by using a private hosted zone within the VPC that both are deployed in.
We map the IP addresses of the webapp and API to DNS entries for each environment:
- Webapp:
webapp.${env}.beacons.local - service:
service.${env}.becons.local
With the current configuration, the webapp will be able to communication with the service using the DNS label above as route 53 will manage the resolving of the DNS to an IP address that resolves to an instance of the service within the VPC.
Issue
Whilst making an update to the name of the dns resource I encountered an issue with not being able to deploy the update as the steps that Terraform needed to do were:
- Delete the existing DNS hosted zone
- Create the new one DNS hosted zone
- Register the new services
Terraform failed on the first step, see the pipeline as it was unable to de-register the existing ECS tasks that were linked to the existing DNS hosted zone. There is an outstanding GitHub issue for this exact issue with Terraform but unfortunately no proper solution in place currently.
Workaround
The workaround to resolve this is to:
- Manually stop the ECS tasks that are running
- Re-deploy the Terraform configuration which will remove the DNS hosted zone (without the services registered)
- Create the new DNS hosted zone and register the new tasks that are re-created by Terraform
Potential Impact
The impact of this is high as it requires a manual intervention to stop the tasks and re-run the deployment (~ 1 minute to go through) and some down time in stopping the tasks.
The likelihood of it happening should be low as we do not expect to make changes to the dns zone resource often.
Alternatives
- Network traffic between the webapp and the service goes over the internet, i.e: a request is made outside of the AWS