A discussion on DAP and central hosting vs self-hosting #152
Description
I reviewed the proposed decision for DAP to move to self-hosting. I recognize this appears to still be in draft, but given the magnitude of the change and the complexity of the topic, I want to create this issue for open discussion of the proposal.
For reader context, in a prior life, I'm one of the folks who made the original decision to promote central hosting, that this proposal would change. But my view has always been that straightforward just-centrally-host-it is a risky strategy over the long run and that it should not remain the status quo. I completely understand why DAP has continued to lean heavily on central hosting, as it has a strong updatability story, the risk generally doesn't feel urgent on any given day, and as it is a single static file in S3/Cloudfront, the architecture is simple to understand and (as architectures go) is at least conceptually straightforward in terms of access management.
But the risk is very real and does merit continued investment in mitigation, and this is a fine time to revisit the assumptions behind the current approach.
I'll also mention that at this point in my life, I consider myself a security professional who spends most of his time pushing to raise the bar on security. And that has almost always meant arguing against a central hosting strategy, or on embedding 3rd party assets. I've even blocked teams from launching a product until unnecessary 3rd party embeds have been removed. It is a real issue and I think the web developer community, despite occasional melodramas popping up in the news, remains insensitive to the risks the web bears.
However, the calculus for DAP is different. The proposal as written -- to lean more into self-hosting, and to encourage those who do so to update with some text and some technical changes -- is not the right answer. This approach would have its own real security and privacy risks for users of government websites, and is tantamount to giving up on DAP having any practical velocity for change.
In the past, I've proposed in-between solutions, such as enlisting a small known group of agencies who could commit to hosting copies of it, to split out the risk while having reasonable updatability. There are definitely other possible approaches (and I'll suggest some below), but all of them require some sort of human coordination and effort to mitigate the risk in both directions.
Agency ability to manage updates
The proposal lists two major things that have changed since DAP's decision to lean on central hosting. The first is:
New tools for semi-automatically managing dependency updates have been introduced and become popular since 2015 (exp., Dependabot, acquired by GitHub in 2019) and, for teams who are willing to integrate such tools, dependency rot is not inevitable.
This is true, but there is no evidence I am aware of to suggest that most .gov websites are integrating such tools or even using automatable dependency management for their web development; or for those that do, that they are continuing to actually pay attention and keep running these tools over time on a regular basis.
The .gov space is filled with websites that 1) are minimally/not maintained post-launch after the agency's attention span has moved on, 2) operated by contractors and sub-contractors that make changes or perform maintenance only when directed and at cost to the agency (thus incentivizing the agency away from asking for changes or maintenance), or 3) operated by federal employees that do not use anything close to contemporary strong web development practices.
The result is a collective molasses of rotten dependencies and ad hoc decisions that live on forever until something bad happens. This has been reflected every time I've looked at an analysis across .gov sites of embedded DAP versions, and of 3rd party dependencies writ large. It's not very inspiring. There are an increasing number of strong digital and web teams in government that are better at this, but not only do they still seem to be the exception, even they cannot always change the maintenance/attention-span factor in how agencies staff their projects post-launch.
As a result, self-hosting the DAP JS creates an environment over time that puts DAP not in control of its own destiny, much less its own security guarantees.
The GSA team that you should consult with about this, which is probably in the best position to be aware of the strengths and weaknesses in these areas spread across the .gov webspace, is the US Web Design System team. I recommend reaching out to them and getting the benefit of their experience (and the troubles that they have had trying to get people to update things over time that are as simple as the banner text that appears when you click the "here's how you know" on top of hundreds of .gov websites).
Ability to not release security flaws
The second thing described as having changed since 2015:
Of the 40 releases since 2015, only X% have included security updates. DAP is a small codebase with no external dependencies - it's not overly hubristic to think that we could mostly avoid releasing versions containing security flaws.
I would be curious to know what the X% is -- but even if it is 0%, this statement is extremely overly hubristic. The fact that DAP is a small codebase with no external dependencies is only lightly relevant. What DAP does, especially given its embedding on a huge variety of websites, is privacy- and security-critical, and even small coding errors -- or incorrect human assumptions about what should be considered sensitive -- could have significant ramifications.
For one, DAP is examining the page and calculating what information to report back into a central collection not owned by the agency who embedded it. That needs to be done with care. The Department of Education got into trouble in 2022 for enabling a checkbox that caused the JS on the page to automatically start grabbing form fields and reporting them back. While the Department has said that enabling this was an accident, in either case it looks like both Meta and the Department made the serious mistake of believing that running the fields through a SHA-2 hashing process afforded some protection to the data being sent. But since there was no salting (and in a front-end context, it's difficult to sync front- and back-end salts, and you obviously can't expect to have a private salt or pepper) reversing this data was trivial and the data may as well have been sent in cleartext. But the presence of SHA-2 hashing confused the issue substantially.
Similarly, DAP is currently built on top of Google Analytics. While most of the trade-offs involved in using GA are sort of "priced in" at this point and have seemed defensible to me, and DAP turns up the privacy knobs all the way, DAP does need to be ready if Google makes a mistake, or makes a service change that is fine for most users but not fine for the government, where DAP has a need to deploy a change that turns a GA feature off.
You also have to think about the contexts in which DAP is deployed. I know that on the program side, a huge amount of ongoing work has gone into data cleaning because of the amount of garbage data and not-real/production sites DAP ends up on. There is also the question of authenticated sessions - while DAP may guide agencies away from using it for authenticated sessions, agencies may do it anyway, by accident or on purpose, and DAP should be threat modeling its code under the assumption that it could be inside sensitive sessions.
It may be that DAP hasn't yet had a "bad day" where it's had to rush out an emergency privacy or security fix to its customers. But it's very easy to imagine reasons why that day may come. Handwaving this away and saying that it's a small codebase and we can probably avoid having to ever do that is a really dangerous sentiment, and not a view that the DAP should ever endorse.
Subresource Integrity
I also want to briefly discuss SRI, since the proposal touches on it. The proposal is right about SRI:
Since around 2018, browsers have universally supported a technology called subresource integrity, which can detect when a file downloaded from a CDN has been altered from its original published state and block the file from loading. Recently, the DAP team has received inquiries about SRI support from several of our customers. Unfortunately, given current technology, the only way to take advantage of SRI's guarantees is to pin your site to a specific version of the third-party library that you are including. This places the user back facing the same tradeoff that they face with self-hosting - if you pin your site to a specific version of DAP, then you can be assured of the code's integrity, but you no longer automatically receive security fixes.
However, the analysis here doesn't go far enough. Not only can Subresource Integrity solve no problems for DAP customers, it persistently misleads all but the most security-literate web developers about what it does. As the analysis notes, SRI does not change the security equation vs self-hosting, while self-hosting has strictly better privacy properties than an SRI-embedded use of the centrally hosted file.
The only use case for SRI that makes any real sense (and I'm fairly certain this was the motivating use case for the creation of the spec) is for sufficiently large applications to protect themselves from themselves. As an example, you can see GitHub's 2015 blog post where they talk about how they're using then-new SRI to mitigate the github.com application from a compromise of their own static asset CDN they were using at the time, which was served from a different origin with presumably different access control. (A glance at github.com's source code today suggests they now have an alternate strategy, as I don't see any integrity attributes on their JS files today, but I don't know for certain.) This narrow use of SRI makes complete sense, since privacy is less of an issue (the same company owns both origins) but security is improved by making it so that a compromise of one component doesn't compromise the entire application. But this is not the DAP use case, as the analysis above states.
I say the analysis doesn't go far enough because the proposal later says that DAP could support SRI if customers keep asking. This would dangerously pander to the widespread ignorance that exists about SRI. For all but the most sophisticated applications, SRI is essentially a security blanket that is worse than self-hosting, and the government's continued moth-like attraction to security blankets leads agencies to perform incorrect security and privacy analysis and to make the wrong decisions for their users.
Since the DAP team knows better, it should demonstrate leadership. If anything, it should take the analysis in the proposal, expand on it and put it in a central place to refer inquiring agencies to. And of course, once offered, DAP can never take this feature back. DAP should limit the invariant commitments it makes to agencies to only those which substantively offer benefits to agencies or to the DAP -- SRI does neither.
Recommendations for the DAP team
Here are a few recommendations on how DAP might move forward at finding a better balance of risks:
-
Dig into the data provided by the Site Scanning API, a fellow GSA program, that tracks what 3rd party JS files appear across the .gov webspace. See which websites appear to be doing a good and sophisticated job at managing dependencies. Note how many websites do not. The last time I took a good, hard look across this dataset, it was very dispiriting. That doesn't mean nothing can be done - it means that your work should start with those who the data shows do a good job.
-
As mentioned above, consult with the US Web Design System team on their experience in working to get agencies to stay up to date, and on deploying security and privacy fixes.
-
DAP should do some convening and develop partnerships to mitigate the risks of central hosting. Pull together the people in the government web community that DAP knows care about problems like this, can (as a group) fully understand the technical and non-technical issues raised in a post like this one, and start identifying what seems like the most practical (or the least impractical, at least!) solution.
-
Ideas for what that group could discuss could be identifying a manageable group of agencies to support semi-central hosting to amortize the risk out some and make each target less worth attacking; it could be establishing signed agreements with agencies that commit them to updates on a certain cadence and have defined break-glass communications channels for emergency updates; perhaps there are greater partnerships with the .gov program or with USWDS that could integrate DAP better into other workflows that agencies are more regularly confronted with.
I don't know, I'm sure there are other ideas. The point is to just start moving on figuring out which ones to pursue, so that the program isn't forced to choose between the risks of the status quo, or giving in to the rot that DAP has always been swimming upstream against since I first came across the program almost 10 years ago.