Skip to content

diamondback-data-desk/doge_scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

208 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Automatically Scraping DOGE Cuts to the University of Maryland

Reasoning

DOGE and the Trump administration have seen cuts to federal contracts, grants and leases as decreasing government waste. Institutions such as universities rely heavily on federal investment through these programs, the University of Maryland is one of these institutions. The Diamondback has relied on self reported grant terminations or this university self reporting data (more data) to report these cuts to the public. To ensure that the paper will have updated and reliable data the need to get the data from DOGE directly became paramount.

DOGE is not expressly open with their data. They have an API, though, they do not allow access to all data types, limiting information. Thus, scraping the contracts, grants and leases table directly off the DOGE website proved to be a viable solution.

Running

This github repository updates automatically with scraping beginning at 9:00 am, ending about an hour later, with cleaning starting at 10:30 am. There is no need to pull the repository to run the programs, though if you would like to receive updates to the data with your own local clone, you will have to pull from the repository.

Data

Raw DOGE cuts are provided in the data/doge_cuts.csv file. DOGE cuts specific to the University of Maryland or University of Maryland, College Park are located in data/umd_cuts.csv.

Update

--October 2, 20205-- Fixed cancelation of run due to job length issue. Increased the speed of the scraper, now it works faster than it did before with a run time of under two hours.

Last Update: --October 1, 2025-- The DOGE scraper is now both fixed and interates through every row. All descriptions of grants, leases and contracts within the savings page are now entirly collected - though, some of them are useless. It takes seven hours though. That's, a bit long. But I'll change the timing of collection times to not be as frequent.

Theodore Rose | Data Reporter

About

a scraper to collect all DOGE reported cuts to UMD

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors