major: update to modern package ecosystem, migrate to Justfile, add automated field detection PR work and other goodies.

.github/workflows/pr-for-new-fields.yml

@@ -0,0 +1,57 @@
+name: New fpds search fields PR
+
+on:
+  schedule:
+     - cron: "0 5 * * *"
+  workflow_dispatch:
+
+jobs:
+  scrape:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: nanasess/setup-chromedriver@v2
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - uses: extractions/setup-just@v3
+        with:
+          just-version: 1.46.0
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.13
+
+      - name: Scrape FPDS Site
+        id: scrape
+        run: just scrape
+
+      - name: Create Pull Request
+        uses: peter-evans/create-pull-request@v6
+        with:
+          commit-message: "minor: updating fields.json with newly identified ezSearch fields"
+          title: "minor: 🤖 [AUTOMATED] new ezSearch fields"
+          body: |
+            Automated updates from FPDS ezSearch scraper.
+
+            ✅  ATOM Feed Version: ${{ steps.scrape.outputs.feed_version }}
+            📃  Data Dictionary [here](${{ steps.scrape.outputs.data_dict_url }})
+
+            Before merging, please review the data dictionary file above and generate the appropriate regex pattern
+            for all new fields.
+
+            🔴 **Failures** 🔴
+            -----------------
+            The following fields were found in the ezSearch page, but their format could not be determined. If the
+            field is marked as ❌, you will need to find relevant metadata in the data dictionary file linked above.
+            If the field is marked as ✅, you can ignore this warning.
+
+            ${{ steps.scrape.outputs.grid }}
+
+          branch: minor/new-ezsearch-fields
+          base: master
+          labels: automated, feature
+          draft: false

.github/workflows/publish.yml

@@ -59,16 +59,20 @@ jobs:
       - name: Install uv
         uses: astral-sh/setup-uv@v5
 
+      - uses: extractions/setup-just@v3
+        with:
+          just-version: 1.46.0
+
       - name: Setup Python
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
       - name: Install Project
-        run: make install
+        run: just install
 
       - name: Run Tests
-        run: make test
+        run: just test
 
   publish:
     needs: [cut-release, build-and-test]
@@ -85,4 +89,4 @@ jobs:
       - name: Publish Package
         env:
           UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
-        run: make publish
+        run: just publish

.github/workflows/test-package.yml

@@ -22,20 +22,30 @@ jobs:
       - name: Install uv
         uses: astral-sh/setup-uv@v5
 
+      - uses: extractions/setup-just@v3
+        with:
+          just-version: 1.46.0
+
       - name: Setup Python
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
       - name: Install Project
-        run: make install
+        run: just install
 
-      - name: Run Tests
-        run: make test
+      - name: Run linters
+        run: just formatters
 
-      - uses: actions/checkout@v4
+      - uses: stefanzweifel/git-auto-commit-action@v5
+        if: ${{ matrix.python-version == '3.13' }}
         with:
-          fetch-depth: 0
+          commit_message: "patch: [skip ci] Run formatters"
+          file_pattern: "*.py"
+          disable_globbing: true
+
+      - name: Run Tests
+        run: just test
 
   test-package-version:
     runs-on: ubuntu-latest

Makefile → Justfile

@@ -1,8 +1,5 @@
-.PHONY: help venv install clean formatters mypy test local-test package publish
-.DEFAULT_GOAL := help
-
-help:
-	@python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
+default:
+  just --list
 
 venv: ## defaults to creating virtual environment in current directory under .venv
 	@if [ -d .venv ]; then \
@@ -11,8 +8,8 @@ venv: ## defaults to creating virtual environment in current directory under .ve
 		uv venv; \
 	fi
 
-install: venv ## checks if uv.lock is up-to-date and manually syncs all deps + extras
-	uv lock --check
+install: venv ## updates uv.lock if needed and manually syncs all deps + extras
+	uv lock
 	uv sync --extra all
 
 clean: ## Remove test and coverage artifacts
@@ -27,7 +24,7 @@ formatters: venv ## https://docs.astral.sh/ruff/formatter/#line-breaks
 	uv tool run ruff format
 
 mypy: ## Typechecking with mypy
-	uv tool run mypy src/
+	uv run mypy src/
 
 test: venv install ## Run unit tests with coverage
 	uv run -m pytest
@@ -40,3 +37,6 @@ package: ## builds project + artifacts in dist/ directory
 
 publish: package ## publishes package to pypi
 	uv publish
+
+scrape: venv install
+	uv run python src/fpds/scripts/scraper.py

README.md

@@ -1,90 +1,127 @@
-# fpds
+<div align="center">
+<pre>
+    __________  ____  _____
+   / ____/ __ \/ __ \/ ___/
+  / /_  / /_/ / / / /\__ \
+ / __/ / ____/ /_/ /___/ /
+/_/   /_/   /_____//____/
+Welcome to a more user-friendly FPDS 🚀
+</pre>
+</div>
 A light-weight, pythonic parser for the Federal Procurement Data System (FPDS) ATOM Feed.
-Reference [here](https://www.fpds.gov/fpdsng_cms/index.php/en/).
 
 
 ## Motivation
-The FPDS ATOM feed limits each request to 10 records, which forces users to deal with pagination. Additonally, data is exported as XML, which proves annoying. `fpds` will handle all pagination and data
-transformation to provide users with a nice JSON representation of the
-equivalent XML data and attributes.
+To make FPDS data more accesible to developers.
 
+This library helps users by doing the following:
+- Automatically handling pagination
+- Converting XML into a flat JSON structure
 
-## Setup
-As of version 1.5.0, this library manages dependencies using `uv`. It is
-_highly_ recommended since this library is tested with it.
-
+This library is based on the FPDS ezSearch interface that can be found
+[here](https://www.fpds.gov/ezsearch/search.do?indexName=awardfull&templateName=1.5.3&s=FPDS.GOV&q=).
 
-### Installing `uv`
 
-You can follow any of the methods found [here](https://docs.astral.sh/uv/getting-started/installation/). If on Linux or MacOS, we recommend using Homebrew:
+## Prerequisites
+As of version 1.5.0, this library manages dependencies using `uv`. It is
+_highly_ recommended since this library is tested with it. Note that this
+README assumes you will install `uv` and therefore runs all commands within
+its context.
 
+### `uv`
+You can follow any of the methods found [here](https://docs.astral.sh/uv/getting-started/installation/).
+If on MacOS, we recommend using Homebrew:
 ```
 $ brew install uv
 ```
 
-Once `uv` is installed, you can use the project Makefile to ensure your local environment is synced with the latest library installation. Start by running `make install` — this will check the status of the `uv.lock` file, and install all project dependencies + extras
+### `just`
+A command runner inspired by `Makefile`, written in Rust.
+```
+$ brew install just
+```
 
-### Local Development
+Once `uv` is installed, you can use the project Justfile to ensure your local environment
+is synced with the latest library installation. Start by running `just install` — this
+will check the status of the `uv.lock` file, and install all project dependencies +
+package extras.
 
-For linting and formatting, we use `ruff`. See `pyproject.toml`
-for specific configuration.
+## Usage
+For a list of valid search criteria parameters, consult FPDS documentation
+found [here](https://www.fpds.gov/wiki/index.php/Atom_Feed_Usage).
 
-```
-$ make formatters
-```
+### CLI
+### `fields`
+Returns fields available for API requests.
 
-You can clean the clutter and unwanted noise from tools using:
+To display all available fields:
 
 ```
-$ make clean
+$ uv run fpds fields
 ```
 
-### Testing
+If successful, you should see a nice, tabulated table in your terminal
+
+![fields-cli-output](img/fpds-fields-cli-output.png)
+<div align="center"><b>Figure 1 - Tabulated fields CLI output</b></div>
+<br />
+
+
+If you wanted to perform a more targeted search, add `-p`. For example, to
+get all fields containing the text "vendor" anywhere in the name, you could
+run the following:
 ```
-$ make local-test
+$ uv run fpds fields -p vendor
 ```
 
-## Usage
-For a list of valid search criteria parameters, consult FPDS documentation
-found [here](https://www.fpds.gov/wiki/index.php/Atom_Feed_Usage). Parameters
-will follow the `URL String` format shown in the link above, with the
-following exceptions:
+![fields-cli-vendor](img/fpds-fields-vendor.png)
+<div align="center"><b>Figure 2 - Matching "vendor" fields</b></div>
+<br />
 
- + Colons (:) will be replaced by equal signs (=)
- + Certain parameters enclose their value in quotations. `fpds` will
-automatically determine if quotes are needed, so simply enclose your
-entire criteria string in quotes.
+### `parse`
+Sends and parses records from an FPDS ATOM feed request
 
- For example, `AGENCY_CODE:"3600"` should be used as `"AGENCY_CODE=3600"`.
+Lets say you wanted records from the OFFICE OF THE INSPECTOR GENERAL.
+Through your research, you identified that agency's code as 7504. For your
+particular project, you are only interested in awards modified within the first
+quarter of 2025. Using the `parse` command, you can easily retrieve records with
+the following command:
 
-Via CLI:
 ```
-$  fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/05/01]" "AGENCY_CODE=7504"
+$  uv run fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/03/31]" "AGENCY_CODE=7504"
 ```
 
-By default, data will be dumped into an `.fpds` folder at the user's
-`$HOME` directory. If you wish to override this behavior, provide the `-o`
-option. The directory will be created if it doesn't exist.
+With this command, you can specify as many filters as you want. Unfortunately due
+to rate limitations with the ATOM feed, you _must_ have at least 1 filter.
 
-As of v1.5.0, you can opt out of regex validation by setting the `-k` flag
-to `False` — this is helpful in scenarios when either the regex pattern has
-been altered by the ATOM feed or a new parameter name is supported, but not
-yet added to the configuration in this library.
+By default, this command will output records to a directory named `.fpds` in
+your home directory. If you wish to output to a different location, specify your
+location with `-o`:
 
 ```
-$  fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/05/01]" "AGENCY_CODE=7504" -o ~/.my-preferred-dir
+$ uv run fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/03/31]" "AGENCY_CODE=7504" -o /Users/Desktop/data
 ```
 
-Same request via python interpreter:
-```
+## Python
+Core parsing and transformation classes are exposed as first-class citizens.
+
+```{python}
+
 import asyncio
-from fpds import fpdsRequest
+from fpds import FPDSRequest
 
-request = fpdsRequest(
-    LAST_MOD_DATE="[2022/01/01, 2022/05/01]",
+request = FPDSRequest(
+    LAST_MOD_DATE="[2022/01/01, 2022/03/31]",
     AGENCY_CODE="7504"
 )
 
+# will return the initial HTTP request a user would make if using Postman
+request_url = request.__url__()
+
+# total number of pages in request
+page_count = request.page_count
+
+
 # returns records as an async generator
 gen = request.iter_data()
 
@@ -97,11 +134,10 @@ async for entry in gen:
 records = asyncio.run(request.data())
 ```
 
-
 # Highlights
 
-Between v1.2.1 and v1.3.0, significant improvements were made with `asyncio`. Here are some rough benchmarks in estimated data extraction + post-processing
-times:
+Between v1.2.1 and v1.3.0, significant improvements were made with `asyncio`. Here are
+some rough benchmarks in estimated data extraction + post-processing times:
 
 | v1.2.1 | v.1.3.0 |
 -------- | --------
@@ -116,4 +152,5 @@ This equates to a <u>**84.89%**</u> decrease in completion time!
 
 # Notes
 
-Please be aware that this project is an after-hours passion of mine. I do my best to accomodate requests the best I can, but I receive no $$$ for any of the work I do here.
+Please be aware that this project is an after-hours passion of mine. I do my best
+to accomodate requests, but I receive no $$$ for any of the work I do here.