diff --git a/DESCRIPTION b/DESCRIPTION index d1ed74d..fa9eb93 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: fcall Title: Parse Farm Credit Administration Call Report Data into Tidy Data Frames -Version: 0.1.5 +Version: 0.1.6 Authors@R: c( person("Michael", "Thomas", email = "mthomas@ketchbrookanalytics.com", role = c("aut", "cre")), diff --git a/NEWS.md b/NEWS.md index 9c49f85..f6a6510 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,11 @@ +# fcall 0.1.6 + +## Bug Fix + +Sometime around 2025-12-08, the Farm Credit Administration (FCA) added a layer to it's website to "Verify you are a human" before hitting the page where the data can be downloaded via hyperlinks. This caused the `download_data()` function (which relies on `utils::download.file()`) to fail, including in CRAN checks/builds. + +To remediate this issue, Ketchbrook Analytics has replicated the files on the FCA website (from 2000-present) in a public AWS S3 bucket. `download_data()` now retrieves files from the public AWS S3 bucket instead of the FCA website directly. + # fcall 0.1.5 Addresses CRAN comments regarding: diff --git a/R/download_data.R b/R/download_data.R index 0ee20bb..c1a2e09 100644 --- a/R/download_data.R +++ b/R/download_data.R @@ -1,4 +1,4 @@ -#' Download data from FCA website +#' Download FCA Call Report Data and Unzip #' #' @param year (Integer) The year of the Call Report (e.g., `2022`) #' @param month (String) The month of the Call Report (e.g., `"March"`); you may @@ -18,6 +18,9 @@ #' `c(3, 6, 9, 12)`, unless there is an anomaly in FCA's reporting/publishing. #' Check to #' ensure the data is available for the quarter you are interested in. +#' Ketchbrook Analytics downloads these files and stores them in a public AWS +#' S3 bucket, which is the location that `download_data()` retrieves them +#' from. #' #' @return Console message informing the user where the data was successfully #' downloaded (and unzipped) into @@ -57,7 +60,8 @@ #' } download_data <- function(year, month, dest, files = NULL, quiet = FALSE) { - # Example valid URL: "https://www.fca.gov/template-fca/bank/2020March.zip" + # Example valid URL: + # "https://fca-call-report-data.s3.us-east-1.amazonaws.com/raw/2020March.zip" # Ensure only one month & one year are specified if (length(year) > 1L) { @@ -89,12 +93,15 @@ download_data <- function(year, month, dest, files = NULL, quiet = FALSE) { } + # Define base URL for AWS S3 bucket + base_url <- "https://fca-call-report-data.s3.us-east-1.amazonaws.com/raw/" + # The download URL convention changed in March 2015 if (year >= 2015L) { # Build the URL to the .zip file url <- paste0( - "https://www.fca.gov/template-fca/bank/", + base_url, year, month, ".zip" @@ -113,7 +120,7 @@ download_data <- function(year, month, dest, files = NULL, quiet = FALSE) { # Build the URL to the .zip file url <- paste0( - "https://www.fca.gov/template-fca/download/CallReportData/", + base_url, month, year, ".zip" diff --git a/README.Rmd b/README.Rmd index 26f011b..8c1c729 100644 --- a/README.Rmd +++ b/README.Rmd @@ -38,9 +38,9 @@ pak::pak("ketchbrookanalytics/fcall") ## Background -FCA publishes Call Report data on a quarterly basis at [https://www.fca.gov/bank-oversight/call-report-data-for-download](https://www.fca.gov/bank-oversight/call-report-data-for-download). +FCA publishes Call Report data on a quarterly basis at [https://www.fca.gov/bank-oversight/call-report-data-for-download](https://www.fca.gov/bank-oversight/call-report-data-for-download). Ketchbrook Analytics replicates these files in a public AWS S3 bucket, which {fcall} interacts with via its `download_data()` function. -As of June 2025, this data represents a set of 72 *.TXT* files. These files represent 36 datasets. The files prefixed with "D_" contain *metadata* (the column names, data types, etc.) of the associated file containing the raw, header-less comma-separated data. For example, the file that starts with *"D_INST"* contains the metadata for the file that starts with *"INST_"*. +As of September 2025, this data represents a set of 72 *.TXT* files. These files represent 36 datasets. The files prefixed with "D_" contain *metadata* (the column names, data types, etc.) of the associated file containing the raw, header-less comma-separated data. For example, the file that starts with *"D_INST"* contains the metadata for the file that starts with *"INST_"*. Further, some of these datasets are structured in a way that makes data analysis difficult. In these cases, we chose to pivot the data to make it more analysis-friendly. See [Metadata File Scenarios](#metadata-file-scenarios) for a more in-depth discussion of when and how this pivoting took place. diff --git a/README.md b/README.md index f1ae976..a87b864 100644 --- a/README.md +++ b/README.md @@ -35,9 +35,11 @@ pak::pak("ketchbrookanalytics/fcall") FCA publishes Call Report data on a quarterly basis at . +Ketchbrook Analytics replicates these files in a public AWS S3 bucket, +which {fcall} interacts with via its `download_data()` function. -As of June 2025, this data represents a set of 72 *.TXT* files. These -files represent 36 datasets. The files prefixed with “D\_” contain +As of September 2025, this data represents a set of 72 *.TXT* files. +These files represent 36 datasets. The files prefixed with “D\_” contain *metadata* (the column names, data types, etc.) of the associated file containing the raw, header-less comma-separated data. For example, the file that starts with *“D_INST”* contains the metadata for the file that diff --git a/cran-comments.md b/cran-comments.md index 33ed917..cabdc12 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -1,7 +1,7 @@ -## Resubmission +## Fix -This is a resubmission of a package that has not yet been on CRAN.; the initial two attempts were rejected. In this version I have made the following changes: +This is a resubmission of a package that is on CRAN but checks are failing due to a "Verify you are a human" page blocking the URL where `utils::download.file()` (within the package's `download_data()` function) attempts to download a file from. The following changes have been made to remediate this issue: -* Removed second "Description:" from the Description field in `DESCRIPTION` -* Removed single quotes around 'FCA' acronym in `DESCRIPTION` -* Replaced `\dontrun{}` with `\donttest{}` in examples that download data (Note: all examples involve downloading data) +* Files to be downloaded via `download_data()` are now hosted in a public AWS S3 bucket +* `download_data()` now points to the public AWS S3 bucket +* Documentation has been updated to let users know that the data is now being downloaded from AWS S3 instead of the FCA website directly diff --git a/man/download_data.Rd b/man/download_data.Rd index 90a7bf3..934b41f 100644 --- a/man/download_data.Rd +++ b/man/download_data.Rd @@ -2,7 +2,7 @@ % Please edit documentation in R/download_data.R \name{download_data} \alias{download_data} -\title{Download data from FCA website} +\title{Download FCA Call Report Data and Unzip} \usage{ download_data(year, month, dest, files = NULL, quiet = FALSE) } @@ -27,7 +27,7 @@ Console message informing the user where the data was successfully downloaded (and unzipped) into } \description{ -Download data from FCA website +Download FCA Call Report Data and Unzip } \details{ FCA publishes Call Report data quarterly. These .zip files are @@ -37,6 +37,9 @@ Therefore, valid values to the \code{month} argument should be limited to \code{c(3, 6, 9, 12)}, unless there is an anomaly in FCA's reporting/publishing. Check \url{https://www.fca.gov/bank-oversight/call-report-data-for-download} to ensure the data is available for the quarter you are interested in. +Ketchbrook Analytics downloads these files and stores them in a public AWS +S3 bucket, which is the location that \code{download_data()} retrieves them +from. } \examples{ \donttest{