Upload Rulemaking OCR text size limitation issue

[fec-jli]

What we’re after

When uploading the OCR text to the OpenSearch service, the task failed with the following error:
opensearchpy.exceptions.TransportError: TransportError(413, '{"Message":"Request size exceeded 10485760 bytes"}')

Ex: rm: 2014-01 REG that include 151 documents.

Related ticket(s)

#6481
#6367

Action item(s)

• Request ITSS to clean the OCR text by removing special characters and reinsert the cleaned data into the table.
• Research OpenSearch to determine whether the request size limit can be increased from 10 MB to a larger value.

Completion criteria

(What does the end state look like - as long as this task(s) is done, this work is complete)

• All OCR text can be uploaded to Opensearch successfully

[patphongs]

OCR / OpenSearch upload issues – summary

Current status

• Received OCR text for 3,500+ documents in stage/dev.
• Upload to stage OpenSearch failed due to technical issues.
• Some rulemakings contain special characters / garbled text, causing documents to exceed size limits and fail indexing.
• 10MIB max document size currently enforced.

Errors & findings

• Upload error starting around 2014-01 (“bytes exceeded”).
• Data successfully loaded from 2026 back to 2014-02, but rulemakings exist back to 1989.
• Some comments are 500+ pages with minimal text per page, creating unnecessary size bloat.
• ITSS attempted cleanup but uploads still failed with the same error.
• Historical precedent: large MUR archive documents previously required manual cleanup or splitting.

Next steps / technical actions

• Traverse rulemakings year-by-year from 1989 forward to identify problematic rulemakings.
• Investigate rulemakings 2014-01 and earlier first.
• Consider splitting documents over ~XXX pages into smaller parts for indexing.
• Evaluate whether to increase OpenSearch capacity temporarily to index large files, then downgrade later.

CMS bug fecgov/fec-cms#6976

• Comment link sometimes missing when doc_description is null.
• Fix: display doc_type_label when doc_description is null.
• Example: https://stage.fec.gov/legal/rulemakings/2014-01/ has many comments that are not linked.

[pkfec]

There are five rulemakings that I was unable to load into the OpenSearch (OS) service in the staging environment. I encountered an out-of-memory error when attempting to reload them multiple times. Here is the list:

1. 2014-01
2. 2011-02
3. 1998-06
4. 1996-08
5. 1997-03

[patphongs]

OCR & document processing notes

• OCR text cannot currently be generated from scanned image PDFs; when attempted, text is often garbled.
• If documents are scanned PDFs, they will not be sent to the web team for OCR processing.
• Very large files may be excluded from OCR processing due to size limitations.
• OGC currently uses its own PDF tool to generate OCR text when needed.
• Need to determine a safe maximum file size for OCR processing and indexing.

[patphongs]

PDF loading bugs

@pkfec Certain PDFs are failing to load in stage, returning either a generic “PDF failed to load” message or a NoSuchKey error from the storage bucket.

Examples:

• PDF fails to load:
  • I noticed that this fails in legacy as well: https://sers.fec.gov/fosers/showpdf.htm?docid=306177
  • https://stage.fec.gov/legal/rulemakings/2014-07/?doc_id=306177
  • https://stage.fec.gov/files/legal/rulemakings/2014-07/306177/Commissioner-statement.pdf

• NoSuchKey bucket error:
  • This loads ok on legacy, perhaps it's just missing from our bucket for some reason? https://sers.fec.gov/fosers/showpdf.htm?docid=306176
  • https://stage.fec.gov/legal/rulemakings/2014-07/?doc_id=306176
  • https://stage.fec.gov/files/legal/rulemakings/2014-07/306176/Commissioner-statement-#2.pdf

[pkfec]

For RM_2014-07 with doc_id 306177 on SERS doesn't open. Reached out to ITSS and awaiting for their response

[pkfec]

For RM_2014-07 with doc_id 306175 and 306176, the filenames contain special characters such as the # symbol, which is encoded as %23 in the URL

To properly handle this edge cases, we should implement percent-encoding in the API before uploading the PDF files and generating the URLs for the OS service. However, the API changes will require additional time to implement the encoding and reload all RMs to the OS service.

@patphongs is currently investigating whether this URL encoding can be implemented on the CMS side instead, so it can also handle other edge cases (for example: #, ?, %, spaces, +, &, commas, [, ], (, and )).

[patphongs]

@pkfec I found another anomaly. Rulemaking 2021-01 has label.doctype.10= as part of the doc_type_label value itself. Therefore the label is outputting like this on the CMS page. It should just be "Hearing Transcript". This is the API call. Do you know why this is? Do you need the CMS app to handle this? Or is there an API fix you could apply?

"level_2_docs": [
{
  "doc_category_id": 2,
  "doc_category_label": "Hearing",
  "doc_date": "2023-03-22",
  "doc_description": "Hearing Transcript",
  "doc_entities": [],
  "doc_id": 422399,
  "doc_type_id": 10,
  "doc_type_label": "label.doctype.10=Hearing Transcript",
  "filename": "230322 Candidate Salaries (revised)",
  "is_comment_eligible": false,
  "is_key_document": false,
  "level_1": 3,
  "level_1_label": "Notice of Hearing",
  "level_2": 2,
  "level_2_label": "Hearing Materials",
  "sort_order": 0,
  "url": "/files/legal/rulemakings/2021-01/422399/230322-Candidate-Salaries-(revised).pdf"
},

[pkfec]

@pkfec I found another anomaly. Rulemaking 2021-01 has label.doctype.10= as part of the doc_type_label value itself. Therefore the label is outputting like this on the CMS page. It should just be "Hearing Transcript". This is the API call. Do you know why this is? Do you need the CMS app to handle this? Or is there an API fix you could apply?

"level_2_docs": [
{
  "doc_category_id": 2,
  "doc_category_label": "Hearing",
  "doc_date": "2023-03-22",
  "doc_description": "Hearing Transcript",
  "doc_entities": [],
  "doc_id": 422399,
  "doc_type_id": 10,
  "doc_type_label": "label.doctype.10=Hearing Transcript",
  "filename": "230322 Candidate Salaries (revised)",
  "is_comment_eligible": false,
  "is_key_document": false,
  "level_1": 3,
  "level_1_label": "Notice of Hearing",
  "level_2": 2,
  "level_2_label": "Hearing Materials",
  "sort_order": 0,
  "url": "/files/legal/rulemakings/2021-01/422399/230322-Candidate-Salaries-(revised).pdf"
},

Good catch, @patphongs. There are a total of 46 rulemakings displaying incorrect doc_type_label values. PR #6497 will correct these labels.