Format Design Document

[krowvin]

Statement

Continue to allow CSV and JSON as formats going forward. CDA will officially support JSON and CSV. Additional formats may be introduced via versioned changes or explicit content negotiation if a documented use case arises.

Purpose / Need

CSV, without a viewing software, is also highly viewable by humans compared to JSON.

CSV isolates tabular datasets and minimizes structural overhead, making it easier to inspect in spreadsheet-oriented workflows. It is easily imported into various end-user client software (MS Excel/Sheets), and does not require extra code on the client in order to convert from one format to another when JSON is the only option.

If a student, stakeholder, news outlet, other agency, and others wanted to easily manipulate data without first needing to parse it in JSON they would be able to do so with CSV. CSV, without a viewing software, is also highly viewable by humans compared to JSON.

Example: Sharing a link to a specific dataset. This can be handled via a direct CDA link that when clicked, downloads the CSV file directly.

Relies on: Content-Type: text/csv being set by default.

JSON

JSON is the default response format. Some legacy endpoints currently return XML due to upstream constraints; these are considered transitional and should be normalized where feasible.

It works well with downstream programming needs and offers additional metadata beyond what CSV provides.

Expectations

1. Query parameter (?format=) overrides Accept header to support browser downloading via URI control.

• If neither is present, default to JSON.

2. CSV must only include the headers and data fields. No additional metadata at the top.

• UTF-8 Encoded
• Empty values MUST return empty fields, NOT null. I.e. date,4.0,, NOT date,4.0,null,null

3. If data response types must change, such as headers or keyvalue pairs in a response, the version MUST be updated to reflect this in the accept header to match the practices put in place already for versioning outlined here:

• Update API design philosophy #942

4. Proper errors must be returned when a format is failed to parse. I.e. invalid json or CSV file that does not contain any data/fails to render CSV.

• For generic errors, caused by the user, make sure to return 400 Bad Request
• For client-side/invalid format errors, must return 406 Not Acceptable or 415 Unsupported Media type where 406 are media types we deem will never be implemented.

Related Issues

• Changing format= Does Not Work with Request Headers #1257

[zack-rma]

I'd like to verify the formatting decision being suggested here. Is this the intended outcome for response formatting?

[zack-rma]

@MikeNeilson @krowvin

[krowvin]

@zack-rma

Almost, that is mostly correct from what im seeing.

Oh the fourth row you have test but I assume that's a typo

You also have the mine type typed out in the row keys

I assume you mean JSON/TEXT/XML/*/no query=

Noting this looks to remove ambiguity of header versus query param error and send it back as the value in your table.
As noted by Ryan's comment in the referenced issue above.

Deferring to @MikeNeilson on final thoughts

Edit: had to clarify what you mean by the row keys being mime types in your table