OpenAPI spec

[bitmotivator]

This is a work-in-progress OpenAPI specification based on the preliminary design document.

Update 2025-07-01

As discussed, here's a partial prototype implementation in Go, other languages coming shortly. Findings from the implementation:

1. The /dmps and /dmps/{id} endpoints are sufficient for any use case except the security one, assuming that the PUT endpoint is correctly implemented. (See the Go implementation here and the underlying client code here.)
2. The PATCH implementation is also unnecessary as it only moves the update burden to the server from the client, but is also a lot harder to implement as JSON pointers would need to be supported.
3. Polymorphism is tricky and we may need to tweak the spec slightly to make it work with various generators. Unfortunately, OpenAPI and JSON schema allow for a lot of things that can't actually be implemented easily.

---

Original text

Note

Please leave your comments on the individual lines in the code so that we may have a threaded discussion.

Tip

You may want to switch to the "Commits" tab and select the individual commits as the full diff includes the Petstore API, which hinders readability.

Note

The DMP object has been added since a lot of the technical decision of this API depend on the DMP endpoints being available.

TODO

• DMP
• Cost
• Project
• Contributor
• Contact
• Funding
• Security and Privacy
• Technical Resource
• Metadata
• Distribution
• License
• Host
• Dataset

Proposal

The OpenAPI spec is already ~2000 lines of code. The final one is looking to be 5-10x that. We should stop adding more endpoints to the spec and implement what we have to see what design flaws pop out of the spec and if maybe a subset of APIs would already fill the requirements.

Conflict resolution on concurrent writes

This section illustrates how conflicts can be resolved when two tools have concurrent write access to a DMP.

Partial update to a DMP

All changes to the DMP are performed locally and sent back to the server in full. The If-Unmodified-Since and If-Match headers guard against race conditions as the client will detect any modifications (see the next illustration).

sequenceDiagram
    autonumber
    actor User as User
    participant DMPTool as DMP tool
    loop "Until successful"
        Note over User,DMPTool: Get the full DMP:
        User->>DMPTool: GET /dmps/{id}
        DMPTool-->>User: 200 OK<br />DMP with ID
        User->>User: apply change to DMP locally
        Note over User,DMPTool: Send the modified DMP:
        User->>DMPTool: PUT /dmps/{id}<br />If-Match header
        DMPTool-->>User: 200 OK<br />DMP with ID
    end

Loading

Conflict resolution on partal update

Here two users modify a DMP concurrently. The second user fails and has to re-apply their modification on top of the first user's modification. We dispense with adding the loop for both users to the graphic for clarity.

sequenceDiagram
    autonumber
    actor User1 as User 1
    actor User2 as User 2
    participant DMPTool as DMP tool
    Note over User1,DMPTool: Both users fetch the DMP:
    User1->>DMPTool: GET /dmps/{id}
    DMPTool-->>User1: 200 OK<br />DMP with ID
    User2->>DMPTool: GET /dmps/{id}
    DMPTool-->>User2: 200 OK<br />DMP with ID


    Note over User1,DMPTool: Both users apply the update locally:
    User1->>User1: apply change to DMP
    User2->>User2: apply change to DMP

    Note over User1,DMPTool: User 1 gets their update in first:
    User1->>DMPTool: PUT /dmps/{id}<br />If-Match header
    DMPTool-->>User1: 200 OK<br />DMP with ID
    Note over User2,DMPTool: User 2 update fails:
    User2->>DMPTool: PUT /dmps/{id}<br />If-Match header
    DMPTool-->>User2: 409 Conflict
    Note over User2,DMPTool: User 2 re-applies the update:
    User2->>DMPTool: GET /dmps/{id}
    DMPTool-->>User2: 200 OK<br />DMP with ID
    User2->>User2: apply change to DMP
    User2->>DMPTool: PUT /dmps/{id}<br />If-Match header
    DMPTool-->>User2: 200 OK<br />DMP with ID

Loading

Current implementation notes, to discuss, in no particular order

Number of endpoints

Currently, there are 13 objects envisioned for this API. Each of these needs at least GET, POST, PUT and DELETE, some needing GET endpoints for listing and possibly PATCH. This means we will need to implement ~50-80 endpoints for this API alone, handling race conditions with partial updates and authentications correctly. Therefore, the current OpenAPI spec marks all but the main DMP endpoints as optional, allowing for a 405 response if the endpoint is not implemented.

POST /dmps/query (listDMPs)

This endpoint, as designed in the preliminary spec, has several issues for usability, but most importantly, it breaks caching. Any DMP tool author wishing to expose their DMPs to the public will run into issues with this endpoint as designed because this is likely the most expensive query of the whole API and AI scrapers are currently ruining the fun for everyone. Just 2 weeks ago we had one of the third party public APIs that DAMAP relies on go down and implement a fairly restrictive firewall because of this.

It is also worth noting that the previously-discussed limitation of URL length is fairly outdated. The original error code (415) was introduced in 1999 and things have changed a lot since then. The current HTTP RFC 9110 specifies that implementations should support at least 8k in length. Cloudflare, for example, supports URLs up to 16 kB. DMP tool vendors are free to tune their servers for however long URLs they want to support, but requests that are complex enough to include more than 8k in the query string will likely cause an unreasonably complex query to be sent to the backing database and should not be allowed anyway.

As a side note, using POST for read requests breaks the HTTP spec and a lot of assumptions various tools and users make about these kinds of requests. Simply opening up the query URL in the browser for visual inspection also makes for a nicer experience than having to crank out a Postman or curl to fetch a simple list.

POST /dmps (createDMP)

The response to the DMP creation in the original draft specifies that the response should merely return a message. However, since the DMP itself doesn't have a canonical ID in the request, the requester would have no way to know what the system-assigned ID for the DMP is. This has been changed to include the full DMP in the response. We can also switch this to return only the ID, optionally in a Location header.

ID fields

Currently, there are several ID fields listed in the specs. However, many of these IDs are ambiguous because the common DMP standard supports multiple types of these IDs (e.g. ORCID, ISNI, etc). If left as-is, the backing system will have to "guess" which identifier type is meant, which is leaves room for interpretation on how this is implemented. If we truly want to search by these identifiers, we should specify how this search should behave. Ideally, we should also expose the canonical, system-assigned identifiers for these objects in the common schema as that is the only way to unambiguously identify these records.

It is also worth noting that the common DMP standard currently specifies some of these IDs as full URLs, but still requires the type to be noted in an extra field. A strict implementation of the listDMPs API, for example, would not find a DMP based on an ORCID ID, only if the full ORCID URL was entered into the query. One solution could be that all IDs need to be submitted as full URLs and the common standard and API spec could make this clear by specifying the appropriate constraints.

Additionally, the previous decision to introduce system-specific IDs outside of the common standard also means that the fields for searching DMPs are becoming ambiguous as a user would not know if this is searching among the system-dependent or the system-independent IDs. Mandating system-specific IDs on every level of the common standard is also wrong because some systems may not have them at all if they store complex objects as one. DAMAP is currently experimenting with this approach and we may do away completely with system-generated contact IDs, etc.

PATCH method

The current description of the PATCH method uses JSON patch. However, since this is somewhat complex and requires the server to perform JSON slicing and possibly locking in the database, this method is included as optional, leaving it open for the server implementations to not support it if it is not feasible. Clients are expected to retry the request as a PUT if they receive a 405 Method Not Allowed.

Datasets API

The datasets API is currently not implemented due to the lack of time. However, it also bears discussing that this API is equivalent to applying a PATCH to the respective subobject. We could opt to make PATCH support mandatory instead, giving more flexibility to clients.

OpenAPI version

The current OpenAPI document is written for version 3.1, which allows for including the common DMP standard JSON document directly. However, this is a fairly new version and a lot of code generators won't support it. Once we are done with this process it may be worth backporting to 3.0.

DMP standard versioning

The current specification only supports the common standard v1.2. Since this standard is (hopefully) here to stay, it will likely have more versions over the years. It bears discussing how different DMP standard versions will be supported in this API, especially if an implementation is not able to automatically upgrade DMPs to a later version. Typical solutions here would include not hard-coding support for a specific DMP schema version, but rather supporting arbitrary versions, even custom ones with extra fields. The server could return the corresponding JSON schema for the DMP it presents and may enforce which schema versions it is willing to accept for creation.

Funding endpoint

While trying to implement the funding endpoint it became clear that the projects have no unique ID within the DMP, only their order in the array. This means that we could only construct an artificial URL, such as:

/dmps/{id}/projects/0/funding

This would mean that the index of the project will become part of the URL, which is less than ideal if the order changes. We should either expose a system-generated project ID (see ID fields note), or make the PATCH method support mandatory to allow for easy updating of a funding object.

Note that the previous decision to add the ID fields only in the OpenAPI spec means that we will have to duplicate data from the common standard and keep it updated in both places.

[bitmotivator]

While implementing the Go SDK, I realized that this is incorrect. It should read:

Suggested change

	DMPListResponse:
	type: object
	properties:
	total_count:
	type: integer
	minimum: 0
	items:
	$ref: "#/components/schemas/DMPWithID"
	DMPListResponse:
	type: object
	required:
	- total_count
	- items
	properties:
	total_count:
	type: integer
	minimum: 0
	items:
	type: array
	items:
	$ref: "#/components/schemas/DMPWithID"

[bitmotivator]

While implementing the Go SDK, I came to the conclusion that supporting the PATCH method would require adding additional dependencies and is likely not worth it. The PUT method can achieve our goals nicely.

[bitmotivator]

Having implemented the Go SDK partially, I believe this endpoint to be unnecessary as the PUT /dmps/{id} endpoint can do the job nicely with conditional updates. The only reason to keep these extra endpoints would be if we have systems that we don't fully trust and want to limit the blast radius of possible changes to only the subordidnate objects.

[bitmotivator]

There should be a generic bad request response defined for every endpoint.

[briri]

Agreed. I see the GenericError response below but don't see any 500's in the endpoint definitions here. Do we also want to include them as well? Not that our APIs would ever have unhandled exceptions 😁

[bitmotivator]

Sadly, it seems that code generators get very very confused by using oneOf in combination with allOf down the chain, despite the discriminators. This needs to be dumbed down. Inheritance is tricky because not all languages support it equally.

[briri]

This looks great @bitmotivator thanks for all of your work putting it together

[briri]

Should we include an explicit example here to help explain the AND/OR difference, something like:
Fetch all DMPs created after June 1, 2025 AND are written in English OR Dutch - ?languages[]=eng&languages[]=nld&created_after=2025-06-01

[bitmotivator]

I agree, the final version should contain a more elaborate example (or even multiple).

[briri]

Not sure if it makes sense to sort on an embargo flag. Embargo is attached to the dataset/distribution so many DMPs would have multiple datasets and possibly a mix of embargo statuses.

I would also expect that this ties heavily into who the caller is and what they're authorized to see. In many situations we wouldn't want to return embargoed information at all. Its embargoed 😄

[bitmotivator]

I was following the spec document from the hackathon which said one should be able to sort by ALL fields. I'm always happy to cut functionality as it makes it easier to implement :D

[briri]

Do we want to include 403 and 500?

[bitmotivator]

I don't think there is any reason why you wouldn't be allowed to list DMPs once authenticated, which would result in a 403. The 500 I also wouldn't include because it basically means something went horribly wrong, so we can't make any statements about the contents of such a response.

In the generated code this would be the "unexpected status code" case, which will need to be handled explicitly. In my prototype SDK (I'll post the code shortly) I am turning these into an exception/error depending on the language.

[briri]

Agreed. I see the GenericError response below but don't see any 500's in the endpoint definitions here. Do we also want to include them as well? Not that our APIs would ever have unhandled exceptions 😁

[briri]

I see the DMPInvalidResponse definition below includes a space for all the details but not sure if we should note here in the description that the response will include detailed info about what was invalid

[bitmotivator]

I'm still working out if this is feasible. I was planning to return a set of validation errors with JSON Path pointers, but I realized that this may actually be quite difficult to compile for implementers who have a different data structure internally. They would need to translate their internal error message pointers to the common schema pointers, which I think would lead to most people just not supplying them at all.

[bitmotivator]

@briri thank you so much for your review! 🙏

I was planning to show up to tomorrow's meeting with a full Typescript client (with JSON Forms) and a Go server demo, but life got in the way, so that will have to wait until the next meeting.

At any rate, here's my prototype Go code, most of it generated. Note that I have slightly modified the OpenAPI spec based on issues I found with various generators.

Things that are worth looking at:

1. The top level client interface
2. The top level backend (server) interface

This already solves the concurrent editing issue and I believe, this makes it unnecessary for us to implement more than the top-level /dmps and /dmps/{id} endpoints. The PATCH method can also be dispensed with since the PUT fully replaces it, moving the DMP update to the client side. The only reason to create separate endpoints would be if we don't trust a certain client to touch or update the whole DMP, but would trust it to see/update a part of the DMP.

[briri]

Thanks again for all your work on this @bitmotivator I'll have a look at the Go code later this week. I really like the introduction of the headers to help manage conflict resolution.

The only reason to create separate endpoints would be if we don't trust a certain client to touch or update the whole DMP, but would trust it to see/update a part of the DMP.

I think this is partially a matter of trust but also a matter of just reducing the likelihood of errors. My team has already had a use case where we only permit an external service to add dataset DOIs to existing DMPs. We created a specific endpoint for them to do that. I can foresee future scenarios though where a grants management system might for example need to be able to modify funding status or attach grant ids to a DMP. They do a fetch for the DMP and do not get the entire DMP metadata record for security reasons. They then send a PUT and some info is lost because they didn't even get it to begin with.

After our last call, I am a little concerned that we might be limiting ourselves early on due to limitations with the technologies we're using/considering (e.g. the length of the openAPI spec or whether or not some languages can efficiently handle JSON patch).

I don't personally have experience with JSON patch but if we don't use that or individual CRUD endpoints for each logical entity now we might be limiting ourselves in the future. The DMP teams will likely need to develop some complex logic to do auth checks for the single PUT endpoint to determine what exactly the caller has access to update. If we don't recommend an approach now we will all develop our own approaches to solve this problem and that will make things harder in the future if we decide to put out a new version.

Those are of course hypothetical use cases, so I am interested to hear what others think.

[bitmotivator]

Hi @briri, very good points. I agree that sometimes, having separate endpoints would just be very convenient, but there's a workaround for that on the client side (see below). The main issue I had with the approach we were doing things is the sheer number of endpoints that would need to be implemented. If it turns out that the few endpoints implemented in the prototype are not enough, we should very selectively and carefully pick what we need instead of going for the wholesale all-objects-must-have-endpoints approach as that would result in an unimplementable API.

Now, for the workaround. The reason why I suggested the SDKs instead of going OpenAPI without them is precisely because the partial update mechanism needs to be made easier. With the Go SDK (once finished), the update of adding a dataset would look like this:

cli, err := client.New("https://your.dmp.service")
if err != nil {
    panic(err)
}
cli.UpdateDMP(
    context.TODO(),
    "1234", // <- DMP ID
    // Lambda function that can be called repeatedly during reconciliation:
    func(ctx context.Context, dmp model.DMPWithID) (model.DMPWithID, error) {
        dmp.Dmp.Dataset = append(dmp.Dmp.Dataset, model.Dataset{
            // Add your dataset features here
        })
        return dmp, nil
    },
)

The lambda function lets the SDK run the reconciliation repeatedly until it succeeds. If the DMP has been updated in the meantime, the client SDK would pull the new changes and just retry the function that modifies the DMP.

This may not be the most straight-forward to implement if done from scratch, but that's why having the SDK would be so useful. It would also make it more likely that server-side implementers would implement the API correctly since that part is really easy. Personally, I'd really like to get to a spec that's a no-brainer for DMP system authors to implement well, if at the expense of having to use an SDK on the client side.

Re: JSON patch, this would shift the burden of reconciliation to the server side, but it wouldn't make it go away. It also adds quite a bit of complexity for folks with strictly and statically typed languages. The only benefit here would be permission checking as the server side directly sees what the desired changes are and can bail out instead of having to diff entire the DMP object.

If we on the other hand add endpoints (e.g. for datasets), that would just mean that the server side has to implement locking on the server side anyway since the main DMP endpoint also returns the datasets. The if-modified-since/if-match gets a lot more complicated to implement.

Alternatively, we could make the main endpoint not return the datasets, that way the datasets endpoints would be the only way to access them. That, however, also means that there's no easy way to pull out an entire DMP from the system which defeats the purpose a little bit.

Honestly, I don't know what a good solution here is. I think we need to try and see how painful it is to use the SDKs once we have at least a few working and then decide how to proceed.

[vknaisl]

After extensive discussions, we reached a consensus on using a POST-based query endpoint. While I personally agree that a GET endpoint would be more appropriate for querying resources—especially in line with HTTP semantics—and perhaps even that a GraphQL-based approach could be the ideal long-term solution, the collective decision was to proceed with the POST-based endpoint for now.

Regarding concurrent updates, we should clarify whether we intend to support both If-Unmodified-Since and If-Match, or just one of them. From an implementation and maintenance perspective, I would strongly prefer that we choose one mechanism and stick with it to avoid unnecessary complexity.

I've reviewed the Python ecosystem, and it looks like OpenAPI Specification 3.1.0 is now supported—so from our side, everything should be good to go.

As for update semantics, my understanding is that we agreed to implement PUT endpoints first, and then revisit the idea of adding PATCH support later, based on need and experience. This ties into the broader architectural question: whether we want to expose individual CRUD endpoints for each entity (e.g., Dataset, Project, Funding), or just operate through a single CRUD interface on the root DMP object. My impression was that we agreed to provide full CRUD support for all major DMP sub-entities.

In that context, I share @briri's view that enabling partial updates via dedicated endpoints will be more practical and user-friendly. While SDKs can abstract some of this complexity for clients, they are not a universal solution. We must still build and maintain those SDKs, and developers working in unsupported or less common programming languages will be forced to interact with the raw API—making a well-structured endpoint design even more important.

Finally, I fully agree with the concern about the growing size and complexity of the OpenAPI YAML file. As the number of endpoints increases, so will the risk of maintenance overhead, copy-paste errors, and difficulties with future updates. To mitigate this, I propose we avoid excessive duplication by extracting common components such as standard error responses (400, 401, 403, 404) into the components section. This would simplify the spec and make it easier to maintain and evolve over time.

[MarekSuchanek]

This means you suggest versioning together with DCS? Full, or only major/minor?

[MarekSuchanek]

and = always both?

[bitmotivator]

The server must support both, while the client is free to chose only one. On a practical note, Etags would be superior since clocks are tricky. We could drop If-Unmodified-Since.

[MarekSuchanek]

OK, I wouldn't mind both (but not giving option to servers).

[MarekSuchanek]

more typical is limit in combination with offset (as alternative to page and size)

[bitmotivator]

This was transcribed from the original document I believe. No strong opinion on naming.

[MarekSuchanek]

I would prefer to avoid any freedom to implement it differently in various DMP platforms. Separating direction and fields can naturally cause such cases. Is there reason why not to follow best practices on using sort[] (if we need [] at all) such as:
?sort[]=title,asc&sort[]=modified,desc&sort[]=language

[MarekSuchanek]

So always full-date T full-time (incl. timezone)? Cannot use only ISO-compliant date for instance? (at least we do not have to deal with inclusivity of before/after...)

[MarekSuchanek]

/costs for consistency?

[bitmotivator]

I believe, there is only one cost object for a DMP, not multiple?

[MarekSuchanek]

a DMP has 0..n costs according to https://github.com/RDA-DMP-Common/RDA-DMP-Common-Standard?tab=readme-ov-file#dmp_table

[MarekSuchanek]

a single cost? how? or again array of the costs associated of a DMP?

[MarekSuchanek]

How to delete "the cost" (not "a cost" when compared to previous?) when there is no ID individual cost?

[MarekSuchanek]

same as for cost(s) - one or array to be updated by this?

[MarekSuchanek]

same as for costs (how to select which project to delete)

[bitmotivator]

Closing this in favor of #3, we can reopen it or take parts of it later if needed.