Project: FY26 Dataverse Performance Improvements

[cmbz]

Overview

• Harvard Dataverse and other large Dataverse installations sometimes experience performance issues (e.g., lengthy page loads) due to their size (e.g., number of collections, datasets, and files), complexity, and factors such as their operating environments.
• Although some of these problems are due (in part) to JSF UI inefficiencies and may be mitigated when the SPA is deployed in production, other performance issues will remain.
• This issue describes a year long project to investigate and mitigate these issues, particularly for the Harvard Dataverse Repository.
• Related issues will be labeled clearly and added as sub-issues below.

Workplan

• Master issue for improving stability of the production service.  dataverse.harvard.edu#380
• A local issue for tracking development issues in the main queue that are crucial for the stability of HDV dataverse.harvard.edu#401

Timeline

• Start: July 2025
• End: June 2026

[landreev]

Status: November 2025

I keep track of the prod. stability-related work in the master issue in the local queue: IQSS/dataverse.harvard.edu#380
It has a few sub-issues. The most important one of which is this one, that keeps track of dev. efforts in the main repo that are critical or at least very relevant to keeping the prod. servers operational: IQSS/dataverse.harvard.edu#401. Not all of them are performance related, stricly speaking, some are for relatively minor bug fixes or improvements that are nevertheless critically important for HDV.
The above has been updated to reflect the progress made during the last month.

[rnichols7]

Status: December 2025

The primary focus of the prod. work performed was on instituting and enforcing limits, on storage (quotas) and on the number files in individual datasets (the latter is essentially to "protect the users from themselves", proactively preventing them from creating unmanageable/unaccessible datasets). This involved working with curation on designing the policy and development work for fixing and/or adding the components necessary for enabling these feature in the specific HDV environment. A prime example was the PR IQSS/dataverse#11987, completed and immediately deployed in prod. via a service patch. (this was reflected in the master dev. issue IQSS/dataverse#401 as well).
Additional efforts involved refining the defenses against spam; and working with curation on potential redesigns of the role assignment framework for the datasets in the root collection that could streamline and ease their curation workload. (L.A.)

[landreev]

Status: January 2026

The post-6.9 performance investigation (IQSS/dataverse#12071) was completed.
Per its results, a PR improving the indexing performance (IQSS/dataverse#12082) was made by Jim Myers and merged; another dev. issue, (IQSS/dataverse#12102) was opened.
Further refinements were added to the still open PR IQSS/dataverse#11971 and deployed in prod., stabilizing some issues with Globus transfers.
Some effort was put into making our process of building custom deployment Dataverse war file more formalized, creating dedicated branches and pull requests for the extra pieces that are added on top of the official release.