Fix/applics 1683 improve loading speed examination timetable items by JanPhillips · Pull Request #2968 · Planning-Inspectorate/back-office

JanPhillips · 2025-10-05T12:17:36Z

Optimisation of loading speed for examination timetable items

The loading speed of the examination timetable items page is slow due to how the number of documents within a folder and its subfolders is counted. Currently, a recursive traversal of the folder structure, which involves calls to the database, this happens for each examination timetable item, when there are many timetable items on the page, this causes the page to load slowly. The document count is required to determine if an examination timetable item can be deleted, items with no files can be deleted.

This PR has sped up the loading time by utilising a concept called Materialised Paths. Two new columns have been added to the Folder table, documentCount and path, combined with new logic for setting, updating, and getting these values, only 1 database query is required to get the the document count for any folder. The documentCount for all ancestor folders are update when files are uploaded, deleted or moved. In terms of Big O notation I believe this changes the effort to get the document count of a folder from O(n) (where n = number of child folders) to O(1).

For existing Folders

Included in this PR is a script file, apps/api/src/database/seed/create-paths.js that can by run on a server to set the path value for each Folder. There is also a test script apps/api/src/database/test-scripts/create-paths-test-script.js that can be run afterwards to check the changes were successful.

Before running

Run a migration on the DB, to ensure that the Folder table gets the new columns documentCount and path

Issue ticket number and link

Type of change 🧩

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Other (please explain in the description section above)

Checklist before requesting a review

I have performed a self-review of my own code
I have double checked this work does not include any hardcoded secrets or passwords
I have made corresponding changes to the documentation
I have provided details on how I have tested my code
I have referenced the ticket number above
I have provided a description of the ticket
I have included unit tests to cover any testable code changes

… for new folders (APPLICS-1683)

…deleted (APPLICS-1683)

… is empty (APPLICS-1683)

…tests(APPLICS-1683)

Copilot

Pull Request Overview

This PR improves the loading speed of examination timetable items by implementing a document count tracking system using folder document counts instead of potentially expensive real-time queries.

Adds documentCount and path fields to the Folder database model
Implements functions to efficiently track document counts when documents are added/removed/moved
Updates examination timetable logic to use the cached document count instead of querying submissions

Reviewed Changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
apps/web/src/server/views/applications/case-timetable/components/timetable-items-list.component.njk	Updates template logic to handle numeric submission counts and null states
apps/web/src/server/applications/case-timetable/tests/snapshots/applications-timetable.test.js.snap	Updates test snapshots to reflect removal of delete links for folders with submissions
apps/api/src/server/repositories/folder.repository.js	Adds document count tracking functions and folder path management
apps/api/src/server/applications/examination-timetable-items/examination-timetable-items.controller.js	Replaces submission validation with document count lookup
apps/api/src/server/applications/application/documents/document.service.js	Integrates document count updates when documents are created/deleted
apps/api/src/database/schema.prisma	Adds documentCount and path fields to Folder model

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

apps/api/src/server/repositories/folder.repository.js

...erver/applications/examination-timetable-items/__tests__/examination-timetable-items.test.js

…PPLICS-1683)

…e is correct (APPLICS-1683)

Copilot

Pull Request Overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-06T12:11:50Z

apps/api/src/server/repositories/folder.repository.js

+		return databaseConnector.$executeRaw`UPDATE folder SET documentCount = documentCount - ${decreaseBy}
+		WHERE ${folder.path} LIKE CONCAT(path, '%')`;


This raw SQL query is vulnerable to SQL injection. The folder.path value should be parameterized or sanitized before being used in the query.

My interpretation of the docs on how to use $executeRaw is that this is safe from SQL Injection

Copilot · 2025-10-06T12:11:50Z

apps/api/src/server/repositories/folder.repository.js

+		return databaseConnector.$executeRaw`UPDATE folder SET documentCount = documentCount + ${increaseBy}
+		WHERE ${folder.path} LIKE CONCAT(path, '%')`;


This raw SQL query is vulnerable to SQL injection. The folder.path value should be parameterized or sanitized before being used in the query.

My interpretation of the docs on how to use $executeRaw is that this is safe from SQL Injection

Explained in better detail here:
https://www.bouncesecurity.com/blog/2024/02/20/when-the-safe-is-bad-and-the-unsafe-is-safe.html

prisma/prisma-client-js#727 (comment)

https://wesbos.com/tagged-template-literals

… efficiency (APPLICS-1683)

Copilot

Pull Request Overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

...b/src/server/views/applications/case-timetable/components/timetable-items-list.component.njk

apps/api/src/server/repositories/folder.repository.js

Copilot

Pull Request Overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

apps/api/src/server/repositories/folder.repository.js

apps/api/src/database/test-scripts/create-paths-test-script.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

...rc/server/applications/examination-timetable-items/examination-timetable-items.controller.js

…nation-timetable-items

ypk · 2025-10-10T13:09:03Z

apps/api/src/server/applications/application/documents/document.controller.js


+	// updateDocuments[0] is the array of Document objects updated
+	// updateDocuments[1] is array of DocumentVersion objects
 	if (updateDocuments[0].count === 0 || updateDocuments[1].count === 0) {


we could rename updateDocuments[0] to updatedDocumentObjects and updateDocuments[1] could be renamed to documentVersionObjects something like const [updatedDocumentObjects, documentVersionObjects] = updateDocuments;

ypk

I added a comment

…nation-timetable-items

JanPhillips added 4 commits October 1, 2025 10:54

feat(api): added columns path and documentCount to folder;path is set…

afad79d

… for new folders (APPLICS-1683)

feat(api): documentCount is increased/decreased when a file is added/…

18d18dd

…deleted (APPLICS-1683)

feat(api): examination timetable item can only be deleted when folder…

d7dbd6c

… is empty (APPLICS-1683)

fix(api): documentCount of folder is updated when files moved; fixed …

881ec08

…tests(APPLICS-1683)

JanPhillips requested a review from Copilot October 5, 2025 12:17

JanPhillips self-assigned this Oct 5, 2025

Copilot AI reviewed Oct 5, 2025

View reviewed changes

JanPhillips added 2 commits October 5, 2025 17:41

feat(api): script for populating path column for each Folder in db (A…

e05c80f

…PPLICS-1683)

fix(api): a test script that asserts the of each entry in Folder tabl…

8818d5e

…e is correct (APPLICS-1683)

JanPhillips requested a review from Copilot October 6, 2025 12:10

Copilot AI reviewed Oct 6, 2025

View reviewed changes

JanPhillips added 2 commits October 6, 2025 14:22

fix(api): small refactoring and more error handling (APPLICS-1683)

5493276

fix(api): test script checks paths for all Folders, Promise.All() for…

21448ad

… efficiency (APPLICS-1683)

JanPhillips requested a review from Copilot October 6, 2025 13:29

Copilot AI reviewed Oct 6, 2025

View reviewed changes

JanPhillips requested a review from Copilot October 6, 2025 14:17

Copilot AI reviewed Oct 6, 2025

View reviewed changes

apps/api/src/server/repositories/folder.repository.js Show resolved Hide resolved

apps/api/src/database/test-scripts/create-paths-test-script.js Outdated Show resolved Hide resolved

fix(web): using strict equality on submission count

d05263f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

JanPhillips force-pushed the fix/APPLICS-1683-improve-loading-speed-examination-timetable-items branch from a3664b6 to d05263f Compare October 6, 2025 14:26

JanPhillips requested a review from Copilot October 6, 2025 14:27

Copilot AI reviewed Oct 6, 2025

View reviewed changes

fix(api): using const instead of let (APPLICS-1683)

0835393

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

JanPhillips requested a review from Copilot October 6, 2025 14:34

Copilot AI reviewed Oct 6, 2025

View reviewed changes

...rc/server/applications/examination-timetable-items/examination-timetable-items.controller.js Show resolved Hide resolved

fix(api): refactored recursivelySetPaths() (APPLICS-1683)

8288f8c

JanPhillips marked this pull request as ready for review October 10, 2025 12:14

Merge branch 'main' into fix/APPLICS-1683-improve-loading-speed-exami…

a245d22

…nation-timetable-items

JanPhillips requested review from DavidJWhiteDev and emil-placheta October 10, 2025 12:27

ypk reviewed Oct 10, 2025

View reviewed changes

Merge branch 'main' into fix/APPLICS-1683-improve-loading-speed-exami…

b31ff44

…nation-timetable-items

		return databaseConnector.$executeRaw`UPDATE folder SET documentCount = documentCount - ${decreaseBy}
		WHERE ${folder.path} LIKE CONCAT(path, '%')`;

Conversation

JanPhillips commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Optimisation of loading speed for examination timetable items

For existing Folders

Before running

Issue ticket number and link

Type of change 🧩

Checklist before requesting a review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

JanPhillips Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

JanPhillips Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

ypk Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

ypk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JanPhillips commented Oct 5, 2025 •

edited

Loading

JanPhillips Oct 6, 2025 •

edited

Loading

ypk left a comment •

edited

Loading