feat: add getRoutes method to list registered routes

[bjohansebas]

There is a real need to be able to list routes correctly, as clearly shown in issue (AlbertoFdzM/express-list-endpoints#99, expressjs/express#5961, expressjs/express#5858, expressjs/express#6481, https://github.com/wesleytodd/express-openapi/blob/main/lib/generate-doc.js#L16-L74, #122).
Additionally, I’ve been chatting with a Vercel employee who wants to make it possible to deploy Express servers on Vercel without any special configuration, the only blocker for them so far is that they can’t retrieve a list of the application's routes.

Previously, before the upgrade to path-to-regexp@8 (#117), it was possible to access the regex used for routing. However, this was replaced by the matcher property, which returns a function that extracts the path and parameters. This approach adds extra processing overhead, as it transforms a regex into a function—something unnecessary for such a simple goal. That said, it could still serve as an option for those attempting to list routes.

Currently, route layers have a path property, but it's initialized as undefined because the actual path is only resolved at runtime. This is due to the fact that req.url.pathname isn't defined until the request is received. Once the request arrives and the pathname is parsed (as shown in

router/index.js

Line 237 in 3d574ed

match = matchLayer(layer, path)

), the path property in the layer is then set with the correct value.

Proposed Solution

The solution would be to add a new property to the layer that gets initialized at the time the layer is created, using the path it was registered with. This would allow us to reliably retrieve the original path and make it possible for external tools or modules to list routes correctly.

Additionally, I'm proposing a new function that exposes the current stack's route definitions. This would reduce the need for third-party dependencies,or at least make it easier for those dependencies to support the new version of Express without relying on brittle workarounds.

Examples:

const router = new Router({ strict: true, caseSensitive: true })
const admin = new Router({ strict: true, caseSensitive: false })

admin.use((req, res, next) => {
  // some middleware for admin routes
  next()
})

admin.get('/', (req, res, next) => {
  res.end('Hello')
})

router.use("/admin", admin)

router.all('/:id', function (req, res) {
  res.end('Hello')
})

console.log(router.getRoutes())
// [
//   {
//     name: 'router',
//     path: '/admin',
//     methods: undefined,
//     keys: undefined,
//     router: [
//       {
//         name: 'handle',
//         path: '/',
//         methods: ['GET'],
//         keys: undefined,
//         router: undefined,
//         options: { strict: true, caseSensitive: false, end: true },
//       }
//     ],
//     options: { strict: true, caseSensitive: true, end: false }
//   },
//   {
//     name: 'handle',
//     path: '/:id',
//     methods: ['_ALL'],
//     keys: [{ name: 'id', type: "param" }],
//     router: undefined,
//     options: { strict: true, caseSensitive: true, end: true }
//   }
// ]

If this were to be released, in Express the function would be used like this:

import express from 'express'

const app = express()

app.get('/', (req, res) => {
  res.send('Hello World')
})

console.log(app.router.getRoutes())
// [
//   {
//     name: 'handle',
//     path: '/',
//     methods: ['GET'],
//     keys: undefined,
//     router: undefined,
//     options: { strict: true, caseSensitive: true, end: true },
//   }
// ]

Why not reverse-engineer the path?

In Express 4, there were some hacks based on inspecting the generated regular expressions. But these approaches aren't solid or future-proof. If path-to-regexp changes the regex structure, those solutions break. So it's not a sustainable or effective approach.

Furthermore, there are plans to reintroduce some features to path-to-regexp (pillarjs/path-to-regexp#380, pillarjs/path-to-regexp#379), so it's highly likely that the internal regex structure will change again. On top of that, unlike in Express 4, the generated regex is no longer directly accessible.

close expressjs/express#5961, close expressjs/express#6481, close #122).

[jonchurch]

We need clarification on the exact requirements. Can you have them open a feature request?

There are different things that could be exposed, which of these is the request?

1. what routes did a user register?

• e.g. router.get('/users/:id') returns { path: '/users/:id', methods: ['GET'] }
• This is what most of the linked issues want, for documentation/debugging purposes
• This is what the current implementation in PR provides

2. what will match what the user registered?

• e.g. with strict: false (default) /users matches /users and /users/, with strict: true it matches only /users
• e.g. with caseSensitive: false (default) /Users matches /Users and /users, with caseSensitive: true it matches only /Users
• This is necessary for traffic routing purposes, and what Im assuming vercel wants to do, route paths into the express app without passing through all traffic (assuming here without more info)
• The implementation does not cover this

If the ask is number 2, they would need to decide if they apply the routing rules themselves, tell users which options are supported, or we'd need to enhance the ouput to include either router options or otherwise take options into account.

(v good PR description ❤️)

Edit: More importantly, we'd want to answer this for ourselves if we add this feature. If folks depend on this they'll need to know what information is actually conveyed by the returned mapping. Strings they used, or strings router will recognize.

Current approach flattens the whole stack, so we are losing the nuance that different routers can be configured differently. Sweet spot between the two options could be expressed in output as:

  [
    {
      path: '/api',
      methods: ['GET'],
      options: { strict: false, caseSensitive: false }
    },
    {
      path: '/v1/users',
      methods: ['GET'],
      options: { strict: true, caseSensitive: true }
    }
  ]

[jonchurch]

#174 (comment)

[bjohansebas]

@jonchurch thanks for the quick review! Yes, what they shared with me (and I also asked them to comment here about their needs) is that they want to be able to see the routing structure and then map it to their Build Output API (https://vercel.com/docs/build-output-api).

In my opinion, the solution you're proposing (combining both outputs) is also the right one, as it covers those needs as well as other use cases or certain hosting providers. Plus, it's flexible enough for developers to adapt it to their specific requirements.

[
    {
      path: '/api',
      methods: ['GET'],
      options: { strict: false, caseSensitive: false }
    },
    {
      path: '/v1/users',
      methods: ['GET'],
      options: { strict: true, caseSensitive: true }
    }
  ]
``` ref https://github.com/pillarjs/router/pull/174#issuecomment-3129927730

[wesleytodd]

I left a few comments, but I think we should workshop this just a bit. I think ideally we don't need to do this work when we call routes() but instead we can keep internal track of these in such a way that they are much more easily operated on. That would likely only be possible in the next major, but if we can design this api such that it is our "ideal internal book keeping structure" then we can ease that transition in the next breaking change.

So I think my ask is that we add some usage examples in the main text of the PR (or maybe in the readme as docs in this pr) and then we can propose some small changes to the exposed api that way before we invest too much more work in the internals and tests?

EDIT: also, I was thinking that maybe one good pressure test for the api would be if I tried to integrate it into my openapi generator package. If it works well for that, I think it would work well for all the similar use cases. If we can organize this work I am happy to make it a priority.

[ctcpip]

I have thoughts on this PR, but won't have time to review for a bit. If it's urgently needed, I don't want to be the blocker, but otherwise would like time to review. 🙏

[bjohansebas]

I have thoughts on this PR, but won't have time to review for a bit. If it's urgently needed, I don't want to be the blocker, but otherwise would like time to review. 🙏

It's not entirely urgent, so feel free to do the review whenever you can.

[blakeembrey]

This looks like a great starting point, love it! @jonchurch has some really good feedback to think about here too, re what Vercel may need.

Can Vercel handle the regex support?

[blakeembrey]

For .use, could it make sense to allow consumers to iterate recursively themselves? It's something that could be added in a follow up, but the MVP could be simply { path, method, router }. Every field could also be optional, I guess, since path: undefined with .use(fn), method: undefined when all is used, and router: undefined when no nested router.

[bjohansebas]

The path will never be undefined .use always sets the path to '/' when no path is explicitly provided in the arguments.

[blakeembrey]

That's the kind of thing that should be nailed down for the API, but understood. It probably is reasonable to keep it as / and the object was intended to be hypothetical.

[blakeembrey]

E.g. why / vs ""? Does one make it harder for consumers than the other? How do these interact with internal routing behaviors that aren't being exposed in this API? How much can move these expectations/behaviors to be static instead of magic (e.g. removing the trailing / is the one that comes to mind, people need to know how the package works internally).

[IamLizu]

Mostly looks good to me now. However, I wonder if there's possibility of infinite recurrsion at collectRoutes.

Imagine a user acceddientally creates circular route references and then calls to getRoutes would go on forever.

Perhaps there should a protection, maybe check if the current layer is visited in the current branch. Here's a possible sample code,

function collectRoutes (stack, prefix, routeMap, options, visited = new WeakSet()) {
  for (const layer of stack) {
    if (layer.pathPatterns && layer.handle && layer.handle.stack && !layer.route) {
      // Check for circular reference
      if (visited.has(layer.handle)) {
        continue  // Skip circular reference
      }
      
      visited.add(layer.handle)
      collectRoutes(layer.handle.stack, pathPrefix, routeMap, options, visited)
      visited.delete(layer.handle)  // Allow re-visiting in different branches
    }
  }
}

Other than this, I think getRoutes is awesome feature to the router.

Good work @bjohansebas 👏

cc: @blakeembrey @wesleytodd @jonchurch

Update

Just to be safe, @bjohansebas can we add more complex and nested routes in the test often found in real world scenarios to be safe?

[blakeembrey]

Note: options will need to contain end: boolean for .use routes, which are prefix matched vs fully matched.

[nronnei]

@bjohansebas you're my hero; eagerly awaiting this change.

[Megapixel99]

Thank you for your work on this pull request. While I was working on: wesleytodd/express-openapi#77, I had to rely on the changes in this PR to get the full path of the routes.

Would it be possible to expose the middleware stack of each route within getRoutes()? Currently I have to get the middleware stack directly from the router (https://github.com/wesleytodd/express-openapi/pull/77/files#diff-ffcbf454c1de79602c5a1cbbc810c181b8d8552f743952d9049ffc13ca8b1f1eR31-R33) despite using getRoutes() to get the full path of the URL earlier in my code (https://github.com/wesleytodd/express-openapi/pull/77/files#diff-ffcbf454c1de79602c5a1cbbc810c181b8d8552f743952d9049ffc13ca8b1f1eR12)

[bjohansebas]

Yep, that could be added in my opinion. I need to partially modify this PR to adapt it to the idea in #174 (comment), I have it in my priorities, so you’ll need to adapt your PR to that format once I make the change.

[jeffsee55]

👋 we (Vercel) have just added an experimental feature in our Express builder that allows us to extract metadata from the Express app like routes, static assets folders, and view rendering engines so we can build Express apps with first-class features like o11y out of the box.

Here's the initial PR with more details, but at the moment the route metadata is very basic (doesn't capture sub-routers) so I wanted to check in on this PR and ask if there's anything we can do to make route introspection better.

You can try it out by using this repo, there's a Vercel deploy button in the README.

[tknickman]

@bjohansebas - Anything we can help with to get this shipped? We're excited to leverage this on Vercel's side.

[bjohansebas]

Hey, sorry for the delay, i’ve been away from open source for a while, and I’m slowly getting back to catching up on all the pending work.

@tknickman and express community, it would be great to know if you have any thoughts on how this API should be presented. We can’t cover every possible route style that this library allows, but we can try to handle the majority of them.

What I still have pending to do is:

• Retrieve the keys when the route path is a regexp
• Address request feat: add getRoutes method to list registered routes #174 (comment), which I think is a good option

[IamLizu]

Hey @bjohansebas!

I think there's infinite recursion on cyclic routers (index.js lines ~479-548). collectRoutes walks mounted routers without a visited-set, so any accidental cycle (reusing routers across mounts) makes getRoutes() blow the stack. For example,

const Router = require('./')

const app = new Router()
const admin = new Router()
const account = new Router()

admin.get('/dashboard', () => {})
account.get('/:accountId', () => {})

app.use('/admin', admin)
app.use('/accounts', account)

// Later refactor reuses routers and creates a cycle:
admin.use('/accounts', account)
account.use('/admin', admin)

app.getRoutes() // RangeError: Maximum call stack size exceeded

Maybe add a WeakSet (or similar) guard plus a regression test that mounts routers cyclically?

[blakeembrey]

@tknickman Does Vercel need the keys from strings and regexes? What's the MVP needed?

Asking because it feels like the MVP to ship could be an array with no recursion (exposing router, not nested routes), and skip keys for now. With that additional features can be exposed in the array in a follow up PR.

For the original PR use-case of a flattened list that could be easily consumed, that is the type of thing that can be added and exported from the main package once this API is stable, since it's just a more user-friendly (but specific) output.

[jeffsee55]

@blakeembrey we just need the paths themselves, and ideally the methods. Eg. /users/:id and [GET], etc

[bjohansebas]

Asking because it feels like the MVP to ship could be an array with no recursion (exposing router, not nested routes), and skip keys for now. With that additional features can be exposed in the array in a follow up PR.

For the context of the keys #174 (comment)

[blakeembrey]

@bjohansebas I get that, but it's something that can be added in another PR without a breaking change. It seems like there's the bones of an implementation that can be shipped today and iterated on without any breaking changes if you define the basic API clearly.