Non-Deterministic and Priority Sorting of Middleware and Routes

[psenger]

This document outlines a critical feature request to implement deterministic and predictive, convention-based ordering for routes and middleware in express-auto-router. This addresses a fundamental routing conflict issue and provides users with explicit control over route registration order.

Problem Statement

Current Issue: Route Conflicts

The current implementation has a critical flaw where route registration order depends on filesystem directory order (typically alphabetical), leading to non-deterministic behavior and route conflicts.

Example Conflict:

routes/
├── users/
│   ├── [id]/index.js       # Registers: /users/:id/
│   └── all/index.js        # Registers: /users/all/

Problem: If [id] is processed before all (bracket comes before 'a' in ASCII), Express registers /users/:id/ first, causing it to intercept requests to /users/all/ because the dynamic parameter :id matches "all". This is a bad API design, however the application does not provide a mechanism to overcome the desired result.

Current Issue: Middleware Function Order Conflict

The current implementation has no mechanism to control the execution order of middleware functions as they are merged and inherited through the directory tree hierarchy. When middleware from parent directories combines with middleware from child directories, the resulting execution order is non-deterministic.

Example Conflict:

// routes/_middleware.js (parent)
module.exports = (options) => {
  return [fn-1, fn-2]  // Parent middleware
}

// routes/users/_middleware.js (child)
module.exports = (options) => {
  return [fn-3, fn-4]  // Child middleware
}

// routes/users/admin/_middleware.js (grandchild)
module.exports = (options) => {
  return [fn-5]  // Grandchild middleware
}

Problem: When accessing /users/admin/, middleware functions are merged from all levels, but the merging and ordering logic is non-deterministic:

• Current execution: Could be [fn-1, fn-2, fn-3, fn-4, fn-5] OR [fn-3, fn-4, fn-1, fn-2, fn-5] OR other combinations
• Desired execution: Predictable order like [fn-1(priority:5), fn-3(priority:10), fn-2(priority:20), fn-5(priority:30), fn-4(priority:40)]
• Result: CORS, auth, and other middleware execute in wrong order causing failures

Inheritance Tree Problem:

• Functions from / merge with functions from /users/
• Combined result merges with functions from /users/admin/
• No control over how functions are ordered during merging process
• No way to specify that a parent function should execute before/after child functions

Current Limitations

1. Non-deterministic Route Order: Routes register in filesystem order
2. No User Control: Developers cannot explicitly control route priority
3. Silent Failures: Conflicts go undetected until runtime
4. Middleware Ordering: Similar issues exist with middleware execution order
5. Scalability Issues: Complex routing hierarchies become unpredictable

Real-World Impact

• Production Bugs: Routes may work in development but fail in production due to different filesystem ordering
• Maintenance Nightmares: Debugging route conflicts is extremely difficult
• API Reliability: Users can't guarantee their API endpoints will work as expected
• Team Confusion: Different developers may experience different route behavior

[psenger]

Proposed Solution: Optional Numeric Priority Convention

Convention Overview

Implement an optional numeric prefix convention that provides deterministic, user-controlled ordering for both routes and middleware.

🔹 IMPORTANT: All priority features are completely OPTIONAL

• Existing projects work unchanged - no migration required
• Opt-in enhancement - only use priorities where needed
• Backward compatible - non-prefixed directories and plain middleware functions continue to work exactly as before

Format: {priority}-{name} or {priority}-[{name}] (OPTIONAL)

• priority: 2-digit number (00-99)
• name: route/directory name
• [{name}]: path property name
• Lower numbers = higher priority (execute/register first)

Example Implementation

Priority Overrides Alphabetical Order:

routes/
├── charlie/
│   └── index.js                    # Priority 50 (default) → /charlie/
├── 10-alpha/
│   └── index.js                    # Priority 10 → /alpha/
├── beta/
│   └── index.js                    # Priority 50 (default) → /beta/
├── 05-zulu/
│   └── index.js                    # Priority 5 → /zulu/
└── users/
    ├── profile/
    │   └── index.js                # Priority 50, static → /users/profile/
    ├── [id]/
    │   └── index.js                # Priority 50, dynamic → /users/:id/
    ├── 15-[userId]/
    │   └── index.js                # Priority 15, dynamic → /users/:userId/
    ├── 10-all/
    │   └── index.js                # Priority 10 → /users/all/
    └── 90-admin/
        └── index.js                # Priority 90 → /users/admin/

Without priority (alphabetical): alpha, beta, charlie, zulu
With priority (deterministic): zulu (05), alpha (10), beta (50), charlie (50)

Dynamic Route (Named Parameter) Priorities

🔑 KEY FEATURE: Dynamic routes can have priorities just like static routes using placeholder directory prefixes:

Format: {priority}-[paramName]

• Example: 10-[userId] becomes route /users/:userId/ with priority 10
• Example: 05-[id] becomes route /posts/:id/ with priority 5

routes/
├── users/
│   ├── 10-all/index.js          # Priority 10, static → /users/all/
│   ├── 15-[id]/index.js         # Priority 15, dynamic → /users/:id/
│   ├── 20-admin/index.js        # Priority 20, static → /users/admin/
│   ├── 05-[userId]/index.js     # Priority 5, dynamic → /users/:userId/
│   ├── [sessionId]/index.js     # Priority 50, dynamic → /users/:sessionId/
│   └── profile/index.js         # Priority 50, static → /users/profile/

Registration order: /users/:userId/ (05) → /users/all/ (10) → /users/:id/ (15) → /users/admin/ (20) → /users/profile/ (50) → /users/:sessionId/ (50)

Why this matters:

• Solves route conflicts: Specific routes can register before broad dynamic routes
• Flexible positioning: Dynamic routes don't have to come last
• Multiple dynamic routes: Different dynamic routes can have different priorities
• Mixed routing: Static and dynamic routes can be interleaved by priority

Common use cases:

• 01-[id] - High priority dynamic route that should catch specific patterns first
• 10-all/ + 20-[userId]/ - Static "all" route before dynamic user routes
• 90-[catchAll]/ - Low priority catch-all dynamic route

Mixed Priority/Non-Priority Behavior

Three-Level Sorting Logic:

1. Priority Level (00-99, default 50 for non-prefixed)
2. Route Type (static routes before dynamic routes)
3. Alphabetical (for same priority and type)

Example with Mixed Numbering:

routes/
├── users/
│   ├── 10-all/index.js          # Priority 10 → /users/all/
│   ├── profile/index.js         # Priority 50, static → /users/profile/
│   ├── [id]/index.js            # Priority 50, dynamic → /users/:id/
│   ├── admin/index.js           # Priority 50, static → /users/admin/
│   └── 90-settings/index.js     # Priority 90 → /users/settings/

Route Registration Order

With the enhanced sorting, routes register in this deterministic order:

1. /zulu/ (priority 5)
2. /users/:userId/ (priority 5, dynamic from 05-[userId]/)
3. /alpha/ (priority 10)
4. /users/all/ (priority 10)
5. /users/:id/ (priority 15, dynamic from 15-[id]/)
6. /users/admin/ (priority 20)
7. /beta/ (priority 50, alphabetically before charlie)
8. /charlie/ (priority 50)
9. /users/profile/ (priority 50, static)
10. /users/:sessionId/ (priority 50, dynamic - comes after static at same priority)

Middleware Function Priority System (OPTIONAL)

🔹 COMPLETELY OPTIONAL: Middleware functions can be assigned individual priorities using object notation to control cross-hierarchy merging. Existing middleware functions work unchanged.

// routes/_middleware.js (parent)
module.exports = (options) => {
  return [
    { fn: corsMiddleware, priority: 5 },        // Must execute FIRST
    { fn: authMiddleware, priority: 20 },       // Needs CORS headers
    { fn: loggingMiddleware, priority: 90 }     // Should execute LAST
  ]
}

// routes/users/_middleware.js (child)
module.exports = (options) => {
  return [
    { fn: userValidationMiddleware, priority: 15 }, // After CORS, before auth
    { fn: userContextMiddleware, priority: 50 }     // Default priority
  ]
}

// routes/users/admin/_middleware.js (grandchild)
module.exports = (options) => {
  return { fn: adminMiddleware, priority: 30 }  // After auth, before user context
}

Deterministic execution order for /users/admin/:
All middleware functions from the inheritance tree are collected and sorted by priority:

1. CORS (priority 5) - from /
2. User validation (priority 15) - from /users/
3. Auth (priority 20) - from /
4. Admin middleware (priority 30) - from /users/admin/
5. User context (priority 50) - from /users/
6. Logging (priority 90) - from /

Key points:

• Cross-hierarchy control: Functions from any level can specify their execution priority
• Deterministic merging: All functions are sorted by priority regardless of which _middleware.js file they come from
• Inheritance preserved: Child routes still inherit parent middleware, but now with predictable ordering
• Same priority handling: When functions have identical priorities, they are sorted alphabetically by function name (fn.name)
• Anonymous functions: Functions with no name (fn.name === "") are grouped together and sorted by array position
• Arrow function names: Arrow function names are inferred by the variable name they're assigned to
• Backward compatible: Functions without priority objects default to priority 50
• Flexible format: Can return single object, single function, or arrays of mixed types

Four-Level Sorting Logic for Middleware Functions:

1. Priority Level (00-99, default 50)
2. Function name alphabetically (fn.name - arrow functions get inferred names from variables)
3. Anonymous functions last (functions where fn.name === "" sorted by original array position)
4. Source path specificity (longer paths = more specific = later execution as final tie-breaker)

Technical Specification

Core Functions to Modify

1. parseDirectoryPriority(dirName) - New utility function
2. buildRoutes(basePath, baseURL) - Add priority-based sorting
3. buildMiddlewareDictionary() - Add middleware priority sorting
4. URL processing logic - Handle priority prefix removal

Implementation Details

1. Directory Priority Parsing

/**
 * Parses directory name for priority prefix and route type
 * @param {string} dirName - Directory name (e.g., "10-users", "users", "[id]", "05-[userId]")
 * @returns {Object} - { priority: number, name: string, hasPrefix: boolean, isDynamic: boolean }
 *
 * @example
 * parseDirectoryPriority("10-[userId]")
 * // Returns: { priority: 10, name: "[userId]", hasPrefix: true, isDynamic: true }
 *
 * @example
 * parseDirectoryPriority("05-admin")
 * // Returns: { priority: 5, name: "admin", hasPrefix: true, isDynamic: false }
 *
 * @example
 * parseDirectoryPriority("[sessionId]")
 * // Returns: { priority: 50, name: "[sessionId]", hasPrefix: false, isDynamic: true }
 */
function parseDirectoryPriority(dirName) {
  const match = dirName.match(/^(\d{2})-(.+)$/)
  if (match) {
    const name = match[2]  // Could be "users", "[userId]", "[id]", etc.
    return {
      priority: parseInt(match[1], 10),
      name,
      hasPrefix: true,
      isDynamic: isPlaceholder(name)  // Detects if name contains [param]
    }
  }
  return {
    priority: 50, // Default middle priority for non-prefixed directories
    name: dirName,
    hasPrefix: false,
    isDynamic: isPlaceholder(dirName)
  }
}

2. Route Building with Priority

export function buildRoutes(basePath, baseURL) {
  const result = []
  const queue = [[basePath, baseURL + '/']]

  while (queue.length > 0) {
    const [currentPath, currentURL] = queue.shift()
    const files = readdirSync(currentPath)

    // Check for index.js in current directory
    const indexFile = files.find((file) => file === 'index.js')
    if (indexFile) {
      const indexFilePath = path.join(currentPath, indexFile)
      result.push([currentURL, indexFilePath])
    }

    // Process directories with enhanced three-level sorting
    const directories = files
      .filter((file) => statSync(path.join(currentPath, file)).isDirectory())
      .map((dir) => path.join(currentPath, dir))
      .sort((a, b) => {
        const aParsed = parseDirectoryPriority(path.basename(a))
        const bParsed = parseDirectoryPriority(path.basename(b))

        // Primary sort: by priority (00-99)
        if (aParsed.priority !== bParsed.priority) {
          return aParsed.priority - bParsed.priority
        }

        // Secondary sort: static routes before dynamic routes
        if (aParsed.isDynamic !== bParsed.isDynamic) {
          return aParsed.isDynamic ? 1 : -1
        }

        // Tertiary sort: alphabetical by name
        return aParsed.name.localeCompare(bParsed.name)
      })

    directories.forEach((dir) => {
      const dirInfo = parseDirectoryPriority(path.basename(dir))
      const routeName = dirInfo.name // Use name without priority prefix

      let entryURL = `${currentURL}${routeName}/`
      if (isPlaceholder(routeName)) {
        entryURL = `${currentURL}${replaceUrlPlaceholders(routeName)}/`
      }
      queue.push([dir, entryURL])
    })
  }

  return result
}

3. Middleware Function Priority Support

/**
 * Normalizes middleware to priority objects
 * @param {Function|Object|Array} middleware - Middleware function(s) or priority objects
 * @returns {Array} Array of {fn, priority} objects
 */
function normalizeMiddlewarePriority(middleware) {
  const items = Array.isArray(middleware) ? middleware : [middleware]
  return items.map(item => {
    if (typeof item === 'function') {
      return { fn: item, priority: 50 }
    }
    if (item && typeof item.fn === 'function') {
      return { fn: item.fn, priority: item.priority || 50 }
    }
    throw new Error('Invalid middleware: must be function or {fn, priority} object')
  })
}

/**
 * Sorts and flattens middleware functions by four-level priority system
 * @param {Array} middlewareArray - Array of {fn, priority, sourceIndex, sourcePath} objects
 * @returns {Array} Array of middleware functions sorted by priority
 */
function sortMiddlewareFunctions(middlewareArray) {
  return middlewareArray
    .sort((a, b) => {
      // Level 1: Priority (00-99)
      if (a.priority !== b.priority) {
        return a.priority - b.priority
      }
      
      // Level 2: Function name alphabetically (arrow functions get inferred names)
      const aName = a.fn.name || ""
      const bName = b.fn.name || ""
      if (aName !== bName) {
        // Anonymous functions ("") come after named functions
        if (aName === "" && bName !== "") return 1
        if (aName !== "" && bName === "") return -1
        return aName.localeCompare(bName)
      }
      
      // Level 3: For anonymous functions, sort by original array position
      if (aName === "" && bName === "") {
        if (a.sourceIndex !== b.sourceIndex) {
          return a.sourceIndex - b.sourceIndex
        }
      }
      
      // Level 4: Source path specificity (longer paths = more specific)
      return a.sourcePath.length - b.sourcePath.length
    })
    .map(item => item.fn)
}

export function buildMiddlewareDictionary(basePath, baseURL, options) {
  const dictionary = {}

  const traverseDirectory = (currentPath, currentURL) => {
    const dirEntries = readdirSync(currentPath, { withFileTypes: true })

    dirEntries.forEach((entry) => {
      const entryPath = path.join(currentPath, entry.name)

      if (entry.isDirectory()) {
        let entryURL = `${currentURL}/${entry.name}`
        if (isPlaceholder(entry.name)) {
          entryURL = `${currentURL}/${replaceUrlPlaceholders(entry.name)}`
        }
        traverseDirectory(entryPath, entryURL)

      } else if (isMiddlewareFile(entry)) {
        try {
          const middleware = require(entryPath)(options)
          const middlewareURL = entryURL.replace('_middleware.js', '')
          const normalizedMiddleware = normalizeMiddlewarePriority(middleware)
          dictionary[middlewareURL] = normalizedMiddleware
        } catch (e) {
          console.error(e)
          console.error(`Failed to load ${entryPath}`)
        }
      }
    })
  }

  traverseDirectory(basePath, baseURL)
  return dictionary
}

/**
 * Updated function to handle priority-sorted middleware
 */
export function dictionaryKeyStartsWithPath(dictionary, path) {
  const allMiddleware = Object.entries(dictionary)
    .filter(([key]) => path.startsWith(key))
    .sort(([aKey], [bKey]) => aKey.length - bKey.length)
    .flatMap(([, middlewareObjects]) => middlewareObjects)
    .filter(Boolean)

  return sortMiddlewareFunctions(allMiddleware)
}

Backward Compatibility

• Non-prefixed directories continue to work with default priority (50)
• Existing projects require no immediate changes
• Gradual migration possible by adding prefixes incrementally
• Validation warnings can be added to detect potential conflicts

Configuration Options

Consider adding configuration options for:

• Default priority for non-prefixed directories
• Strict mode requiring priority prefixes
• Validation level (warnings vs errors for conflicts)
• Custom priority parsing patterns

Benefits

For Developers

1. Predictable Routing: Routes always register in the same order
2. Explicit Control: Clear, visual priority system
3. Conflict Prevention: Eliminates route shadowing issues
4. Better Debugging: Route order is immediately visible
5. Team Collaboration: Consistent behavior across environments

For the Library

1. Production Ready: Eliminates critical routing bugs
2. Scalability: Supports complex routing hierarchies
3. Maintainability: Clear conventions reduce support burden
4. Future-Proof: Foundation for advanced routing features

[psenger]

Migration Strategy

Phase 1: Core Implementation

• Implement parseDirectoryPriority() function
• Update buildRoutes() with priority sorting
• Update buildMiddlewareDictionary() with priority sorting
• Add comprehensive test coverage

Phase 2: Documentation & Examples

• Update README with priority convention examples
• Add migration guide for existing projects
• Create example directory structures
• Update API documentation

Phase 3: Validation & Tooling

• Add route conflict detection
• Implement validation warnings
• Consider CLI tools for migration assistance
• Add development-time route inspection

Phase 4: Advanced Features (Future)

• Custom priority patterns
• Configuration options
• Integration with development tools
• Performance optimizations

Testing Requirements

Unit Tests

• parseDirectoryPriority() function
• Route sorting logic
• Middleware priority ordering
• Backward compatibility

Integration Tests

• Complex directory structures
• Mixed priority/non-priority directories
• Route conflict scenarios
• Middleware execution order

Real-World Scenarios

• Large application structures
• Migration from non-priority setup
• Performance impact assessment
• Cross-platform filesystem behavior

Success Criteria

1. Route Conflicts Eliminated: No more silent route shadowing
2. Deterministic Behavior: Same route order across all environments
3. User Control: Developers can explicitly set route priority
4. Backward Compatible: Existing projects continue to work
5. Well Documented: Clear examples and migration guides
6. Thoroughly Tested: Comprehensive test coverage

Conclusion

This feature addresses a fundamental flaw in the current routing system while providing a robust, scalable solution for route and middleware ordering. The numeric priority convention is simple, visible, and gives developers the explicit control they need for production applications.

The implementation maintains backward compatibility while providing a clear migration path to deterministic routing behavior.