New Pipeline Architecture

[ArthurCRodrigues]

Main goals:

• Improve readability
• Solve Technical Debts
• Increase Modularity
• Optimize Performance
• Increase error handling
• Implement new features

Main problems to fix:

• Current Architecture is sharing state through static variables (autograder_facade)
• Despite being modular, all classes are coupled because of the autograder_facade
• Adding new behaviors increases complexity
• Responsibilities are scattered across the system (ex: multiple /models)
• Domain organization is confusing
• System is not optimized for batch grading (grading multiple submissions in one call)

Conclusion:

The main goal is to completely remove the autograder_facade from the system, as it causes really high coupling and makes it really hard to include new features in the grading process.

The substitute will be using a pipeline architecture, which is not actually orchestrated as the so called "orchestrator" does not care about what step is running and what should happen if that step fails or succeeds. Instead, the system will take some form of coreography as each step will know exactly what to expect (doesn't matter the previous one) and what to do with whats given and if something goes wrong, they're are the ones to publish that error.

The "orchestrator" simply executes steps and looks if they returned an error, if so, it stops the process. In the old architecture, the orchestrator was responsible for executing each step knowing exactly what step was being executed and it was responsible for the decisions after each step's result.

Increasing Modularity

The pipeline architecture is the best architectural pattern for achieving this goal. It was designed for that. We'll achieve modularity by having the entire grading process as a final pipeline and have each step as its building blocks, or as its filters .

Each filter represent a crucial step in the grading process, but may not be required (example: feedback generation)
The autograder will then take the entire configuration and generate the appropriate pipeline from that. If for example, include_feedback = False then it will not include the FeedbackStep or setup_config is missing then it will not add a PreFlight step. By the end of this assembly, there will be a configured pipeline that can be reutilized for any submission.

Some filters will always be present and need no checks (ex: TemplateImporting, Grading and Exporting)

graph TD
    A[Start] --> B{Setup Config?}
    B -->|Yes| C[PreFlightStep]
    B -->|No| D[TemplateLoaderStep]
    C --> D
    D --> E[GradeStep]
    E --> F{Include Feedback?}
    F -->|Yes| G[FeedbackStep]
    F -->|No| H[ExporterStep]
    G --> H
    H --> I[End]

Loading

Adding support to Batch Grading

The goal is to allow multiple submissions to be sent at once. And also to allow the same configured pipeline to be used for grading a single submission and multiple submissions without extra configuration. The goal is to implement the pipeline so that it behaves differently from the number of submissions given.

There's one important thing to consider:

• It doesn't make sense, for a single submission, to build the CriteriaTree then traverse it in order to generate the ResultTree. Instead, is better to simply build the ResultTree from the configuration

This is one of the major behaviour examples. But it may not be necessary. Maybe, the time difference from building the CriteriaTree then building the ResultTree and simply building the ResultTree from the criteria config may be really small. And adding such change of behaviour may not be worth it.

If this holds true (not worth it to add the skip CriteriaTree behaviour), then the only different behavior the pipeline will have to assume when dealing with multiple submissions will be to execute the steps for each submission (iterate) except for the BuildTree step since it is not related to a submission, but to an assignment.