diff --git a/README.md b/README.md index e7775ee..cccfd11 100644 --- a/README.md +++ b/README.md @@ -291,6 +291,541 @@ GET /problemStatements/{problemStatementId}/tasks/{taskId}/subtasks GET /problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId} ``` +**Get Blueprint for a Subtask** + +```http +GET /problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/blueprint?detailed={boolean} +``` + +This endpoint returns the complete model configuration blueprint for all models in a subtask, showing available parameters and data inputs. + +**Query Parameters:** + +- `detailed` (optional, boolean, default: false): Controls the level of parameter detail returned + - `false` or omitted: Returns basic parameter info (id, value) only + - `true`: Returns full ModelParameter details including type, description, min, max, etc. + +## Programmatic Workflow Guide + +This section describes how to programmatically use the Ensemble Manager to select models, configure parameters, and bind data for scientific modeling workflows. + +### Overview + +The typical workflow follows these steps: + +1. Create Problem Statement +2. Create Task +3. Create Subtask +4. Select Model Configurations +5. Get Blueprint (to see available parameters and inputs) +6. Configure Parameters +7. Bind Data +8. Submit for Execution + +### Step-by-Step Workflow + +#### 1. Create Problem Statement + +First, establish the research context: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Ethiopia Agricultural Productivity Analysis 2024", + "regionid": "ethiopia", + "dates": { + "start_date": "2000-01-01T00:00:00Z", + "end_date": "2017-12-31T23:59:59Z" + } + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "name": "Ethiopia Agricultural Productivity Analysis 2024", + "regionid": "ethiopia", + "dates": { + "start_date": "2000-01-01T00:00:00Z", + "end_date": "2017-12-31T23:59:59Z" + } +} +``` + +#### 2. Create Task + +Create a task within the problem statement: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Crop Yield Analysis", + "dates": { + "start_date": "2000-01-01T00:00:00Z", + "end_date": "2017-12-31T23:59:59Z" + }, + "regionid": "ethiopia" + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "name": "Crop Yield Analysis", + "dates": { + "start_date": "2000-01-01T00:00:00Z", + "end_date": "2017-12-31T23:59:59Z" + }, + "regionid": "ethiopia" +} +``` + +#### 3. Create Subtask + +Create a subtask to contain your model configuration: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Cycles Agricultural Analysis", + "dates": { + "start_date": "2000-01-01T00:00:00Z", + "end_date": "2017-12-31T23:59:59Z" + } + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "name": "Cycles Agricultural Analysis", + "dates": { + "start_date": "2000-01-01T00:00:00Z", + "end_date": "2017-12-31T23:59:59Z" + } +} +``` + +#### 4. Select Model Configurations + +Add ModelConfiguration or ModelConfigurationSetup instances to your subtask: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/models" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "modelIds": [ + "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu" + ] + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "modelIds": [ + "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu" + ] +} +``` + +#### 5. Get Blueprint + +Retrieve the complete configuration blueprint to understand available parameters and data inputs: + +```bash +curl -X GET "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/blueprint" \ + -H "Authorization: Bearer $JWT_TOKEN" +``` + +**For detailed parameter information (including type, description, min/max values):** + +```bash +curl -X GET "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/blueprint?detailed=true" \ + -H "Authorization: Bearer $JWT_TOKEN" +``` + +**Example Blueprint Response:** + +```json +[ + { + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu", + "parameters": [ + { + "id": "https://w3id.org/okn/i/mint/886ebf8c-6f0b-453d-a36c-fc8678c74109", + "value": "2000" + }, + { + "id": "https://w3id.org/okn/i/mint/a7607d91-a832-4f05-85f0-4b9e481ac8e1", + "value": "2017" + }, + { + "id": "https://w3id.org/okn/i/mint/a46a3d56-207e-4f47-a157-00b299b3536b", + "value": "Maize" + }, + { + "id": "https://w3id.org/okn/i/mint/d4b84b70-01ee-4f14-a1fc-357f45af5c1d", + "value": "100" + }, + { + "id": "https://w3id.org/okn/i/mint/6dff2c27-b5b6-4e07-836e-c0075d41d333", + "value": "149" + }, + { + "id": "https://w3id.org/okn/i/mint/e2cd6662-06f2-4d51-a2ab-111e9b84f7df", + "value": "0" + }, + { + "id": "https://w3id.org/okn/i/mint/02cbd74e-40d4-49b9-9ea2-033dd0f461e0", + "value": "0.05" + }, + { + "id": "https://w3id.org/okn/i/mint/768babb7-2685-4a16-b1ee-23623b225c47", + "value": "FALSE" + } + ], + "inputs": [ + { + "id": "https://w3id.org/okn/i/mint/13f1ba62-7b1e-45df-bb5c-4cbffc62872a", + "dataset": { + "id": "", + "resources": [] + } + }, + { + "id": "https://w3id.org/okn/i/mint/493f44ac-8d70-4c41-bfbc-4b6207d72674", + "dataset": { + "id": "", + "resources": [] + } + } + ] + } +] +``` + +#### 6. Configure Parameters + +Use the blueprint information to set parameter values: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/parameters" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu", + "parameters": [ + { + "id": "https://w3id.org/okn/i/mint/768babb7-2685-4a16-b1ee-23623b225c47", + "value": "FALSE" + }, + { + "id": "https://w3id.org/okn/i/mint/e2cd6662-06f2-4d51-a2ab-111e9b84f7df", + "value": "0" + }, + { + "id": "https://w3id.org/okn/i/mint/02cbd74e-40d4-49b9-9ea2-033dd0f461e0", + "value": "0.05" + }, + { + "id": "https://w3id.org/okn/i/mint/a46a3d56-207e-4f47-a157-00b299b3536b", + "value": "Maize" + }, + { + "id": "https://w3id.org/okn/i/mint/a7607d91-a832-4f05-85f0-4b9e481ac8e1", + "value": "2017" + }, + { + "id": "https://w3id.org/okn/i/mint/886ebf8c-6f0b-453d-a36c-fc8678c74109", + "value": "2000" + }, + { + "id": "https://w3id.org/okn/i/mint/6dff2c27-b5b6-4e07-836e-c0075d41d333", + "value": "149" + }, + { + "id": "https://w3id.org/okn/i/mint/d4b84b70-01ee-4f14-a1fc-357f45af5c1d", + "value": "100" + } + ] + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu", + "parameters": [ + { + "id": "https://w3id.org/okn/i/mint/768babb7-2685-4a16-b1ee-23623b225c47", + "value": "FALSE" + }, + { + "id": "https://w3id.org/okn/i/mint/e2cd6662-06f2-4d51-a2ab-111e9b84f7df", + "value": "0" + }, + { + "id": "https://w3id.org/okn/i/mint/02cbd74e-40d4-49b9-9ea2-033dd0f461e0", + "value": "0.05" + }, + { + "id": "https://w3id.org/okn/i/mint/a46a3d56-207e-4f47-a157-00b299b3536b", + "value": "Maize" + }, + { + "id": "https://w3id.org/okn/i/mint/a7607d91-a832-4f05-85f0-4b9e481ac8e1", + "value": "2017" + }, + { + "id": "https://w3id.org/okn/i/mint/886ebf8c-6f0b-453d-a36c-fc8678c74109", + "value": "2000" + }, + { + "id": "https://w3id.org/okn/i/mint/6dff2c27-b5b6-4e07-836e-c0075d41d333", + "value": "149" + }, + { + "id": "https://w3id.org/okn/i/mint/d4b84b70-01ee-4f14-a1fc-357f45af5c1d", + "value": "100" + } + ] +} +``` + +#### 7. Bind Data + +Select and bind datasets to model inputs: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/data" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu", + "data": [ + { + "id": "https://w3id.org/okn/i/mint/13f1ba62-7b1e-45df-bb5c-4cbffc62872a", + "dataset": { + "id": "oromia-weather-soil-2000-2017", + "resources": [ + { + "id": "weather-soil-resource-id", + "url": "https://data.mint.isi.edu/files/cycles-input-data/oromia/weather-soil/Arsi_Amigna_7.884865046N_40.19527054E.zip" + } + ] + } + }, + { + "id": "https://w3id.org/okn/i/mint/493f44ac-8d70-4c41-bfbc-4b6207d72674", + "dataset": { + "id": "cycles-crops-configuration", + "resources": [ + { + "id": "crops-config-resource-id", + "url": "https://raw.githubusercontent.com/mintproject/MINT-WorkflowDomain/master/WINGSWorkflowComponents/cycles-0.10.2-beta-collection/data/crops-horn-of-africa.crop" + } + ] + } + } + ] + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu", + "data": [ + { + "id": "https://w3id.org/okn/i/mint/13f1ba62-7b1e-45df-bb5c-4cbffc62872a", + "dataset": { + "id": "oromia-weather-soil-2000-2017", + "resources": [ + { + "id": "weather-soil-resource-id", + "url": "https://data.mint.isi.edu/files/cycles-input-data/oromia/weather-soil/Arsi_Amigna_7.884865046N_40.19527054E.zip" + } + ] + } + }, + { + "id": "https://w3id.org/okn/i/mint/493f44ac-8d70-4c41-bfbc-4b6207d72674", + "dataset": { + "id": "cycles-crops-configuration", + "resources": [ + { + "id": "crops-config-resource-id", + "url": "https://raw.githubusercontent.com/mintproject/MINT-WorkflowDomain/master/WINGSWorkflowComponents/cycles-0.10.2-beta-collection/data/crops-horn-of-africa.crop" + } + ] + } + } + ] +} +``` + +#### 8. Verify Configuration + +Get the updated blueprint to verify your configuration: + +```bash +curl -X GET "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/blueprint" \ + -H "Authorization: Bearer $JWT_TOKEN" +``` + +**For detailed verification with full parameter information:** + +```bash +curl -X GET "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/blueprint?detailed=true" \ + -H "Authorization: Bearer $JWT_TOKEN" +``` + +**Copy-paste ready:** + +``` +// 8. Verify Configuration - No request body needed (GET request) +// Add ?detailed=true for full parameter details +``` + +#### 9. Submit for Execution + +Submit the configured subtask for execution: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/submit" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu" + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu" +} +``` + +### Alternative: One-Step Setup + +For convenience, you can configure models, parameters, and data in a single call: + +```bash +curl -X POST "https://ensemble-manager.mint.tacc.utexas.edu/v1/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/setup" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu", + "parameters": [ + { + "id": "https://w3id.org/okn/i/mint/a46a3d56-207e-4f47-a157-00b299b3536b", + "value": "Teff" + }, + { + "id": "https://w3id.org/okn/i/mint/e2cd6662-06f2-4d51-a2ab-111e9b84f7df", + "value": "150" + } + ], + "data": [ + { + "id": "https://w3id.org/okn/i/mint/13f1ba62-7b1e-45df-bb5c-4cbffc62872a", + "dataset": { + "id": "oromia-weather-soil-2000-2017", + "resources": [ + { + "id": "weather-soil-resource-id", + "url": "https://data.mint.isi.edu/files/cycles-weather-soil/oromia_2000_2017.tar.gz" + } + ] + } + } + ] + }' +``` + +**Copy-paste ready request body:** + +```json +{ + "model_id": "http://api.models.mint.local/v1.8.0/modelconfigurations/f87802e0-b60f-4c9e-97fd-75fad348b7ee?username=mint@isi.edu", + "parameters": [ + { + "id": "https://w3id.org/okn/i/mint/a46a3d56-207e-4f47-a157-00b299b3536b", + "value": "Teff" + }, + { + "id": "https://w3id.org/okn/i/mint/e2cd6662-06f2-4d51-a2ab-111e9b84f7df", + "value": "150" + } + ], + "data": [ + { + "id": "https://w3id.org/okn/i/mint/13f1ba62-7b1e-45df-bb5c-4cbffc62872a", + "dataset": { + "id": "oromia-weather-soil-2000-2017", + "resources": [ + { + "id": "weather-soil-resource-id", + "url": "https://data.mint.isi.edu/files/cycles-weather-soil/oromia_2000_2017.tar.gz" + } + ] + } + } + ] +} +``` + +### Key Concepts for Programmatic Use + +#### Model Selection + +- **ModelConfiguration**: Specific model instances with pre-defined parameters +- **ModelConfigurationSetup**: Model templates that allow parameter customization +- Use the `/models` endpoint to add these to your subtask + +#### Blueprint-Driven Configuration + +- Always call the `/blueprint` endpoint after adding models +- The blueprint shows you exactly what parameters and inputs are available +- Use blueprint information to guide your parameter and data configuration +- Use `?detailed=true` to get comprehensive parameter metadata (type, description, min/max values, etc.) +- Default blueprint returns basic parameter info (id, value) for faster responses + +#### Parameter Values + +- Parameters can have single values: `"150"` +- Parameters can have multiple values for ensemble runs: `["100", "150", "200"]` +- The system will create execution combinations based on parameter arrays + +#### Data Binding + +- Each data input requires a dataset with resources +- Resources specify the actual data files to use +- Dataset IDs typically come from CKAN or other data catalogs + +#### Error Handling + +- Always check HTTP status codes +- 400 errors typically indicate missing required parameters +- 404 errors indicate resources not found +- Use the blueprint endpoint to verify available options + #### Execution Management **Submit Modeling Thread for Execution** diff --git a/src/api/api-doc.ts b/src/api/api-doc.ts index 964e1a3..94f0dbb 100644 --- a/src/api/api-doc.ts +++ b/src/api/api-doc.ts @@ -285,6 +285,85 @@ const ModelCatalogSchema = { }, title: "Parameter", type: "object" + }, + ModelParameter: { + description: "A model parameter with comprehensive metadata", + type: "object", + properties: { + id: { + type: "string", + description: "Unique identifier for the parameter" + }, + name: { + type: "string", + description: "Name of the parameter" + }, + type: { + type: "string", + description: "Data type of the parameter" + }, + description: { + type: "string", + nullable: true, + description: "Description of what this parameter controls" + }, + min: { + type: "string", + nullable: true, + description: "Minimum acceptable value" + }, + max: { + type: "string", + nullable: true, + description: "Maximum acceptable value" + }, + unit: { + type: "string", + nullable: true, + description: "Unit of measurement for the parameter" + }, + default: { + type: "string", + nullable: true, + description: "Default value for the parameter" + }, + value: { + type: "string", + nullable: true, + description: "Current or assigned value" + }, + adjustment_variable: { + type: "array", + nullable: true, + description: "Variable that this parameter adjusts" + }, + accepted_values: { + type: "string", + nullable: true, + description: "Accepted values for the parameter (comma-separated string)" + }, + position: { + type: "number", + nullable: true, + description: "Position of the parameter in the model configuration" + } + }, + required: ["id", "type"] + }, + BasicModelParameter: { + description: "Basic model parameter information with just id and value", + type: "object", + properties: { + id: { + type: "string", + description: "Unique identifier for the parameter" + }, + value: { + type: "string", + description: "Current or assigned value" + } + }, + required: ["id", "value"] } }; const MintSchema = { @@ -941,7 +1020,7 @@ const SubtaskSchema = { CreateSubtaskRequest: { type: "object", description: "A Subtask creation request definition", - required: ["name", "dates"], + required: ["name", "dates", "driving_variables", "response_variables", "regionid"], properties: { name: { description: "The name of the subtask", diff --git a/src/api/api-v1/paths/problemStatements/tasks/subtasks/index.ts b/src/api/api-v1/paths/problemStatements/tasks/subtasks/index.ts index 812a11c..dc1dc81 100644 --- a/src/api/api-v1/paths/problemStatements/tasks/subtasks/index.ts +++ b/src/api/api-v1/paths/problemStatements/tasks/subtasks/index.ts @@ -266,8 +266,8 @@ const subtasksRouter = (): Router => { * @openapi * /problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/models: * post: - * summary: Add models to a subtask - * description: Adds models to a subtask + * summary: Select model configurations for a subtask + * description: Select and add ModelConfiguration or ModelConfigurationSetup instances to use in this subtask for execution * security: * - BearerAuth: [] * oauth2: [] @@ -300,7 +300,7 @@ const subtasksRouter = (): Router => { * $ref: '#/components/schemas/AddModelsRequest' * responses: * 200: - * description: Models added successfully + * description: Model configurations selected and added to subtask successfully * content: * application/json: * schema: @@ -348,8 +348,8 @@ const subtasksRouter = (): Router => { * @openapi * /problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/data: * post: - * summary: Add data to a subtask - * description: Adds data to a subtask + * summary: Select data for a subtask + * description: Select the data to use in the subtask per ModelConfiguration/ModelConfigurationSetup. You can obtain a blueprint from the blueprint endpoint to see available data inputs * security: * - BearerAuth: [] * oauth2: [] @@ -382,7 +382,7 @@ const subtasksRouter = (): Router => { * $ref: '#/components/schemas/AddDataRequest' * responses: * 200: - * description: Data added successfully + * description: Data selected and configured for subtask successfully * content: * application/json: * schema: @@ -753,6 +753,133 @@ const subtasksRouter = (): Router => { } } ); + + /** + * @openapi + * /problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/blueprint: + * get: + * summary: Get blueprint (parameters and inputs) for a subtask + * description: Returns the complete model configuration blueprint for all models in a subtask + * security: + * - BearerAuth: [] + * oauth2: [] + * tags: + * - Subtasks + * parameters: + * - in: path + * name: problemStatementId + * required: true + * schema: + * type: string + * description: The problem statement ID + * - in: path + * name: taskId + * required: true + * schema: + * type: string + * description: The task ID + * - in: path + * name: subtaskId + * required: true + * schema: + * type: string + * description: The subtask ID + * - in: query + * name: detailed + * required: false + * schema: + * type: boolean + * default: false + * description: If true, returns full ModelParameter details. If false, returns basic info (id, value) only. + * responses: + * 200: + * description: Complete blueprint for the subtask + * content: + * application/json: + * schema: + * type: array + * items: + * type: object + * properties: + * model_id: + * type: string + * description: The model identifier + * example: "https://w3id.org/okn/i/mint/c07a6f98-6339-4033-84b0-6cd7daca6284" + * parameters: + * type: array + * description: Returns BasicModelParameter (id, value) by default, or full ModelParameter when detailed=true + * items: + * anyOf: + * - $ref: '#/components/schemas/BasicModelParameter' + * - $ref: '#/components/schemas/ModelParameter' + * inputs: + * type: array + * items: + * type: object + * properties: + * id: + * type: string + * description: Input identifier + * example: "https://w3id.org/okn/i/mint/ce32097e-641d-42af-b3f1-477a24cf015a" + * dataset: + * type: object + * properties: + * id: + * type: string + * description: Dataset identifier + * example: "18400624-423c-42b5-ad56-6c73322584bd" + * resources: + * type: array + * items: + * type: object + * properties: + * id: + * type: string + * description: Resource identifier + * example: "9c7b25c4-8cea-4965-a07a-d9b3867f18a9" + * url: + * type: string + * description: Resource URL + * example: "https://ckan.tacc.utexas.edu/dataset/18400624-423c-42b5-ad56-6c73322584bd/resource/9c7b25c4-8cea-4965-a07a-d9b3867f18a9/download/barton_springs_2001_2010average.wel" + * default: + * description: Default error response + * content: + * application/json: + * schema: + * type: object + * properties: + * message: + * type: string + */ + router.get( + "/:subtaskId/blueprint", + async ( + req: Request<{ subtaskId: string }, unknown, unknown, { detailed?: string | boolean }>, + res: Response + ) => { + const authorizationHeader = req.headers.authorization; + if (!authorizationHeader) { + return res.status(401).json({ message: "Authorization header is required" }); + } + const { subtaskId } = req.params; + const { detailed } = req.query; + const isDetailed = detailed === true || detailed === "true"; + try { + const blueprint = await subTasksService.getBlueprint( + subtaskId, + authorizationHeader, + isDetailed + ); + res.status(200).json(blueprint); + } catch (error) { + if (error instanceof HttpError) { + return res.status(error.statusCode).json({ message: error.message }); + } + res.status(500).json({ message: error.message }); + } + } + ); + router.use("/:subtaskId/executions", executionsRouter()); /** diff --git a/src/api/api-v1/services/subTasksService.ts b/src/api/api-v1/services/subTasksService.ts index 9367d83..363f0e7 100644 --- a/src/api/api-v1/services/subTasksService.ts +++ b/src/api/api-v1/services/subTasksService.ts @@ -1,4 +1,4 @@ -import { Thread, ThreadInfo, Execution } from "@/classes/mint/mint-types"; +import { Thread, ThreadInfo, Execution, ModelParameter } from "@/classes/mint/mint-types"; import { addThread, insertModel, @@ -19,6 +19,7 @@ import { import problemStatementsService from "./problemStatementsService"; import { convertApiUrlToW3Id, + convertModelConfigurationW3IdToApiUrl, fetchModelConfiguration, fetchModelConfigurationSetup } from "@/classes/mint/model-catalog-functions"; @@ -39,7 +40,11 @@ import { getConfiguration } from "@/classes/mint/mint-functions"; import { MockExecutionService } from "@/classes/common/__tests__/mocks/MockExecutionService"; import { ExecutionCreation } from "@/classes/common/ExecutionCreation"; import { DatasetSpecification } from "@mintproject/modelcatalog_client/dist"; -import { getThread as getThreadV2 } from "@/classes/graphql/graphql_functions_v2"; +import { + getThread as getThreadV2, + checkVariableExistsById as checkVariableExistsByIdV2, + checkRegionExistsById as checkRegionExistsByIdV2 +} from "@/classes/graphql/graphql_functions_v2"; function getExecutionEngineService( executionEngine: string, @@ -106,6 +111,25 @@ export interface SubTasksService { model_id: string, authorizationHeader: string ): Promise; + getBlueprint( + subtaskId: string, + authorizationHeader: string, + detailed?: boolean + ): Promise< + Array<{ + model_id: string; + parameters: Array<{ id: string; value: string } | ModelParameter>; + inputs: Array<{ + id: string; + dataset: { + id: string; + resources: Array<{ id: string; url: string }>; + }; + }>; + }> + >; + checkVariableExistsByName(variableName: string, authorizationHeader: string): Promise; + checkRegionExistsById(regionId: string, authorizationHeader: string): Promise; } const subTasksService: SubTasksService = { @@ -189,6 +213,34 @@ const subTasksService: SubTasksService = { throw new NotFoundError("Task not found in the specified problem statement"); } + // Validate driving variables exist + if (subtask.driving_variables && subtask.driving_variables.length > 0) { + for (const variableName of subtask.driving_variables) { + const exists = await checkVariableExistsByIdV2(variableName, access_token); + if (!exists) { + throw new BadRequestError(`Driving variable '${variableName}' does not exist`); + } + } + } + + // Validate response variables exist + if (subtask.response_variables && subtask.response_variables.length > 0) { + for (const variableName of subtask.response_variables) { + const exists = await checkVariableExistsByIdV2(variableName, access_token); + if (!exists) { + throw new BadRequestError(`Response variable '${variableName}' does not exist`); + } + } + } + + // Validate region exists + if (subtask.regionid) { + const regionExists = await checkRegionExistsByIdV2(subtask.regionid, access_token); + if (!regionExists) { + throw new BadRequestError(`Region '${subtask.regionid}' does not exist`); + } + } + subtask.events = [ { event: "CREATE", @@ -362,6 +414,99 @@ const subTasksService: SubTasksService = { return await useModelsService.getModelParametersByModelId(model_id); }, + async getBlueprint(subtaskId: string, authorizationHeader: string, detailed: boolean = false) { + const access_token = getTokenFromAuthorizationHeader(authorizationHeader); + if (!access_token) { + throw new UnauthorizedError("Invalid authorization header"); + } + + const subtask = await getThread(subtaskId); + if (!subtask) { + throw new NotFoundError("Subtask not found"); + } + + const bindings: Array<{ + model_id: string; + parameters: Array<{ id: string; value: string } | ModelParameter>; + inputs: Array<{ + id: string; + dataset: { + id: string; + resources: Array<{ id: string; url: string }>; + }; + }>; + }> = []; + + if (subtask.model_ensembles) { + for (const modelId of Object.keys(subtask.model_ensembles)) { + const model = await getModel(modelId); + if (!model) { + throw new NotFoundError("Model not found"); + } + + try { + const formattedParameters: Array< + { id: string; value: string } | ModelParameter + > = []; + + if (model.input_parameters && Array.isArray(model.input_parameters)) { + for (const param of model.input_parameters) { + if (param && param.id) { + if (detailed) { + formattedParameters.push(param); + } else { + const basicParam = { + id: param.id, + value: param.value || param.default || "" + }; + formattedParameters.push(basicParam); + } + } + } + } + + const formattedInputs: Array<{ + id: string; + dataset: { + id: string; + resources: Array<{ id: string; url: string }>; + }; + }> = []; + + if (model.input_files && Array.isArray(model.input_files)) { + for (const input of model.input_files) { + if (input && input.id) { + formattedInputs.push({ + id: input.id, + dataset: { + id: "YOUR_DATASET_ID", + resources: [] + } + }); + } + } + } + + const binding = { + model_id: await convertModelConfigurationW3IdToApiUrl(modelId), + parameters: formattedParameters, + inputs: formattedInputs + }; + bindings.push(binding); + } catch (error) { + console.warn(`Failed to get bindings for model ${modelId}:`, error); + bindings.push({ + model_id: modelId, + parameters: [], + inputs: [] + }); + } + } + } + + return bindings; + }, + async submitSubtask(subtaskId: string, model_id: string, authorizationHeader: string) { const w3id = convertApiUrlToW3Id(model_id); const access_token = getTokenFromAuthorizationHeader(authorizationHeader); @@ -387,7 +532,7 @@ const subTasksService: SubTasksService = { // Collect all executions let submissionResult: SubmissionResult = { submittedExecutions: [], failedExecutions: [] }; - + if (executionCreation.executionToBeRun.length > 0) { submissionResult = await executionService.submitExecutions( executionCreation.executionToBeRun, @@ -398,9 +543,15 @@ const subTasksService: SubTasksService = { subtask.model_ensembles[w3id].id ); if (submissionResult.failedExecutions.length > 0) { - console.warn("Some executions failed to submit:", submissionResult.failedExecutions); + console.warn( + "Some executions failed to submit:", + submissionResult.failedExecutions + ); } - console.log("Successfully submitted executions:", submissionResult.submittedExecutions.length); + console.log( + "Successfully submitted executions:", + submissionResult.submittedExecutions.length + ); } else { console.log("No executions to run"); } @@ -411,6 +562,27 @@ const subTasksService: SubTasksService = { executions: executionCreation.executionToBeRun, submissionResult }; + }, + + async checkVariableExistsByName( + variableName: string, + authorizationHeader: string + ): Promise { + const access_token = getTokenFromAuthorizationHeader(authorizationHeader); + if (!access_token) { + throw new UnauthorizedError("Invalid authorization header"); + } + + return await checkVariableExistsByIdV2(variableName, access_token); + }, + + async checkRegionExistsById(regionId: string, authorizationHeader: string): Promise { + const access_token = getTokenFromAuthorizationHeader(authorizationHeader); + if (!access_token) { + throw new UnauthorizedError("Invalid authorization header"); + } + + return await checkRegionExistsByIdV2(regionId, access_token); } }; diff --git a/src/classes/graphql/graphql_functions_v2.ts b/src/classes/graphql/graphql_functions_v2.ts index 32c557d..de52ffc 100644 --- a/src/classes/graphql/graphql_functions_v2.ts +++ b/src/classes/graphql/graphql_functions_v2.ts @@ -7,6 +7,8 @@ import { Problem_Statement, Task, Thread } from "./types"; import { KeycloakAdapter } from "@/config/keycloak-adapter"; import listTasksByProblemStatementGQL from "./queries/task/listTasksByProblemStatement.graphql"; import getThreadGQL from "./queries/thread/get.graphql"; +import checkVariableByIdGQL from "./queries/variable/check-by-id.graphql"; +import checkRegionByIdGQL from "./queries/region/check-by-id.graphql"; export const getProblemStatements = async (access_token: string): Promise => { const APOLLO_CLIENT = GraphQL.instanceUsingAccessToken(access_token); @@ -85,3 +87,45 @@ export const getThread = async (thread_id: string, access_token?: string): Promi } return result.data.thread_by_pk; }; + +export const checkVariableExistsById = async ( + variableId: string, + access_token: string +): Promise => { + const APOLLO_CLIENT = GraphQL.instanceUsingAccessToken(access_token); + + const result: ApolloQueryResult<{ variable: { id: string; name: string }[] }> = + await APOLLO_CLIENT.query({ + query: checkVariableByIdGQL, + variables: { + id: variableId + } + }); + + if (!result || (result.errors && result.errors.length > 0)) { + throw new InternalServerError("Error checking variable " + result.errors[0].message); + } + + return result.data.variable.length > 0; +}; + +export const checkRegionExistsById = async ( + regionId: string, + access_token: string +): Promise => { + const APOLLO_CLIENT = GraphQL.instanceUsingAccessToken(access_token); + + const result: ApolloQueryResult<{ region_by_pk: { id: string; name: string } | null }> = + await APOLLO_CLIENT.query({ + query: checkRegionByIdGQL, + variables: { + id: regionId + } + }); + + if (!result || (result.errors && result.errors.length > 0)) { + throw new InternalServerError("Error checking region " + result.errors[0].message); + } + + return result.data.region_by_pk !== null; +}; diff --git a/src/classes/graphql/queries/region/check-by-id.graphql b/src/classes/graphql/queries/region/check-by-id.graphql new file mode 100644 index 0000000..bc8a507 --- /dev/null +++ b/src/classes/graphql/queries/region/check-by-id.graphql @@ -0,0 +1,6 @@ +query CheckRegionById($id: String!) { + region_by_pk(id: $id) { + id + name + } +} \ No newline at end of file diff --git a/src/classes/graphql/queries/variable/check-by-id.graphql b/src/classes/graphql/queries/variable/check-by-id.graphql new file mode 100644 index 0000000..ff209b6 --- /dev/null +++ b/src/classes/graphql/queries/variable/check-by-id.graphql @@ -0,0 +1,5 @@ +query CheckVariableById($id: String!) { + variable(where: { id: { _eq: $id } }) { + id + } +} diff --git a/src/classes/mint/model-catalog-functions.ts b/src/classes/mint/model-catalog-functions.ts index 6081800..f828f2a 100644 --- a/src/classes/mint/model-catalog-functions.ts +++ b/src/classes/mint/model-catalog-functions.ts @@ -6,6 +6,7 @@ import { DatasetSpecification } from "@mintproject/modelcatalog_client"; import { KeycloakAdapter } from "@/config/keycloak-adapter"; +import { getConfiguration } from "./mint-functions"; const W3_ID_URI_PREFIX = "https://w3id.org/okn/i/mint/"; @@ -143,3 +144,51 @@ export const convertApiUrlToW3Id = (url: string) => { const id = urlParts.pop(); return W3_ID_URI_PREFIX + id; }; + +export enum ModelCatalogType { + ModelConfiguration = "modelconfiguration", + ModelConfigurationSetup = "modelconfigurationsetup" +} + +//TODO: This is a temporary function to convert a W3 ID to an API URL. +export const convertModelConfigurationW3IdToApiUrl = async (w3Id: string) => { + const config = getConfiguration(); + const modelCatalogApi = config.model_catalog_api; + const modelConfigurationUrl = + modelCatalogApi + + "/" + + ModelCatalogType.ModelConfiguration + + "s/" + + w3Id.replace(W3_ID_URI_PREFIX, "") + + "?username=" + + "mint@isi.edu"; + const modelConfigurationSetupUrl = + modelCatalogApi + + "/" + + ModelCatalogType.ModelConfigurationSetup + + "s/" + + w3Id.replace(W3_ID_URI_PREFIX, "") + + "?username=" + + "mint@isi.edu"; + try { + const modelConfiguration = await await rp.get({ + url: modelConfigurationUrl, + json: true + }); + if (modelConfiguration) { + return modelConfigurationUrl; + } + } catch (error) { + try { + const modelConfigurationSetup = await await rp.get({ + url: modelConfigurationSetupUrl, + json: true + }); + if (modelConfigurationSetup) { + return modelConfigurationSetupUrl; + } + } catch (error) { + throw new Error("Model not found"); + } + } +}; diff --git a/src/classes/tapis/adapters/TapisExecutionService.ts b/src/classes/tapis/adapters/TapisExecutionService.ts index 743be4d..b7d8e45 100644 --- a/src/classes/tapis/adapters/TapisExecutionService.ts +++ b/src/classes/tapis/adapters/TapisExecutionService.ts @@ -9,6 +9,13 @@ import { TapisJobService } from "@/classes/tapis/adapters/TapisJobService"; import errorDecoder from "@/classes/tapis/utils/errorDecoder"; import { TapisJobSubscriptionService } from "@/classes/tapis/adapters/TapisJobSubscriptionService"; import { BadRequestError, NotFoundError } from "@/classes/common/errors"; + +interface SerializableError { + message: string; + stack?: string; + name: string; + cause?: unknown; +} import { getExecution, getModelOutputsByModelId, @@ -99,9 +106,12 @@ export class TapisExecutionService implements IExecutionService { model: Model, threadId: string, threadModelId: string - ): Promise<{ submittedExecutions: { execution: Execution; jobId: string }[]; failedExecutions: { execution: Execution; error: Error }[] }> { + ): Promise<{ + submittedExecutions: { execution: Execution; jobId: string }[]; + failedExecutions: { execution: Execution; error: SerializableError }[]; + }> { const submittedExecutions: { execution: Execution; jobId: string }[] = []; - const failedExecutions: { execution: Execution; error: Error }[] = []; + const failedExecutions: { execution: Execution; error: SerializableError }[] = []; for (const seed of this.seeds) { console.log("Processing seed", JSON.stringify(seed)); @@ -109,8 +119,18 @@ export class TapisExecutionService implements IExecutionService { const jobId = await this.submitSingleExecution(app, seed, model, threadId); submittedExecutions.push({ execution: seed.execution, jobId }); } catch (error) { + console.error("Error submitting single execution", JSON.stringify(error)); await this.handleSingleExecutionFailure(seed, error, threadModelId); - failedExecutions.push({ execution: seed.execution, error }); + + // Create a serializable error object + const serializableError: SerializableError = { + message: error instanceof Error ? error.message : String(error), + stack: error instanceof Error ? error.stack : undefined, + name: error instanceof Error ? error.name : 'Error', + ...(error instanceof Error && (error as any).cause && { cause: (error as any).cause }) + }; + + failedExecutions.push({ execution: seed.execution, error: serializableError }); } } @@ -126,15 +146,15 @@ export class TapisExecutionService implements IExecutionService { const name = this.generateValidJobName(app, seed.execution.id); const description = "Job for " + model.name + " execution " + seed.execution.id; const jobRequest = this.jobService.createJobRequest(app, seed, model, name, description); - + console.log("Job request", JSON.stringify(jobRequest)); const jobId = await this.submitJob(jobRequest); - + await updateExecutionRunId(seed.execution.id, jobId); - + const subscription = TapisJobSubscriptionService.createRequest(seed.execution.id, threadId); await this.jobSubscriptionService.submit(jobId, subscription); - + return jobId; } @@ -143,8 +163,14 @@ export class TapisExecutionService implements IExecutionService { error: Error, threadModelId: string ): Promise { - console.error(`Failed to submit job for execution ${seed.execution.id}:`, error); - + console.error(`Failed to submit job for execution ${seed.execution.id}:`, { + message: error.message, + stack: error.stack, + name: error.name, + executionId: seed.execution.id, + modelId: seed.execution.modelid + }); + // Mark the individual execution as failed try { await TapisExecutionService.updateExecutionStatusOnGraphQl( @@ -158,11 +184,13 @@ export class TapisExecutionService implements IExecutionService { statusUpdateError ); } - + await decrementThreadModelSubmittedRuns(threadModelId); } - private handleSubmissionResults(failedExecutions: { execution: Execution; error: Error }[]): void { + private handleSubmissionResults( + failedExecutions: { execution: Execution; error: SerializableError }[] + ): void { if (failedExecutions.length > 0) { if (failedExecutions.length === this.seeds.length) { throw new Error("All jobs failed to submit"); @@ -441,7 +469,8 @@ export class TapisExecutionService implements IExecutionService { private async getExecutionResultsFromJob( jobUuid: string, execution: Execution, - isPublic: boolean + // eslint-disable-next-line @typescript-eslint/no-unused-vars + _isPublic: boolean ): Promise { const outputFolders = [ "", diff --git a/swagger.json b/swagger.json index 776a68c..c544491 100644 --- a/swagger.json +++ b/swagger.json @@ -735,6 +735,91 @@ "title": "Parameter", "type": "object" }, + "ModelParameter": { + "description": "A model parameter with comprehensive metadata", + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "Unique identifier for the parameter" + }, + "name": { + "type": "string", + "description": "Name of the parameter" + }, + "type": { + "type": "string", + "description": "Data type of the parameter" + }, + "description": { + "type": "string", + "nullable": true, + "description": "Description of what this parameter controls" + }, + "min": { + "type": "string", + "nullable": true, + "description": "Minimum acceptable value" + }, + "max": { + "type": "string", + "nullable": true, + "description": "Maximum acceptable value" + }, + "unit": { + "type": "string", + "nullable": true, + "description": "Unit of measurement for the parameter" + }, + "default": { + "type": "string", + "nullable": true, + "description": "Default value for the parameter" + }, + "value": { + "type": "string", + "nullable": true, + "description": "Current or assigned value" + }, + "adjustment_variable": { + "type": "array", + "nullable": true, + "description": "Variable that this parameter adjusts" + }, + "accepted_values": { + "type": "string", + "nullable": true, + "description": "Accepted values for the parameter (comma-separated string)" + }, + "position": { + "type": "number", + "nullable": true, + "description": "Position of the parameter in the model configuration" + } + }, + "required": [ + "id", + "type" + ] + }, + "BasicModelParameter": { + "description": "Basic model parameter information with just id and value", + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "Unique identifier for the parameter" + }, + "value": { + "type": "string", + "description": "Current or assigned value" + } + }, + "required": [ + "id", + "value" + ] + }, "Task": { "type": "object", "description": "A Task definition", @@ -1005,7 +1090,10 @@ "description": "A Subtask creation request definition", "required": [ "name", - "dates" + "dates", + "driving_variables", + "response_variables", + "regionid" ], "properties": { "name": { @@ -3983,8 +4071,8 @@ }, "/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/models": { "post": { - "summary": "Add models to a subtask", - "description": "Adds models to a subtask", + "summary": "Select model configurations for a subtask", + "description": "Select and add ModelConfiguration or ModelConfigurationSetup instances to use in this subtask for execution", "security": [ { "BearerAuth": [], @@ -4035,7 +4123,7 @@ }, "responses": { "200": { - "description": "Models added successfully", + "description": "Model configurations selected and added to subtask successfully", "content": { "application/json": { "schema": { @@ -4064,8 +4152,8 @@ }, "/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/data": { "post": { - "summary": "Add data to a subtask", - "description": "Adds data to a subtask", + "summary": "Select data for a subtask", + "description": "Select the data to use in the subtask per ModelConfiguration/ModelConfigurationSetup. You can obtain a blueprint from the blueprint endpoint to see available data inputs", "security": [ { "BearerAuth": [], @@ -4116,7 +4204,7 @@ }, "responses": { "200": { - "description": "Data added successfully", + "description": "Data selected and configured for subtask successfully", "content": { "application/json": { "schema": { @@ -4470,6 +4558,152 @@ } } }, + "/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/blueprint": { + "get": { + "summary": "Get blueprint (parameters and inputs) for a subtask", + "description": "Returns the complete model configuration blueprint for all models in a subtask", + "security": [ + { + "BearerAuth": [], + "oauth2": [] + } + ], + "tags": [ + "Subtasks" + ], + "parameters": [ + { + "in": "path", + "name": "problemStatementId", + "required": true, + "schema": { + "type": "string" + }, + "description": "The problem statement ID" + }, + { + "in": "path", + "name": "taskId", + "required": true, + "schema": { + "type": "string" + }, + "description": "The task ID" + }, + { + "in": "path", + "name": "subtaskId", + "required": true, + "schema": { + "type": "string" + }, + "description": "The subtask ID" + }, + { + "in": "query", + "name": "detailed", + "required": false, + "schema": { + "type": "boolean", + "default": false + }, + "description": "If true, returns full ModelParameter details. If false, returns basic info (id, value) only." + } + ], + "responses": { + "200": { + "description": "Complete blueprint for the subtask", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "model_id": { + "type": "string", + "description": "The model identifier", + "example": "https://w3id.org/okn/i/mint/c07a6f98-6339-4033-84b0-6cd7daca6284" + }, + "parameters": { + "type": "array", + "description": "Returns BasicModelParameter (id, value) by default, or full ModelParameter when detailed=true", + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/BasicModelParameter" + }, + { + "$ref": "#/components/schemas/ModelParameter" + } + ] + } + }, + "inputs": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "Input identifier", + "example": "https://w3id.org/okn/i/mint/ce32097e-641d-42af-b3f1-477a24cf015a" + }, + "dataset": { + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "Dataset identifier", + "example": "18400624-423c-42b5-ad56-6c73322584bd" + }, + "resources": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "Resource identifier", + "example": "9c7b25c4-8cea-4965-a07a-d9b3867f18a9" + }, + "url": { + "type": "string", + "description": "Resource URL", + "example": "https://ckan.tacc.utexas.edu/dataset/18400624-423c-42b5-ad56-6c73322584bd/resource/9c7b25c4-8cea-4965-a07a-d9b3867f18a9/download/barton_springs_2001_2010average.wel" + } + } + } + } + } + } + } + } + } + } + } + } + } + } + }, + "default": { + "description": "Default error response", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "message": { + "type": "string" + } + } + } + } + } + } + } + } + }, "/problemStatements/{problemStatementId}/tasks/{taskId}/subtasks/{subtaskId}/outputs": { "post": { "summary": "Publish all executions for a subtask",