Create Back-End API endpoints and data extraction/transformation to replace MVP FSKX interaction

[JackJackUM]

The MVP for FSKX integration handles interaction with the FSKX run time server directly via the following repo:

link to repo

As a proof of concept for the back end setup, the goal is to replace the above (with exception of urls.py and views.py) with

• The first version of the backend API gateway
• The backend connection to the FSKX run time server(whether this is directly routed to from the gateway, or stored in a database)
• Deploy publicaly and give access to Pranav/Felipe so that the MVP can be connected to the backend API gateway and the old connection to FSKX be replaced.

[JackJackUM]

@antoniskappa : we created this issue in our stand-up on Wednesday 28th of May. Can you have a look at the issue and start working on it, so we can discuss it in the stand-up next Wednesday? The idea is to have an API endpoint of the data management platform that the dashboard can call to get data from various models via the data management platform. You can discuss with Pranav @pranavnbapat about the details. Thx, Jack.

[JackJackUM]

PS: this links with issue ambrosia2024/project#34 about a component in the data management platform that is taking care of contacting various models (climate change or food safety or integrated) that are available somewhere (either FSKX or Copernicus or others).

[letp1985]

From Technical Standup: 20250604

Questions from Antonis

• How frequently will this data request be ran
• Does data need to be persisted
• What is the run time for requesting data from FSKX

Actions for SCIO in short term

• Go for asynchronous processing (asyncio and httpx - background process)
• < 1 minute per call
• Backend database refreshes periodically (once per day, week)

Actions for UM/SCIO

• interface displays already available scenarios / parameters stored in the database
• user can request other scenarios, but this will take time to run as back end needs to retrieve from FSKX

Long term solution

• Pub Sub bus (e.g. Kafka)

Decisions to be made / clarifications

• How often is FSKX data changing? (BFR) Is the persisted data up to date? Do we schedule runs
• Is there a flag in FSKX server indicating data has been refreshed
• Front end / back end caching strategy - when we do and don't need it (based on size of data being returned and complexity of back end operations)

[JackJackUM]

@antoniskappa : any news on this issue? I recall that last week there would be an API available for testing? Let us know via this issue. Thx, Jack

[antoniskappa]

@JackJackUM we are working on the establishment of the deployment environment on premises. I will update you when there is progress and external testing can begin.

[JackJackUM]

Call per crop, initial temperature, initial concentration of pathogen, temperature range....and this for multiple consecutive simulations over a time period of ....months? Each simulation runs for 100 hours.

[arruda]

General requests for API based on region, crop, timescale, and possibly climate model:

POST /run-simulations/

{
  "crop": Maize,
  "nuts_id": "NL42",
  "climate_model": "RCP4.5",
  "time_period": [2025-05-11, 2025-12-11],
  "time_scale": "monthly"
}

Expected output:

{
  risk_index: [.....],
  yield: [....],
  discrite_time_period: [....],
  climate_change_parameters: {
    temperature: [....],
    precipitation: [....]...
  }
  uncertainty: {?},
  pathogens: {
    "salmonella" : {
       risk_index: [...]
       description: "???"
   },
   "listeria" : {
       risk_index: [...]
       description: "???"
   },  ...
 },
mitigation_strategies: [ 
    {
      "crop": "Lettuce",
      "strategy": "crop rotation",
     // ???
    },
]
}

[vageeshSaxena]

Inputs: I think we should also allow None values for “time_scale” or atleast set it internally to None if the “time_period” parameter is provided by the user and vice-versa. - TBD

{
  "crop": “Maize”, // type=str, possible values=[”Maize, Wheat, Legumes, Tomatoes, etc.” ], description=Crop (name) of interest
  "nuts_id": "NL42",  // type=str, possible values=[”NL42, DE112, etc”], description=NUT3 region ID of interest
  "climate_model": "RCP4.5",  // type=str, possible values=[”RCP2.6, RCP4.5, RCP6, and RCP8.5”], description=Climate model of interest
  "time_period": [2025-05-11, 2025-12-11],  // format=[YYYY-MM-DD (starting period), YYYY-MM-DD (end period)], type=list of string values, description=starting and end data of interest
  "time_scale": "monthly"  //  type=str, possible values=[”monthly”, “quarterly”, “yearly”, “decade”], description=time scale of interest.
}

[pranavnbapat]

Yes, I agree. Allow time_scale = null if time_period is provided.

If time_scale is provided and no time_period, then apply a (pre?)configured default period, the last full year.

If both are provided, aggregate to time_scale across the given time_period.

If neither is provided, return a 400 error with the expected combinations.

[pranavnbapat]

Also, about the NUTS level, we've always used the NUTS2 level. For some reason, I think we were discussing the NUTS3 level in the last meeting. If not, then ignore.

[vageeshSaxena]

Outputs:

Q1: Is there going to be a risk index for every date in the time period or just one category (R2) for the entire period?
Q2: What do you think about effects entry in mitigation_strategy?

{
 "risk_index": 0.72, // type=float, possible values  ∈ R,  description=Overall risk for a crop under a climate projection
“risk_level”: “moderate”, // type=str, possible values = [“low”, “moderate”, “high”], description=”Risk level based on risk index”
 "Yield": 123.433, // type=float, possible values  ∈ R, description = crop productivity/hectare or km2
 "discrete_time_period": [2025-05-11, 2025-12-11],  // format=[YYYY-MM-DD (starting period), YYYY-MM-DD (end period)], type=list of string values, description=starting and end data of projections
 "climate_change_parameters": {
    	"temperature": [30.2, 29.6, ….],  // type=list of floats, possible values  ∈ R, description = 	projected temperature for every day in the time period.
    	"precipitation": [30.2, 29.6, ….] // type=list of floats, possible values  ∈ R, description = 	projected precipitation for every day in the time period. 
  				}
 "uncertainty": {
  	"climate": {
		"temperature": [[30.2, 29.6, ….], [30.2, 29.6, ….]], // type=nested list of floats, 			possible values  ∈ R, description = 	uncertainty in the temperature for every day – 		likely to be a distribution for each day
		"precipitation": [[30.2, 29.6, ….], [30.2, 29.6, ….]], // type=nested list of floats, 			possible values  ∈ R, description = 	uncertainty in the temperature for every day – 		likely to be a distribution for each day
		}
	"yield": [123.433, 127.234, 432.234], // type= list of floats, possible values  ∈ R,  	uncertainty in crop yield for a chosen crop under that time period.
		},
  pathogens: {
    	"salmonella" : {
       		"risk_index": 0.55, //  description = Risk of the pathogen salmonella for a given crop and time period.
                         "risk_level": "moderate",
       	"description": "A small description about the pathogen salmonella, including side effects on 	crop and human health?",  // 	type=str
   			},
   	"listeria" : {
      		"risk_index": 0.22, //  description = 	Risk of the pathogen listeria for a given crop and time period.
               "risk_level": "low",
       	"description": "A small description about the pathogen listeria, including side effects on crop and human health",  // type=str
   },  
 },
mitigation_strategies: [ 
    {
      "crop": "Lettuce", // type=str, possible values = [Wheat, Maize, Lentils, Lettuce], description=”Alternative recommendation of crop suggested”
      "strategy": "crop rotation", // type=str, possible values = [crop rotation, tilting, change in sowing period, etc], description=”farming strategy suggested to improve crop health, productivity, and sustainability”.
      “effects”: 
	{
	“Yield”: “+10%”, type=str, possible values = [-100,100], description= % change in yield due 		to alternative crop or mitigation strategies,
	“risk index”: “0.10”,
	“risk level”: “low”, 
	“Environment”:
		{
		“GHG”: “-5%”, type=str, possible values = [-100,100],  description= % change in 		GHG due to alternative crop or mitigation strategies
		“soil_health”: “+5%”, type=str, possible values = [-100,100],  description= % change  		in soil health due to alternative crop or mitigation strategies
		}
	}
    },
]
}

[pranavnbapat]

Enhancing the example output we discussed last week, I’ve drafted a more detailed response schema that we could hope the API would return:

{
  "job_id": "2f4d9d97-1a71-4f55-9d2a-4a2a2f7a3f7e",
  "status": "completed",
  "request": {
    "crop": "Maize",
    "nuts_id": "NL42",
    "climate_model": "RCP4.5",
    "time_period": ["2025-05-11", "2025-12-11"],
    "time_scale": "monthly"
  },
  "time_grid": [
    "2025-05-31",
    "2025-06-30",
    "2025-07-31",
    "2025-08-31",
    "2025-09-30",
    "2025-10-31",
    "2025-11-30",
    "2025-12-31"
  ],
  "risk_index": [0.21, 0.28, 0.33, 0.31, 0.27, 0.25, 0.22, 0.20],
  "yield": [7.2, 7.0, 6.9, 6.8, 6.7, 6.7, 6.6, 6.6],
  "climate_change_parameters": {
    "temperature_mean": [14.2, 16.1, 18.3, 18.0, 15.5, 12.1, 9.8, 7.4],
    "precipitation_total_mm": [62, 48, 41, 52, 68, 75, 81, 93]
  },
  "uncertainty": {
    "risk_index": {
      "type": "stddev",
      "values": [0.03, 0.04, 0.05, 0.05, 0.04, 0.03, 0.03, 0.02]
    },
    "yield": {
      "type": "ci95",
      "lower": [6.9, 6.7, 6.5, 6.4, 6.3, 6.3, 6.2, 6.2],
      "upper": [7.5, 7.3, 7.2, 7.1, 7.1, 7.0, 7.0, 7.0]
    }
  },
  "pathogens": {
    "salmonella": {
      "risk_index": [0.05, 0.06, 0.08, 0.07, 0.06, 0.05, 0.04, 0.04],
      "description": "Some description"
    },
    "listeria": {
      "risk_index": [0.04, 0.05, 0.06, 0.06, 0.05, 0.04, 0.03, 0.03],
      "description": "Some description."
    }
  },
  "mitigation_strategies": [
    {
      "crop": "Maize",
      "strategy": "Irrigation scheduling",
      "notes": "Some additional notes from the model/API if returning, or we can inject our own"
    },
    {
      "crop": "Maize",
      "strategy": "Crop rotation",
      "notes": "Some additional notes from the model/API if returning, or we can inject our own"
    }
  ],
  "provenance": {
      "source": "FSKX",                        // system used
      "fskx_model_id": "fskx-maize-risk-v2.1", // model identifier/version
      "fetched_at": "2025-08-26T09:37:12Z",    // timestamp of retrieval
      "cache_status": "miss_then_store"        // whether result came from cache
  }
}

Does this make sense?

[letp1985]

Question from VS : Are we going to simulate mitigation strategies using the same time parameters ? OOS for time being but to be discussed in future.