[Feature]: Adapter should persist, not immediately unloaded after inference.

[yx-lamini]

🚀 The feature, motivation and pitch

Use llamafactory to tune a adapter:

kubectl port-forward svc/lamini-operator 8000:8000
curl -X POST "http://localhost:8000/v1alpha/tune" \
  -H "Content-Type: application/json" \
  -d '{
    "job_name": "test-job-21",
    "base_model": "meta-llama/Llama-3.2-3B",
    "dataset": "test.jsonl",
    "hyperparameters": {
      "max_steps": "1"
    }
  }'

Wait for the above job to finish, then send an inference:

kubectl port-forward svc/inference-router 8001:8000
curl http://localhost:8001/v1/completions \
    -H "Content-Type: application/json" --header 'Authorization: Bearer sk-1234' \
    -d '{
        "model": "meta-llama/Llama-3.2-3B-Instruct",
        "prompt": "San Francisco is a good test adf",
        "lora_request": {
            "lora_name": "test_adapter",
            "lora_path": "/app/lamini/jobs/test-job-21"
        }
    }'

Notice that the adapter got unloaded immediately after finishing the inference

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.