add response_mime_type to GenerationConfig

[0xDing]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Is your feature request related to a problem? Please describe.

The GenerationConfig as described in the API reference includes new features such as response_mime_type and seed. However, these features are currently not available in the SDK. Could these be implemented or updated in the SDK to match the documentation?

Describe alternatives you've considered

Fork it

Additional context

[Markus-ipse]

We're currently battling with inconsistent JSON output, as the model has a tendency to either wrap the response in backticks or adding comments about the JSON response outside of the actual JSON blob.

I am really hoping being able to specify response_mime_type will solve these issues

[Giasinit]

I keep getting this response:
GoogleGenerativeAIError: [VertexAI.GoogleGenerativeAIError]: got status: 500 Internal Server Error. {"error":{"code":500,"message":"Internal error encountered.","status":"INTERNAL"}}

[Markus-ipse]

Last week I tried this forcing typescript to ignore the unrecognized param responseMimeType by casting the ´generationConfigasany`.

{
    model: 'gemini-1.5-pro-preview-0409',
    generationConfig: {
      responseMimeType: 'application/json',
    } as any, // remove `as any` after updating the Vertex AI SDK
}

And it seemed to work, I was more consistently getting a JSON response! Until when I tested it today again and now it's all of a sudden it is broken I get the same error as @Giasinit

"[VertexAI.GoogleGenerativeAIError]: got status: 500 Internal Server Error. {\"error\":{\"code\":500,\"message\":\"Internal error encountered.\",\"status\":\"INTERNAL\"}}"

If I remove the generationConfig then it stops throwing errors, but it also stop returning JSON, even-tough I have prompts pleading for it to do so :(

[deathemperor]

Just checked and the 500 issue has been resolved

[Giasinit]

@deathemperor i see.. thanks

[Giasinit]

@3lue9 we solved the problem finally LOL.
I will explain later, in the meantime read the discussion to understand what I am referring to.

[tamerxkilinc]

@Giasinit Could you share your solution with me? Facing the same problem.

[Giasinit]

@Giasinit Could you share your solution with me? Facing the same problem.

@tamerxkilinc, if you want, you can share you code, to figure out what language you are using, and what configuration!

[tamerxkilinc]

For us the solution mentioned above is working. Just adding response_mime_type to generationConfig and declaring it as any works and gives us no errors.

[Firdaus-prog]

Today, Its not working for us anymore ☹:

ERROR Invoke Error
{
"errorType": "GoogleGenerativeAIError",
"errorMessage": "[VertexAI.GoogleGenerativeAIError]: got status: 500 Internal Server Error. {"error":{"code":500,"message":"Internal error encountered.","status":"INTERNAL"}}",
"name": "GoogleGenerativeAIError",
"stack": [
"GoogleGenerativeAIError: [VertexAI.GoogleGenerativeAIError]: got status: 500 Internal Server Error. {"error":{"code":500,"message":"Internal error encountered.","status":"INTERNAL"}}",
" at throwErrorIfNotOK (file:///var/task/src/functions/vertexai-generate-insights.mjs:57113:15)",
" at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
" at async generateContent (file:///var/task/src/functions/vertexai-generate-insights.mjs:57573:7)",
" at async generateCallInsights (file:///var/task/src/functions/vertexai-generate-insights.mjs:71188:21)",
" at async lambdaHandler (file:///var/task/src/functions/vertexai-generate-insights.mjs:71472:22)",
" at async runRequest (file:///var/task/src/functions/vertexai-generate-insights.mjs:71396:27)"
]
}