Here's how I imagine this working. I record a replay, giving me an audio file (say 30min long) in mp3 format. I upload this to metauniService along with some identifier for the replay (consisting of the pocketId and replayId, the in-game UI should provide this string for me to just paste into a field on the webpage). Then the code
(a) splits the mp3 file into 6min chunks
(b) processes the mp3 file to generate a transcript (OpenAI has an API for this, should be easy)
(c) uploads the files to Roblox Cloud, gets asset IDs
(d) modifies the DataStore entry for the replay to add the asset IDs and timestamp data, along with the transcript data
This requires me to understand the data structure storing the replays (presumably straightforward) and to have additional fields added for transcripts (text).
Ideally we would generate embeddings as we were doing with Pinecone, but this is prohibitively expensive to do in a hosted manner at the moment, and further work in this direction would require hosting our own vector DB at metauniService (doable, maybe a day's work).
Here's how I imagine this working. I record a replay, giving me an audio file (say 30min long) in mp3 format. I upload this to metauniService along with some identifier for the replay (consisting of the pocketId and replayId, the in-game UI should provide this string for me to just paste into a field on the webpage). Then the code
(a) splits the mp3 file into 6min chunks
(b) processes the mp3 file to generate a transcript (OpenAI has an API for this, should be easy)
(c) uploads the files to Roblox Cloud, gets asset IDs
(d) modifies the DataStore entry for the replay to add the asset IDs and timestamp data, along with the transcript data
This requires me to understand the data structure storing the replays (presumably straightforward) and to have additional fields added for transcripts (text).
Ideally we would generate embeddings as we were doing with Pinecone, but this is prohibitively expensive to do in a hosted manner at the moment, and further work in this direction would require hosting our own vector DB at metauniService (doable, maybe a day's work).