This repository hosts the MindScapeAI project, an innovative Unity-based framework designed for creating immersive AI-powered Non-Player Characters (NPCs) in virtual reality. Developed by team NeuralNexus at the Stanford Immerse The Bay Hackathon, the project integrates OpenAI API, Azure Voice API, Google Cloud Speech to Text, and Oculus Lip Sync to create a unique VR conversational experience.
MindScapeAI aims to revolutionize the realm of AI interaction by merging it with the visually immersive world of VR. Our goal is to elevate standard dialogues into extraordinary visual journeys, providing an engaging and creative platform for AI engagement in VR.
- AI-Powered NPCs: Leveraging the power of OpenAI's GPT-4 to create responsive, intelligent NPCs.
- Voice Interaction: Integrating Azure Voice and Google Cloud Speech to Text for seamless voice communication.
- Visual Storytelling: Utilizing OpenAI's image generation API to create dynamic, contextually relevant visuals.
- Immersive VR Experience: Building on Unity 2021.3.x, the project offers a fully immersive VR experience.
- Customizable Personalities: Featuring diverse AI personalities like Travel Guide, Medical Professional, Friendly Companion, and Storyteller.
- Oculus Lip Sync: Enhancing realism with accurate lip-syncing for NPCs.
Ensure the following services are set up with the provided instructions and create configuration files in the StreamingAssets folder:
Review setup instructions for these repositories:
services_config.json template:
{
"OpenAI_APIKey": "your_openai_api_key",
"AzureVoiceSubscriptionKey": "your_azure_voice_subscription_key",
"AzureVoiceRegion": "your_azure_voice_region"
}gcp_credentials.json template:
{
// Your Google Cloud credentials here
}After setting up, run the scene to interact with the AI NPC. Ask questions and experience the AI's responses in an immersive VR environment.
This project honed our skills in debugging and problem-solving under complex scenarios. It highlighted the importance of leveraging team strengths and selecting ambitious yet attainable goals.
Looking ahead, we plan to enhance MindScapeAI with better models for even more fluid conversations, integrate Vision capabilities to give the Avatars the ability to see and be aware of their surroundings, and expand platform compatibility. By incorporating user feedback, we aim to continually evolve our platform to exceed expectations in the rapidly advancing AI-VR landscape.