A sample Unity project for Microsoft HoloLens that lets you interact with a floating robot head using speech recognition, and getting answers spoken back by the "bot" using Text-to-Speech. The commands sent to the bot are powered by the Microsoft Bot Framework and LUIS from Microsoft Cognitive Services.
To get started with HoloLens, visit the Windows Holographic Dev Center.
- Hovering bot head (aka HoloBot) with looping sound
- Tap the HoloBot to move it to a different location, tap again to place
- Gaze at the HoloBot to trigger the speech recognizer, you will hear a "Ping" sound
- Speak "commands" using natural language
- The HoloBot speaks back to you using Speech Synthesis (aka Text-to-Speech, or TTS)
- All sounds and speech use spatial audio that originate from the HoloBot
- The "brain" of the HoloBot can be any public bot built with the Microsoft Bot Framework
- The HoloBot model and sounds come from the Holographic Academy tutorial: Holograms 240: Sharing Holograms
- Uses Gaze Manager & Gesture Manager from HoloLens Unity Toolkit (Input scripts)
- Uses Text to Speech Manager from HoloLens Unity Toolkit (Utilities scripts)
- Make sure to copy a UWP build of Newtonsoft.Json.dll in the /Plugins folder of the HoloBot Unity project
- Edit the Inspector settings for the Newtonsoft.Json.dll plugin as follows:
- Create and register your bot as per the intructions at https://dev.botframework.com. Since HoloBot uses free natural language dictation, it is highly recommended that your bot support NLP via Natural Language Understanding Service (LUIS) from Microsoft Cognitive Services
- Enable the Direct Line channel on your bot
- Generate and copy your Direct Line secret (aka API key)
- Open BotService.cs in the /Scripts folder of the HoloBot Unity project and paste your Direct Line secret in the _APIKEY private string
I want to offer special thanks to the following people who have helped me in building this sample app:
- Jarez Bienz, for the Text-to-Speech component that he wrote the the HoloLens Unity Toolkit, and for his help in integrating my UWP Bot Framework code into Unity
- Kat Haris, for her awesome Unity skills and helping me with audio sources triggers
- The whole Microsoft Holographic Academy team & mentors - especially Pat - for their awesome training, resources, patience and help
- Twitter: @ActiveNick
- Blog: AgeofMobility.com