HoloLens – Seeing IS Believing

I’m in Seattle this week for the Microsoft S4 Conference and am enjoying seeing the truly global perspective of Microsoft as a company. Virtually all of the sessions I’ve attended are covered by the NDA (Non Disclosure Agreement) I’ve signed as an employee so I can’t discuss anything from those unfortunately, but yesterday I did have some great luck.

I was provided a HoloLens unit for the night to have a good play with. If you’re not familiar with what HoloLens is then the official description is:

Microsoft HoloLens is the first self-contained, holographic computer, enabling you to engage with your digital content and interact with holograms in the world around you.

I’ve known about the HoloLens for a while now and, I have to admit, I had been a bit skeptical about videos like the above – surely these were “mocked up” demos of what could be possible, not what was actually already here with the current iterations of HoloLens.

I was dead wrong.

The last time I was truly blown away by a bit of tech I was hands on with was around February last year (2016) when I saw the HP Sprout in Melbourne. The HoloLens is staggeringly immersive – I definitely “lost” three or four hours playing on it last night with most of that on Fragments:

You become the detective in a high-tech crime thriller. Experience compelling new possibilities for storytelling and gameplay.

Some thoughts on Fragments in no particular order (keep in mind I’m not a gamer at all, this did a lot to draw me in!):

• You start by scanning/mapping the room(s) you’re in so that the game understands your environment – you are not restricted to a pre-build world to play in.
• The audio – it’s amazing surround sound from the HoloLens – you just get used to hearing someone talking “over your shoulder” and turning around and seeing the holographic representations of people; it’s entirely believable and lifelike audio.
• The game play – it responds to your environment i.e. characters sit down on your furniture, they walk through your doors. It’s really difficult to describe just how amazing that is and the value it adds to game play until you’ve actually experienced it for yourself.
• You very quickly adapt to the tools at your disposable in this mixed-reality environment:
  • You forget within 5 minutes that you’re wearing a headset (honestly, you no longer notice it).
  • You start to rely on virtual tools e.g. you have your “crime lab” and “maps” pinned to one of your walls and you just get used to returning to that time and again during the investigation.
  • Voice commands – leveraging Cortana you just speak naturally during the game e.g. “examine this evidence” or “launch scan” “close” – you can use the hand gestures to do all of this as well, but you often find it faster using voice. Further to this, you quickly increase your speed of interacting – you pick up, examine, dispose of evidence faster and faster as you get used to it.
• You’re very active
  • You’re literally walking around your room, scanning for evidence and exploring what is being holographically represented on the floors, walls, roof, tables etc
  • As you move around the room the characters move too – they watch you, their eyes follow your movements etc.
  • When the action freezes as part of the game play, you can “walk around” or “circle” the scenes/characters to see the full 3D elements of the game play.
• You’ll lose track of time! I only stopped playing because the battery was going critically flat!

If I sound excited, it’s because I am. I’ve seen Occulus Rift before and they’re amazing too however you are always tethered to a PC generating the content. To be able to wander around within a mixed reality environment with no cables was liberating.

I could go on, but you probably get the idea. Another app I played with was the HoloLens HoloTour:

This was also very immersive and educational. I’ve never been to the Colosseum in Rome before but with this, being able to walk around inside it, it felt like the next best thing. Whilst I am sure it is very expensive to develop this kind of content, you can definitely see how this has a place in education. Speaking of education, the HoloLens Insight Heart app is a good medical training application that shows a proof of concept about how students could potentially be trained. Again, the voice commands were super easy and responsive e.g. “make bigger/smaller” or “rotate” or “pause” – the app responded immediately to these commands:

The guy that loaned me the HoloLens said he now uses it as his primary device for completing “work” at home – from email to Excel spreadsheets to browsing the web, he just sits on his couch with a bluetooth paired keyboard/mouse and gets to work. He will pin a browser to one wall, his email to another and flip between them by moving his head. Because HoloLens will remember the layout of your house/room and always keep your pinned apps in the same place you can do things like pin a weather app to the back of your front door so you always know the weather forecast as you’re exiting the house.

Furthermore, the natural interaction between the real vs holographic world is evident and natural. For example, if you have an open app as you move it around the room to “pin” it somewhere you may decide you want to put in on table. As you move your hands to lower the app onto the table it will meet slight “resistance” as HoloLens recognises the real, physical table and provides resistance as a prompt so you can set the app down on the surface of the table. You can override this of course by pushing a bit harder, but this highlights the fact that HoloLens has a true awareness of the physical reality around you and applies that to everything you’re doing virtually.

I think this is a key point: whilst the truly immersive 3D apps are still largely in development, you can run any Win32 app via HoloLens still because it is a fully functioning Win10 device. This means it is both useful immediately and will only increase in usefulness as more apps are developed.

Here’s an example of a developer building such an app for a proof of concept:

The take home message is this – if you get a chance to play with a HoloLens jump at it! It’s an experience that you’re not likely to replicate easily elsewhere and it is truly hinting at the innovation and future that is coming.