We had built a SaaS application for a client so that non-technical users can create and configure their own 3D environments via a simple web user-interface. They have the option to choose the type of landscape, weather conditions as well as uploading their 3D models to position, rotate, and scale them within the environment. The purpose of this application is for sales & marketing, to demonstrate to clients how their large-scale products would look in a real world setting. The web application can render the scene and allow users to move around in the scene using a mouse and/or keyboard. With WebVR integration it's also possible to explore the 3D environment with VR headsets such as the HTC Vive.
Another display method they requested was the MS Hololens, the user would load the Hololens app, select any new or existing project and then have the project load at runtime downloading all assets/models etc from the server.
Typically, when developing a Hololens app, a developer would create the whole project using Unity/C#, compile the application including all models and assets then load the self-contained app on the Hololens. However, as we had a need to download and render models and scenes at runtime we encountered significant technical limitations of the device.
Attempted solutions & outcomes
1) Loading binary FBX formated models and rendering at runtime. Binary FBX is a convenient format to use as it compiles all of the vertex data as well as textures and images into a single file which means users don't need to upload folders or zip folders containing multiple files. It's sensible not to 're-invent the wheel' so we looked for existing FBX runtime loaders on the Unity Asset Store, however we just could not find any working libraries. The problems included:
- Dependencies on other installed applications such as ASIMP for decoding the binary FBX files.
- Some only cover ASCII format FBX.
- Any others were very poorly documented and poorly supported, having sent the publisher's questions and never receiving answers.
- The HoloJS library is still very experimental and immature. There is limited functionality around gesture inputs and not all features are supported.
- We first tried deploying models at compile time, but the HoloJS was unable to even load those.
- We tried downloading the fbx files to the device at runtime and attempted to render using the Three.js FBXLoader, however this failed due to the lack of support for this file format via HoloJS.
- Three.js has its own JSON file format for models, so we experimented downloading and rendering models using this format, which was successful. However, when we tried using complex models, particularly where there was a lot of textures, the application crashed due to lack of memory. You only get around 1GB or RAM to play with on the device which seems quick to exhaust when rendering at runtime.
3) After failing with FBX and JSON model formats, we then moved to OBJ/MTL. This format combo is more widely adopted throughout the 3D modelling community due to its open and simple format.
- There are a lot more Asset Bundles in the Unity Asset Store for rendering OBJ models at runtime and this looked to be promising. We tried several including SimpleObj which did indeed allow us to render models with textures in the Unity editor, but when trying to render on the Hololens device, all faces are rendered pink and no textures are applied. If we had time we could have attempted to trace this incompatibility bug and fix it, however due to other attempts to render textures at runtime exhausted all RAM, we expected a similar outcome here too.
3D rendering is complex and resource intensive, it generally requires high-spec machines and graphics cards to use it seamlessly. As version 1 of the MS Hololens is itself an experimental prototype it can be forgiven for its limited resources especially as it is a tetherless device unlike the VR headsets such as the Vive and Rift. As a result, we must accept there are hard limits to what can currently be achieved.
One may have seen highly complex 3D model applications using a Hololens device and ask ‘so how do they do it?’ and the simple answer is that pre-compiling models that are then deployed to the device when installed are far more efficiently used by the device.
So what now? If like us you still want to be able to efficiently download and render 3D models at runtime, the only real option (to date) is to use Unity AssetBundles. An AssetBundle allows a project to precompile assets into proprietary file formats, these can then be downloaded and used as efficiently as models deployed at installation. The big caveat being that Unity Pro version is required, which is expensive. One will also need to setup bash scripts to compile your 3D models to AssetBundles if you want to use user-uploaded content.