Interactive Story Labs Halves Presentation
Project Description
Project Anamnesis is a 14-week exploration of interactive storytelling that combines live-action footage with advanced technology to create an emotionally immersive experience. Designed for the C.A.V.E.R.N. system at the Entertainment Technology Center (ETC), the project utilizes a 270-degree curved projection screen and a 20-foot-wide interactive play area, enabling dynamic local player interactions within virtual environments. The project’s goal is to showcase the system’s capabilities, focusing on player interaction design, narrative depth, high-fidelity audio, and real-time user tracking. Team Interactive Story Labs will craft a 3-5 minute immersive journey, giving the viewer the power to actively engage with the unfolding narrative. This experience aims to evoke strong emotional connections, allowing the player to feel a sense of personal agency while deepening their involvement in the story’s world.
What I Did
I replicated the CAVERN display in the Unity3D with the same scales by adding a cylinder mesh. I also added a canvas object of 5060 x 2060 pixels, which are the pixels of the CAVERN display. For the particular way the projectors are set up, there needs to be two image frames with the same image or video and in the same position for film to be displayed correctly.
Screenshot of the virtual CAVERN in Unity3D.
I added a Stickman Manager from the Body Tracking Unity Asset, which creates a meta skeleton prefab when the sensor captures a person. The meta skeleton tracks the joints of the user by creating points from the world and translating them into the scene, which I used to identify the points that represent the left and right ears and eyes and the chin. I added a prism as a visual indicator of the head orientation as a child of the Stickman Prefab. Later, I implemented a script that calculates the middle point between the ears, and the middle point between the chin and both eyes to place the prism between them.
In another script, I look for the prism once the Stickman is in scene, and create a Raycast that begins in the prism and shoots forward towards the CAVERN mesh. When the raycast collides with the mesh collider, it displays an image that is translated to the canvas object with UV coordinates. This creates a visual representation of the head tracking in the CAVERN display. Like so:To create better stability, I implemented a queue to store the rolling average for each uv coordinate that enqueues the newest coordinate after dequeuing the bottom coordinate when the indicated maximum was reached. With different testing, I noticed that a queue with a maximum of 150 make the gaze tracker smoother. This is the current process so far.
I also assisted with the film shooting by managing the costumes and props for all the scenes.
Tools and Software
Unity3D - C# - ORBBEC Fento Bolt Sensor