Matterport 3D to Text Adventure

Generating Interactive Fiction From 3D Scanned Model

Type: 3D Scan to Text Adventure

Date: July 2024 - July 2025

Advisor: Daragh Byrne

Role: Research Assistant, Developer, System Designer

Technology: Javascript, HTML, CSS, LLM API

Play Game 🎮

Generated example text adventure is hosted and playable via web! Navigate and explore the world only using 'texts'. The base 3D scan is from the E.A.A.T exhibition scan, you can find more information here.

Project Overview

Matterport 3D Scan

Matterport 3D scanning is a technology that creates immersive, three-dimensional digital representations of real-world spaces. Users interact with Matterport 3D scans primarily through three different viewing modes that each offer unique perspectives on the space. The walkthrough mode allows users to navigate by clicking on circular navigation points scattered throughout the scan, creating the experience of actually walking through the physical location.

3D to Text Adventure API

Step 2. Configure clusters and cull sweep points.

Step 6. Take screen shots of culled sweep points.

From 3D Data to Game - System Design

A system diagram that shows how the API leverages LLM to create text adventure from 3D data.

The Accessible Text Adventure

Making the output text adventure was not an easy task. Since I wanted to keep the style of texts kept being accumulated throughout the play, it caused a problem of screen reader announcing the dynamic web content. In other words, I need a way for the text reader to only announce the newly added texts in a correct order. To solve this problem, the solution I took was to maintaining the sperate unseen division that contains only the updated texts, and when it is all announced properly, then the content is cleared out for the next run. Below is the code snippet of the how the text adventure could be announced properly and be accessible.

function getLiveRegion() {
    let liveRegion = document.getElementById('liveRegion');
    if (!liveRegion) {
        liveRegion = document.createElement('div');
        liveRegion.setAttribute('id', 'liveRegion');
        liveRegion.setAttribute('aria-live', 'polite');
        liveRegion.setAttribute('aria-atomic', 'true');
        liveRegion.style.position = 'absolute';
        liveRegion.style.left = '-9999px';
        document.body.appendChild(liveRegion);
    }
    return liveRegion;
}