From a single text prompt to an interactive VR scene.

Text2VR is a research-driven system that turns natural language descriptions into VR-ready 3D environments and assets, combining panorama generation, asset extraction, 3D reconstruction, and VR deployment in a single pipeline.

The organization currently maintains a single main repository:

• Repository: Text2VR/Text2VR
  – LangGraph-based backend pipeline (app/, microservice clients, workflows)
  – React + TypeScript frontend (src/)
  – Service directories for panorama, segmentation, inpainting, and 3D generation
  – Docker Compose configuration and pretrained checkpoint folders

High-level system architecture (see the main repo for full details):

Text2VR is developed in the context of HCI and game/VR research:

• Submitted to Journal of Korea Computer Game Society (JKCGS)
• Text2VR: A Modular Pipeline for Interactive VR Scenes from a Single Text Prompt
  CM. Lee*, S. Myoung*, M. An*, Y. Jin*, EC. Lee†.
  Note: (*: Co-first Author, †: Corresponding Author)
  
  • Extended journal version emphasizing VR content authoring workflows, automatic asset/background generation, and interactive VR scene construction using Text2VR.

The code in this organization corresponds to these works and is intended to be reproducible and extensible for further research.

At a high level, the Text2VR pipeline:

1. Rewrites user prompts into optimized generation queries (LLM-based query rewrite via LangGraph).
2. Generates a 360° panorama from the prompt using DreamScene360 (Stable Diffusion + Stitch Diffusion), with optional self-refinement.
3. Segments interactive objects using GPT-4o + GroundingDINO + SAM and crops them as RGBA assets.
4. Converts 2D crops into 3D assets with TRELLIS, producing GLB models.
5. Inpaints the background after object removal using Stable Diffusion 2 Inpaint (wrap-aware).
6. Trains a 3D Gaussian Splatting background from the inpainted panorama (PLY) for immersive VR rendering.
7. Packages everything for VR, so scenes can be explored in a VR engine (e.g., Unity with a Gaussian Splatting plugin and colliders/interaction scripts).

A more detailed pipeline overview is available in the Text2VR repository:

• LangGraph Orchestration
  Explicit workflow graph managing query rewrite, panorama generation, segmentation, inpainting, 3D asset generation, and Gaussian Splatting.

• Microservice Architecture (via Docker Compose)
  Separate containers for:

  • Panorama generation (DREAMSCENE360/ → panorama-api)
  • Segmentation (ASSET_SEG/ → segmentation-api)
  • Inpainting (BG_INPAINT/ → inpainting-api)
  • 3D asset generation (TRELLIS → trellis-api)

• Frontend Integration
  React + TypeScript frontend (src/) for:

  • Prompt input and scene naming
  • Real-time pipeline status and intermediate previews
  • Result browsing and download (panorama, assets, PLY, Unity export)

• VR-focused Outputs

  • 3D Gaussian Splatting background (PLY)
  • GLB assets ready for import into VR engines
  • Unity-ready export endpoints (unity_assets API) for building interactive VR scenes

For full details, see the Text2VR repository README. At a glance:

• app/ – FastAPI backend and LangGraph workflow
• src/ – React frontend
• DREAMSCENE360/ – Panorama generation service
• ASSET_SEG/ – Segmentation service
• BG_INPAINT/ – Background inpainting service
• docker-compose.yml – Microservice orchestration
• docs/ – Architecture and pipeline diagrams
• pre_checkpoints/ – Pretrained model weights

Text2VR is developed by Team GARASANI (Capstone Design · Graduation 2025).

Chang-Min Lee @LeeChangmin0310 Project Lead & Maintainer

Soo-Youn Myoung @suyeonmyeong Core Contributor

Moon-Sik An @dalsik Core Contributor

Young-In Jin @0in11 Core Contributor

“Garasani (가라사니): the spark of perception, the thread of insight.”
Text2VR explores how unstructured prompts can be transformed into structured, interactive 3D worlds.

• The HCI conference paper and the computer game journal version both reference this organization as the implementation source.
• To reproduce the system:
  • Start from Text2VR/Text2VR
  • Follow the repository README for environment setup, model downloads, and pipeline execution
• Issues and feature requests can be opened on the main repository.