IMP: LLM-Driven Dynamic User Interface Generation for IoT Environments

Proof-of-Concept Implementation

Abstract

This repository contains a proof-of-concept system demonstrating the feasibility of using Large Language Models (LLMs) to dynamically generate user interfaces for Internet of Things (IoT) devices at runtime. The system ingests Web of Things (WoT) Thing Descriptions, aggregates device capabilities, and employs a requirements-aware LLM pipeline with schema enforcement to produce context-appropriate interfaces delivered to heterogeneous client devices in real time.

This implementation accompanies the associated scientific paper and serves as its reference implementation.

Table of Contents

• Research Context
• System Overview
• Architecture
• Key Contributions
• Installation
• Usage
• Configuration
• Evaluation
• Project Structure
• Troubleshooting
• License
• Citation

Research Context

Traditional IoT interfaces are either statically designed or require manual configuration for each device type. This proof-of-concept explores an alternative approach where:

1. Device capabilities are discovered automatically via standardized Thing Descriptions (W3C Web of Things)
2. User interfaces are generated dynamically by an LLM that reasons about available actions, device form factors, and contextual requirements
3. Schema enforcement ensures validity through constrained generation and automatic retry mechanisms
4. Real-time delivery enables immediate interface updates as the IoT environment changes

The system addresses the research question: Can LLM-driven UI generation provide a viable alternative to static interface design for heterogeneous IoT environments?

System Overview

The system implements a microkernel architecture where a central orchestration service coordinates between:

• Thing providers exposing IoT device capabilities via WoT Thing Descriptions
• Capability modules providing contextual data (e.g., user activity recognition)
• A knowledge base managing requirements documents and LLM interactions
• Client devices rendering generated interfaces across different form factors

The LLM receives a structured prompt containing available actions, device capabilities, and contextual constraints, then produces a schema-conformant UI specification that clients render natively.

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                           Core System (3001)                            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌───────────────┐   │
│  │   Service   │  │    Thing    │  │   Action    │  │  Orchestrator │   │
│  │  Registry   │  │  Ingestion  │  │  Registry   │  │               │   │
│  └─────────────┘  └─────────────┘  └─────────────┘  └───────────────┘   │
│                              │                              │            │
│                              ▼                              ▼            │
│                    ┌─────────────────┐           ┌──────────────────┐   │
│                    │    WebSocket    │           │   REST API       │   │
│                    │     Server      │           │   Endpoints      │   │
│                    └─────────────────┘           └──────────────────┘   │
└─────────────────────────────────────────────────────────────────────────┘
           │                    │                          │
           ▼                    ▼                          ▼
┌──────────────────┐  ┌─────────────────┐       ┌──────────────────────┐
│  Knowledge Base  │  │  Client Devices │       │   Thing Providers    │
│     (3005)       │  │                 │       │       (3006)         │
│  ┌────────────┐  │  │  ┌───────────┐  │       │  ┌────────────────┐  │
│  │    RAG     │  │  │  │Smartphone │  │       │  │  Light Switch  │  │
│  │  Service   │  │  │  │  (5173)   │  │       │  │                │  │
│  ├────────────┤  │  │  ├───────────┤  │       │  ├────────────────┤  │
│  │ Guardrails │  │  │  │  Tablet   │  │       │  │  TractorBot    │  │
│  │  Engine    │  │  │  │  (5174)   │  │       │  │                │  │
│  ├────────────┤  │  │  ├───────────┤  │       │  └────────────────┘  │
│  │    LLM     │  │  │  │  Voice    │  │       └──────────────────────┘
│  │  Interface │  │  │  │ Device    │  │
│  └────────────┘  │  │  └───────────┘  │       ┌──────────────────────┐
└──────────────────┘  └─────────────────┘       │ Capability Modules   │
                                                │       (3003)         │
                                                │  ┌────────────────┐  │
                                                │  │   Activity     │  │
                                                │  │  Recognition   │  │
                                                │  └────────────────┘  │
                                                └──────────────────────┘

Component Descriptions

Component	Port	Description
Core System	3001	Central orchestration service managing device/Thing registration, action aggregation, and UI generation coordination
Service Registry	3000	Maintains registry of all connected services, devices, and capabilities
Knowledge Base	3005	RAG-enabled service handling requirement documents, LLM guardrails, and schema-enforced generation
Things Service	3006	Simulates WoT-compliant IoT devices (light switch, robotic arm)
Activity Recognition	3003	Example capability module providing user activity context
Smartphone Device	5173/3002	Mobile form factor client with touch-optimized rendering
Tablet Device	5174/3012	Large-format client with 12-column grid layout system
Voice Device	—	Audio-first client for hands-free interaction

Key Contributions

This proof-of-concept demonstrates several aspects:

1. Multi-Thing Orchestration

A single client device can control multiple IoT Things simultaneously. The core system aggregates action catalogs from all registered Things, and the LLM generates unified interfaces with proper thingId attribution.

2. Guardrailed LLM Generation

The knowledge base injects strict constraints preventing the LLM from hallucinating non-existent actions:

• Only registered action IDs may appear in generated UIs
• JSON Schema validation ensures structural conformance
• Automatic retry mechanisms handle generation failures

3. WoT-Based Action Registry

Thing Descriptions are automatically parsed into a normalized action registry, abstracting transport-layer details (HTTP, CoAP, etc.) from the UI generation process.

4. Capability Framework

External services register as "capabilities" exposing JSON Tool APIs. The LLM can invoke these during generation to incorporate real-time contextual data (e.g., user activity state).

5. Real-Time Delivery

WebSocket connections maintain persistent channels between the core and client devices. Generated UIs are broadcast immediately and cached per-device for reconnection scenarios.

6. Device-Agnostic Generation

The system generates device-neutral UI specifications that clients interpret according to their form factor, enabling the same logical interface to render appropriately on smartphones, tablets, or voice devices.

Installation

Prerequisites

• Docker and Docker Compose (recommended)
• Node.js 18+ and npm (for local development)
• Access to an LLM endpoint (local or cloud-based)

Docker Deployment (Recommended)

# Clone the repository
git clone https://github.com/[repository-url]/IMP.git
cd IMP

# Configure LLM endpoint (optional - defaults to local endpoint)
export LLM_ENDPOINT="http://localhost:1234/v1/chat/completions"

# Build and start all services
docker compose up --build

The first build may take several minutes. Subsequent starts reuse cached layers.

Local Development

# Install dependencies (workspaces handle sub-packages)
npm install

# Start services individually
npm run dev --workspace packages/core-system
npm run dev --workspace packages/knowledge-base
npm run dev --workspace packages/things
npm run dev --workspace packages/device
npm run dev --workspace packages/tablet-device
npm run dev --workspace packages/activity-recognition

Usage

Service Endpoints

Once running, services are accessible at:

Service	URL
Service Registry API	http://localhost:3000
Core System API	http://localhost:3001
Knowledge Base	http://localhost:3005
Activity Recognition	http://localhost:3003
Smartphone UI	http://localhost:5173
Tablet UI	http://localhost:5174

Verifying System State

# Check registered devices, Things, and capabilities
curl http://localhost:3001/registry

# View available actions from all Things
curl http://localhost:3001/actions

# Manually trigger UI generation
curl -X POST http://localhost:3001/generate-ui

Data Flow

1. Registration Phase: Devices and Things register with the core system
2. Discovery Phase: Core aggregates Thing actions and device capabilities
3. Generation Phase: Knowledge base constructs LLM prompt with guardrails
4. Delivery Phase: Generated UI broadcasts to target devices via WebSocket
5. Interaction Phase: User actions route through core to appropriate Things

Configuration

Environment Variables

Variable	Description	Default
LLM_ENDPOINT	Chat completions API URL	http://localhost:1234/v1/chat/completions
LLM_MODEL	Model identifier for local endpoints	—
OPENROUTER_API_KEY	API key for OpenRouter (if used)	—
OPENROUTER_MODEL	Model identifier for OpenRouter	—

Knowledge Base Persistence

A Docker volume knowledge-base-data persists requirement documents across restarts. To reset:

docker volume rm imp_knowledge-base-data

Evaluation

The repository includes an evaluation harness for systematic testing:

cd packages/eval-harness

# Direct LLM evaluation
npm run eval -- --mode chat --prompt "Generate a light control UI"

# Full pipeline evaluation
npm run eval -- --mode kb --prompt "Generate a light control UI"

Results are saved as timestamped JSON transcripts with heuristic failure detection for regression analysis.

Project Structure

IMP/
├── packages/
│   ├── core-system/           # Orchestration and registry services
│   │   ├── src/
│   │   │   ├── services/      # registry.js, orchestrator.js
│   │   │   ├── transport/     # websocket.js
│   │   │   └── routes.js
│   │   └── action-registry.js
│   ├── knowledge-base/        # RAG and LLM interface
│   │   └── src/
│   │       ├── llm.js         # LLM calls with guardrails
│   │       ├── rag.js         # Document retrieval
│   │       └── store.js       # Requirement storage
│   ├── device/                # Smartphone client
│   │   └── src/
│   │       ├── App.jsx        # React UI shell
│   │       └── api.js         # Device registration
│   ├── tablet-device/         # Tablet client with grid layout
│   │   └── schema.json        # 12-column grid schema
│   ├── voice-device/          # Voice-first client
│   ├── things/                # WoT Thing simulators
│   ├── activity-recognition/  # Capability module example
│   └── eval-harness/          # Evaluation tooling
├── docker-compose.yml         # Full stack orchestration
├── architecture.md            # Detailed design document
└── .env                       # Configuration

Troubleshooting

Issue	Solution
LLM generates invalid actions	Verify Thing registration at /registry; ensure knowledge-base has latest guardrails
No UI appears on devices	Check device registration and WebSocket logs: docker compose logs device-api -f
Capability errors	Verify capability module registration at /registry
Schema validation failures	Check LLM output format; the system retries automatically but persistent failures indicate prompt issues

License

This project is released under the MIT License. See LICENSE for details.

Citation

If you use this proof-of-concept in your research, please cite:

@software{imp_2025,
  author = {[Author Name]},
  title = {{IMP}: {LLM}-Driven Dynamic User Interface Generation for {IoT} Environments},
  year = {2025},
  url = {https://github.com/[repository-url]/IMP}
}

---

Note: This repository serves as the reference implementation for the accompanying scientific paper. For theoretical foundations, experimental methodology, and detailed evaluation results, please refer to the publication.