# The technology behind the experience. > Explore Farwhen, the working name for our modular architecture connecting story intelligence, XR, spatial systems, and agent tooling. ## Imagine an observatory that remembers you A visitor steps into a room inspired by an ancient observatory. Four parts of the architecture could bring the scene to life. - **A guide from another century**: A projected astronomer answers a question, drawing on the history and mythology of the night sky. (Worlds & Characters) - **A different sky**: The visitor selects a constellation; the interactive world shifts to follow that choice. (Interactive Experiences) - **The room joins in**: A shared cue could change the projection, audio, and a robot guide together. (Spatial Orchestration) - **An operator stays in control**: A staff member can check the session and intervene when the experience needs it. (Agents & Operations) This is an architectural illustration, not a filmed installation. It shows how the parts are designed to work together; projects use only what they need. ## Four parts of the platform Each project uses the components it needs. The integrations depend on the experience. Farwhen is the working name for a modular architecture connecting story intelligence, real-time worlds, AI agents, and physical environments. We reuse these components across projects. Open-source modules can also be adopted individually. ### Worlds & Characters Stories that grow with the audience. Story intelligence turns film, video, and written material into a graph of scenes, characters, events, and relationships. Characters keep their identity across a whole story, remember what happened, and respond in conversation. Interactive stories and narrative agents read from the same graph. - What it covers: Story intelligence, Persistent characters, Narrative state, World knowledge, Video understanding, Interactive storytelling - Systems: [Story Engine](https://dreamawakelabs.com/technology#story-engine): Our story and world intelligence: it understands film and video, models stories and characters as a graph, and runs narrative agents over that graph. - Used in: [Story Intelligence for Escape.ai](https://dreamawakelabs.com/work/escape-ai-story-intelligence.md), [God’s Eye View at The Rectangle](https://dreamawakelabs.com/work/gods-eye-view-rectangle.md) ### Interactive Experiences Games made for the places we play. Unity and Unreal foundations for XR, mixed reality, procedural worlds, multiplayer, physics, and gameplay. The same base supports a headset game, a room-scale installation, and an interactive application on a screen. - What it covers: Unity and Unreal, XR and mixed reality, Procedural worlds, Multiplayer, Physics and gameplay, Interactive applications - Systems: [XR foundation](https://dreamawakelabs.com/technology#xr-foundation): Our Unity starting point for mixed-reality apps that run on Meta Quest 3 and Android XR headsets from one codebase; [GameplayFramework](https://dreamawakelabs.com/technology#gameplay-framework): An open-source Unreal Engine plugin that gives a game a ready-made, multiplayer-safe base: abilities, inventory, equipment, AI enemies, saving, quests, and seeded procedural levels; [Spatial Unreal](https://dreamawakelabs.com/technology#spatial-unreal): An open-source Unreal Engine plugin for Unreal games and installations that take part in DASP experiences. - Used in: [Puttscape: Mixed-Reality Mini-Golf](https://dreamawakelabs.com/work/mixed-reality-mini-golf.md), [GameplayFramework for Unreal Engine](https://dreamawakelabs.com/work/gameplay-framework.md), [GlitchWalker](https://dreamawakelabs.com/work/glitchwalker.md), [Collectors Edition](https://dreamawakelabs.com/work/collectors-edition.md) ### Spatial Orchestration The room becomes part of the experience. DASP defines how game engines, story services, and room devices can exchange events. Spatial Venue coordinates headset sessions; lighting, projection, and robot control are integration targets. - What it covers: Device coordination, Open protocol, Shared sessions, Show control, Operator status, Simulation before the room - Systems: [Spatial OS](https://dreamawakelabs.com/technology#spatial-os): A hub that connects story and AI services, game engines, and room devices into one experience through DASP; [DASP](https://dreamawakelabs.com/technology#dasp): An open protocol that lets the parts of an experience (engines, headsets, story and AI services, show systems) tell each other what happened and what should happen next; [Spatial Venue](https://dreamawakelabs.com/technology#venue-runtime): A venue runtime that keeps the headsets in one room in a single shared session and gives floor staff a live status page and a command-line control tool; [Venue devices](https://dreamawakelabs.com/technology#venue-devices): The hardware in the room, discovered and monitored by Spatial Venue: XR headsets, with lighting, audio, displays, and robots addressed as DASP device nodes. - Used in: [Puttscape: Mixed-Reality Mini-Golf](https://dreamawakelabs.com/work/mixed-reality-mini-golf.md), [Agent City](https://dreamawakelabs.com/work/agent-city.md), [God’s Eye View at The Rectangle](https://dreamawakelabs.com/work/gods-eye-view-rectangle.md), [Spatial Venue](https://dreamawakelabs.com/work/venue-runtime.md) ### Agents & Operations Tools for people and agents. Agent Forge exposes application actions to AI tools, with approvals for consequential changes. Atlas and Agent City help operate and observe workflows. - What it covers: AI orchestration, Typed tools, Operator controls, Workflows, Human approval, Visualization - Systems: [Agent Forge](https://dreamawakelabs.com/technology#agent-forge): An open-source library that lets a web app offer its real actions to AI agents as safe, typed tools, with consequential actions flagged; [Atlas](https://dreamawakelabs.com/technology#atlas): Our own operations agents: they turn meeting notes and chat threads into decisions and tasks, and wait for a person before anything is sent outside; [Agent City](https://dreamawakelabs.com/technology#agent-city): A live 3D city that shows what your AI agents are doing, including each step that is waiting for a person to approve it; [Agent-native web apps](https://dreamawakelabs.com/technology#web-apps): Web apps whose real actions are open to AI agents as typed tools, so an agent can help a visitor without scraping the page. - Used in: [The Rectangle](https://dreamawakelabs.com/work/the-rectangle.md), [Agent City](https://dreamawakelabs.com/work/agent-city.md), [Agent Forge](https://dreamawakelabs.com/work/agent-forge.md), [WebMCP Replay Room](https://dreamawakelabs.com/work/webmcp-replay-room.md) ### Under every group Synthetic data, simulation, digital humans, and development tooling support the groups above rather than belonging to one of them. - Synthetic data - Simulation - Digital humans - Development tooling Reusable infrastructure for intelligent, interactive experiences across screens, headsets, and physical spaces. ## Technical architecture For developers: the architecture map, reusable libraries, APIs, project integrations, and open-source references. ## Applications ### Agent-native web apps Web apps whose real actions are open to AI agents as typed tools, so an agent can help a visitor without scraping the page. What it does: - Server-rendered apps in TypeScript, Vue, and Nuxt, with the site’s own actions registered as WebMCP tools through Agent Forge. - Inquiry pipelines with AI screening, routing, CRM sync, and a person’s approval before anything with consequences. - Agent City and Replay Room are public demos; The Rectangle site and this site are built the same way. Links: - [Replay Room demo](https://webmcp-replay-room-seven.vercel.app) (Public demo) Used by: - [The Rectangle](https://dreamawakelabs.com/work/the-rectangle.md): The Rectangle site is one of our agent-native web apps. - [WebMCP Replay Room](https://dreamawakelabs.com/work/webmcp-replay-room.md): One of our agent-native web apps. ### Agent City A live 3D city that shows what your AI agents are doing, including each step that is waiting for a person to approve it. What it does: - Each district is a stage of work: inbox, research, prepare, approval, action, complete. - Consequential actions wait in an approval hall until a person decides. - Click an agent to see what it may do, what needs approval, and what it cannot do. - A privacy layer strips names, contact details, prompts, and secrets before anything is shown, and fails closed. - Agent Forge, DASP, and several agent runtimes feed one event model through adapters. - The public demo runs eight synthetic scenarios with scripted approvals. Links: - [Open the demo](https://agent-city-demo.vercel.app/?scenario=rectangle-business) (Public demo) Used by: - [Agent City](https://dreamawakelabs.com/work/agent-city.md): This is Agent City itself. ## Agent OS ### Atlas Our own operations agents: they turn meeting notes and chat threads into decisions and tasks, and wait for a person before anything is sent outside. What it does: - Five agents that turn meeting transcripts and Slack threads into decisions, action items, GitHub issues, and daily and weekly summaries. - Every tool call passes a policy check: reads are allowed, writes are limited per agent, and outbound or destructive actions wait for human approval in Slack. - A tamper-evident audit log of every call, stored in Postgres. - Per-agent model budgets, with provider fallback that ends at local models. - Runs on our own hardware, and is available to other agents as an MCP server. ### Spatial OS A hub that connects story and AI services, game engines, and room devices into one experience through DASP. What it does: - AI decides what should happen, local systems carry it out, and the results are the record. - A message router with capability discovery and fail-closed routing, so a node only receives what it declared it can handle. - A simulator seam for running an experience against fake devices before the real room. Used by: - [God’s Eye View at The Rectangle](https://dreamawakelabs.com/work/gods-eye-view-rectangle.md): Proposed coordination between the globe, narrative services, and devices. ## Spatial runtime ### XR foundation Our Unity starting point for mixed-reality apps that run on Meta Quest 3 and Android XR headsets from one codebase. What it does: - Reusable packages for hands and controllers, floor placement that persists across launches, and held tools that survive fast swings. - Per-headset build profiles and verification gates, with an editor test suite. - A sample racquet-and-ball game against a real wall. Used by: - [Puttscape: Mixed-Reality Mini-Golf](https://dreamawakelabs.com/work/mixed-reality-mini-golf.md): The Unity AR Foundation base supports the game on Quest and Android XR. ### Spatial Unreal (Open source) An open-source Unreal Engine plugin for Unreal games and installations that take part in DASP experiences. Source code: https://github.com/DreamAwakeLabs/spatial-unreal What it does: - A plugin skeleton with two runtime modules that compiles on Unreal Engine 5. - Architecture and XR integration design documents: a DASP connection with entity and capability registries, idempotent actions, and event publishing. - A design for an installation layer (rooms, zones, sensors, projection) and an optional GameplayFramework adapter. Links: - [Source on GitHub](https://github.com/DreamAwakeLabs/spatial-unreal) (Source code) ### GameplayFramework (Open source) An open-source Unreal Engine plugin that gives a game a ready-made, multiplayer-safe base: abilities, inventory, equipment, AI enemies, saving, quests, and seeded procedural levels. Source code: https://github.com/AyaDogGames/GameplayFramework What it does: - Data-driven abilities and attributes, and a server-authoritative inventory where every item keeps its own identity. - Equipment, item wear and repair, a hotbar and inventory UI, AI characters and spawning, saving, and quests. - Seeded procedural dungeon placement, verified with three players in multiplayer. - GlitchWalker and Collectors Edition are built on it. Apache-2.0. Links: - [Source on GitHub](https://github.com/AyaDogGames/GameplayFramework) (Source code) Used by: - [GameplayFramework for Unreal Engine](https://dreamawakelabs.com/work/gameplay-framework.md): This is the plugin itself. - [GlitchWalker](https://dreamawakelabs.com/work/glitchwalker.md): The game is a thin layer of data and subclasses on the plugin. - [Collectors Edition](https://dreamawakelabs.com/work/collectors-edition.md): Inventory, attributes, spawning, saving, and UI come from the plugin. ### Spatial Venue A venue runtime that keeps the headsets in one room in a single shared session and gives floor staff a live status page and a command-line control tool. What it does: - Server-authoritative group sessions, streamed to each headset 30 times a second over UDP, with reconnects that put a dropped headset back in place. - Operator tooling: a live status page on the venue network and an operator guide served with it. - A DASP control socket with 22 venue capabilities. Every frame in and out is checked against the DASP schemas. - A command-line tool with JSON output, built for people and for LLM agents. - A Unity client package over a native library, and load tests that run 48 players across 8 sessions on about a tenth of a CPU core. Used by: - [Puttscape: Mixed-Reality Mini-Golf](https://dreamawakelabs.com/work/mixed-reality-mini-golf.md): The venue client package connects the game to the Spatial Venue server. - [Spatial Venue](https://dreamawakelabs.com/work/venue-runtime.md): This is the server itself. ## Intelligence ### Story Engine Our story and world intelligence: it understands film and video, models stories and characters as a graph, and runs narrative agents over that graph. What it does: - Video understanding: films split into shots and indexed by captions, entities, events, dialogue, and narrative acts. - Semantic search across video, with answers that cite the exact time codes. - A typed story graph of scenes, beats, character arcs, conflicts, themes, and relationships. Every AI pass records where its facts came from and can be reverted. - Character identity grounding: the same character recognized across a whole film, with a persona and a voice for conversation. - Retrieval agents over the graph, orchestration of conversations among several characters, and interactive stories, generated arcs, and multi-video universes built on top. - A command-line tool and an MCP server, so agents can build and query stories without a user interface. Used by: - [Story Intelligence for Escape.ai](https://dreamawakelabs.com/work/escape-ai-story-intelligence.md): The story intelligence capability: video understanding, the story graph, search, and character agents. - [God’s Eye View at The Rectangle](https://dreamawakelabs.com/work/gods-eye-view-rectangle.md): Proposed analysis of selected public camera feeds using video understanding. ## Capabilities ### Agent Forge (Open source) An open-source library that lets a web app offer its real actions to AI agents as safe, typed tools, with consequential actions flagged. Source code: https://github.com/DreamAwakeLabs/agent-forge What it does: - Define each operation once: a description, an input schema, an effect class (read, reversible write, write, irreversible write), and a consequential flag. - A WebMCP adapter registers each operation as a tool that an in-browser agent can call. - Availability checks, cancellation, and execution events that carry metadata only, never input values. - Used by Agent City, Replay Room, and the Rectangle site. Apache-2.0. Links: - [Source on GitHub](https://github.com/DreamAwakeLabs/agent-forge) (Source code) Used by: - [The Rectangle](https://dreamawakelabs.com/work/the-rectangle.md): Agent tools on the website, with write tools flagged consequential. - [Agent City](https://dreamawakelabs.com/work/agent-city.md): Defines the demo’s five WebMCP tools and emits the execution events the city renders. - [Agent Forge](https://dreamawakelabs.com/work/agent-forge.md): This is the library itself. - [WebMCP Replay Room](https://dreamawakelabs.com/work/webmcp-replay-room.md): Six capabilities, three read and three reversible-write, registered over WebMCP. ### DASP (Open source) An open protocol that lets the parts of an experience (engines, headsets, story and AI services, show systems) tell each other what happened and what should happen next. Source code: https://github.com/DreamAwakeLabs/spatial-protocol What it does: - Messages for nodes, capabilities, events, intents, actions, state, and streams, with JSON Schemas for the envelope and core messages. - WebSocket transport, idempotent actions, and expiry checked before an action runs. - Spatial Venue speaks it as a node with 22 venue capabilities and validates every frame against the schemas. - Published as an open specification, so engines, headsets, and show systems can join the same experience. Links: - [Specification on GitHub](https://github.com/DreamAwakeLabs/spatial-protocol) (Source code) Used by: - [Puttscape: Mixed-Reality Mini-Golf](https://dreamawakelabs.com/work/mixed-reality-mini-golf.md): Automation reaches the venue server through its DASP control socket. - [Agent City](https://dreamawakelabs.com/work/agent-city.md): DASP envelopes are one of the event sources the city adapts. - [God’s Eye View at The Rectangle](https://dreamawakelabs.com/work/gods-eye-view-rectangle.md): Proposed protocol interface for navigation events and room-control commands. - [Spatial Venue](https://dreamawakelabs.com/work/venue-runtime.md): The server is a DASP node with 22 venue capabilities. ## Venue systems ### Venue devices The hardware in the room, discovered and monitored by Spatial Venue: XR headsets, with lighting, audio, displays, and robots addressed as DASP device nodes. What it does: - Headset discovery on the venue network by UDP broadcast and mDNS. - Headset telemetry on the status page: battery, tracking, heartbeat, and frame rate. - DASP device nodes, so lighting, audio, displays, and robots can be driven the same way as headsets. Used by: - [Puttscape: Mixed-Reality Mini-Golf](https://dreamawakelabs.com/work/mixed-reality-mini-golf.md): Headset discovery and telemetry for the venue floor. - [God’s Eye View at The Rectangle](https://dreamawakelabs.com/work/gods-eye-view-rectangle.md): Potential device endpoints for the room-scale concept. - [Spatial Venue](https://dreamawakelabs.com/work/venue-runtime.md): Headset discovery and telemetry on the status page. ## How the systems connect - Agent OS → Agent Forge: Browser agents call typed tools (WebMCP, in browser) - Agent Forge → Agent-native web apps: Each page registers its tools (WebMCP, in browser) - Agent Forge → Agent City: Demo tools, and execution events that feed the city (WebMCP, in browser; in-process events) - Agent-native web apps → Intelligence: Inquiries are screened by a hosted model (HTTP through AI Gateway) - Applications → Spatial Venue: Headset apps join sessions that staff run from the control tool (UDP game plane, 30 Hz) - Agent OS → Spatial Venue: Agents run the venue command-line tool (CLI with JSON output) - DASP → Spatial Venue: 22 venue capabilities, events, and state (DASP over WebSocket) - Spatial Venue → Venue devices: Headset discovery and telemetry (UDP broadcast and mDNS) - GameplayFramework → Applications: Base for Unreal games (In-engine plugin) - Agent Forge → DASP: DASP adapter (DASP over WebSocket) - Atlas → Agent City: Feed of Atlas work (Signed HTTP events) - Agent-native web apps → Agent City: Business events from the Rectangle site (Signed HTTP events) - Spatial Venue → Agent City: Venue feed (DASP events over HTTP) - Spatial OS → DASP: Routes messages between DASP nodes (DASP over WebSocket) - Story Engine → Spatial OS: Narrative provider adapter (Provider API to DASP) - Spatial Unreal → DASP: Unreal DASP client (DASP over WebSocket) - GameplayFramework → Spatial Unreal: Optional DASP adapter (In-engine plugin) - XR foundation → DASP: Unity DASP node package (DASP over WebSocket) - DASP → Venue devices: Lighting, audio, displays, and robots as device nodes (DASP over WebSocket) ## Perspectives ### Interactive experiences Stories, characters, and games that respond to the people playing them, in a headset or in a room: a story engine decides what happens, a protocol carries it, and a venue runtime keeps every headset in one session. - Systems: Story Engine, Spatial OS, DASP, Spatial Venue, XR foundation, GameplayFramework, Spatial Unreal, Venue devices - Projects: [God’s Eye View at The Rectangle](https://dreamawakelabs.com/work/gods-eye-view-rectangle.md), [Puttscape: Mixed-Reality Mini-Golf](https://dreamawakelabs.com/work/mixed-reality-mini-golf.md), [Spatial Venue](https://dreamawakelabs.com/work/venue-runtime.md), [GlitchWalker](https://dreamawakelabs.com/work/glitchwalker.md), [Story Intelligence for Escape.ai](https://dreamawakelabs.com/work/escape-ai-story-intelligence.md) ### Company intelligence AI agents that do real work for a business, such as answering inquiries and preparing invoices, and wait for a person before anything important goes out. You can watch what each agent does and approve the steps that matter. - Systems: Agent-native web apps, Atlas, Agent Forge, Agent City - Projects: [Agent City](https://dreamawakelabs.com/work/agent-city.md), [The Rectangle](https://dreamawakelabs.com/work/the-rectangle.md), [WebMCP Replay Room](https://dreamawakelabs.com/work/webmcp-replay-room.md)