Graduate Client Project · Army Research Laboratory · Multimodal Situation Awareness
MUMOSA Situation Awareness Dashboard
MUMOSA is a client-facing dashboard concept for making sense of complex incidents from multiple evidence streams: reports, images, extracted events, timelines, and spatial reconstructions. Our design work focused on what ARL needs now — post-crisis analysis and training — while keeping the interaction model credible for longer-term real-time situational awareness.
Designing for the system MUMOSA needs now, without boxing in where it goes next.
The near-term product has to help analysts and responders reconstruct what happened after an incident and use that evidence for training. The longer-term vision is more ambitious: support real-time understanding as new text, image, event, and spatial data arrives. My work turns that gap into a concrete interaction model: ask a question, inspect grounded evidence, understand the event over time, and step into a spatial review when the physical scene matters.
- - Short term: post-crisis investigation, lessons-learned review, and responder training.
- - Long term: real-time situational awareness with dynamic timelines, hazard cues, and guided next actions.
- - My lane: source-grounded spatial review in Unreal, connected back to the dashboard workflow.
Role
Research, paper-prototype, and spatial-simulation lead for a source-grounded MUMOSA redesign
Team
Three-person graduate team: Georgi Tsvetanski, Kelly Ehrlich, and Kamilah S.
Client Need
Short term: post-crisis analysis and responder training. Long term: real-time situational awareness, hazard cues, and guided next-step reasoning.
Artifacts
Client report, paper prototype, Axure dashboard package, and Unreal VR proof of concept.
Client Needs And Design Response
We treated ARL's problem as a sensemaking problem, not a dashboard-decoration problem. The user needs to move from scattered fragments to a defensible understanding of what happened, when it happened, and which evidence supports or contradicts that explanation.
Short-Term Use
Support investigators, analysts, and emergency-response instructors after a crisis by making fragmented reports, images, events, and timelines easier to compare and teach from.
Long-Term Direction
Leave room for real-time situational awareness: live evidence intake, dynamic timelines, hazard highlighting, role-specific views, and AI assistance that stays tied to sources.
Design Response
Treat MUMOSA as an evidence workspace first. The dashboard organizes questions, sources, conflicts, timelines, and spatial context before asking users to trust an AI summary.
What MUMOSA Is
MUMOSA is a multimodal situational-awareness dashboard. The core idea is to stop treating crisis evidence as separate silos and instead connect reports, images, extracted events, schema graphs, and 3D or simulation views inside one interface that can support investigation, training, and, later, potentially real-time response.
What makes it interesting to me is that it sits directly in the space I care about most: human factors, high-stakes information flow, and spatial interfaces that help users understand a scene rather than only read about it.
What I Owned
I am not presenting this as if I built the entire MUMOSA platform myself. My contribution spanned the immersive review lane across every phase: the literature review that shaped the VR framing, the paper prototype that tested the spatial interaction model, contributions to the revised dashboard information architecture, and the Unreal Engine proof of concept that turns those ideas into a working spatial review direction.
The dashboard redesign work shown later came from the shared team process and is included here as context. The digital phase keeps that split clear: the flatscreen prototype lives in Axure with my colleagues, while I carried the spatial simulation lane forward in Unreal Engine.
Research Findings That Shaped The Direction
Cognitive Load Comes First
My lit review kept returning to the same problem: responders and investigators are already overloaded. The interface has to reduce fragmentation, not add another noisy control room.
Trust Needs Grounding
The strongest heuristic in the MUMOSA paper is still the right one for our coursework too: every AI summary needs a visible path back to the source evidence.
Resolve Phase Is The Best Fit
The most believable use case stayed the same through prototyping: post-crisis reconstruction and training, where schema graphs, documents, and 3D review become genuinely useful.
Finished Low-Fidelity VR Paper Prototype
The finished low-fidelity prototype translates the research into something concrete: a paper headset window, controller annotations, a sketched reconstruction of the crisis site, and evidence notes that appear in-scene when the investigator asks to verify a claim.
Instead of claiming a full VR build, I focused on the interaction questions that actually matter first. Can users orient themselves in the scene? Can AI summaries be checked against evidence? Does the interface support a resolve-phase workflow without burying the person in more complexity?
Scene-First Orientation
I started with a panoramic sketch of the site so investigators can understand place and hazard layout before chasing UI chrome.
Evidence In Context
The sticky-note overlays simulate AI summaries, timestamps, and next actions anchored directly to the place being inspected.
Low-Tech, Testable Controls
Annotated paper controllers let us test teleportation, source reveal, LiDAR measurement, and zoom/select behavior before building software.
Interaction Model
The controls are intentionally modest. I wanted the paper prototype to prove the interaction logic before promising any technical implementation. The model stays focused on navigation, evidence verification, and selective deep inspection.
- - Teleport between scene zones instead of forcing the user through menu-heavy navigation.
- - Use a visible "show source" action so AI summaries can always lead back to evidence.
- - Reserve LiDAR measurement, zoom, and alternate view controls for deeper inspection moments.
- - Keep VR as a review surface paired with the web dashboard, not a replacement for the broader system.
Evidence Grounding In The Scene
These notes are the core of the concept. They show how the interface could present AI-generated findings without asking users to trust floating summaries blindly. Each card has a timestamped claim, a quick interpretation, and an action prompt that leads back to source evidence.
Revised Dashboard: From Role Picker To Q/A Workspace
The original dashboard prototype started with a role-picker and presented a generic event overview. After aligning more tightly with the MUMOSA source paper, the team redesigned the flow around a structured investigative workflow. The result is a 5-screen Q/A workspace where every interaction has a clear purpose.
Incident + Question Entry
Replaced the old role-picker-first flow with a natural-language Q/A entry. Users select an incident and timeframe, then ask a grounded question from suggested prompts.
Q/A Results Workspace
The central screen shows the AI answer with ranked confidence, textual evidence panel, visual evidence panel, and source metadata — all visible without clicking away.
Evidence Comparison
Side-by-side textual and visual source review with discrepancy callouts, source chains, and investigator notes. Every AI claim is traceable back to source material.
Timeline + Event Map
The most heavily revised screen. A horizontal timeline with event nodes, a schema-backed event relationship map with legend, and a selected-node detail panel showing participants, sources, and conflicts.
Simulation Evidence
The spatial review page, relabeled from "VR Scene Review" to match the MUMOSA paper. Shows 3D reconstruction with question-driven annotation overlays and linked source panels.
Team Dashboard Direction
Even though my emphasis is the VR and spatial simulation lane, the full project was broader than that. The shared planning board tracked flatscreen improvements alongside my lane, and those dashboard ideas matter because spatial review only makes sense as one mode in a larger investigative workflow.
Information Architecture Reset
The flatscreen direction moved from a generic event overview dashboard to a structured Q/A workspace: Ask → Q/A Results → Compare Evidence → Timeline + Event Map → Simulation Evidence. Each screen has a clear investigative purpose.
Source-Grounded AI
The revised dashboard never shows an AI answer without attached source cues. Confidence badges, ranked evidence scores, and discrepancy warnings are built into every answer card.
Timeline/Schema Critique Response
The original timeline view had no legend, unclear node meaning, and hard-to-follow side panels. The revision adds a full legend, source-linked event nodes, participant roles (agent/causer, affected entity, location), and a selected-node detail panel.
Investigator Notes
My teammates also carried forward the note-taking and saved annotations concept so investigators can preserve findings during deeper review.
Design Brief Translation
Reviewing the design brief helped tighten the page narrative. The board makes it clear that the project is not only about adding VR, but about restructuring the whole experience around cognition, role, and investigation flow.
Role-Adaptive Information
The design brief reframed the dashboard around dynamic filtering and role-based views so investigators, responders, and coordinators can enter the same incident from different cognitive starting points.
Overview To Detail To Overview
One of the clearest patterns in the brief is hierarchical exploration: start broad, drill into specific evidence, then move back out to re-establish context.
Training Is Not Secondary
The board treats simulation-based learning, pattern recognition, and decision rehearsal as core outcomes, not side benefits layered on after the fact.
Evidence Capture Pipeline
The spatial review prototype is backed by a realistic data pipeline. Drones, robots, and body cameras capture the scene; AI processes it into grounded evidence; and Nanite renders raw photogrammetry without costly retopology. The result is an end-to-end workflow from crisis site to clickable spatial evidence.
Fidelity Tiers
Active response needs answers in minutes — low-fidelity Gaussian splatting (~5-15 min) shows danger zones and blocked routes immediately. Post-crisis investigation uses full photogrammetry (~30 min - 2+ hrs) for forensic-grade detail. The prototype proves the review layer; the processing speed is an engineering curve, not a research question.
Nanite & Raw Scans
Photogrammetry produces messy scans with holes and artifacts. Nanite renders the raw mesh at full detail — no retopology needed for static evidence review. Key objects (railcar, ignition zone) can get AI-assisted cleanup; everything else renders directly. The prototype uses photoscanned Megascans debris to demonstrate the visual quality the pipeline would produce.
Query & AI Integration
The user asks a natural-language question through the dashboard. The AI (local or API) queries the evidence store, determines relevant markers, highlights them in the Unreal scene via the MCP bridge, and populates the source panel. The prototype mocks this with structured JSON data — the same shape a real AI query would return — so the interaction model is proven regardless of backend.
Planned Usability Test
The paper-prototype package included a usability script. That matters because the low-fidelity phase was not just a sketch dump; it has a concrete evaluation plan for orientation, timeline understanding, and deeper evidence review.
Task 1: Initial Orientation
Participants first explore the dashboard freely, then explain what they would do first to understand the event. This tests whether overview information and entry points are discoverable.
Task 2: Timeline Reconstruction
The script asks users to find when the event happened and reconstruct the sequence leading up to it, focusing on timeline discoverability and information hierarchy.
Task 3: Evidence And Deeper Investigation
Participants are asked to locate supporting evidence and describe how they would inspect the scene more closely, which directly probes whether VR mode and deeper analysis tools feel legible.
My Lane Now: Unreal Engine Spatial Simulation
The VR paper prototype proved the interaction model, then I carried that direction into an early Unreal Engine proof of concept. The spatial review module is PC-first and designed around the same principles the paper prototype validated: source-grounded evidence, physical context for hazards and events, and a clear relationship to the dashboard.
PC-First, VR-Ready
The spatial module starts as a keyboard-and-mouse walkthrough using Unreal's First Person C++ template. VR/OpenXR support comes after the core interaction model is proven.
Evidence Marker System
A reusable C++ actor class stores marker ID, label, AI interpretation, confidence level, status, timeline event, discrepancy note, and linked source records. Clicking a marker selects it and opens the evidence panel.
Source-Grounded Panel
Every marker opens a UI panel showing the AI interpretation, linked sources, confidence badge, timeline context, and any discrepancy warnings. The "show source" action is the primary interaction, not an afterthought.
Hazard Overlay + Timeline States
Translucent danger/smoke/uncertainty volumes can be toggled on and off. A 4-state timeline controller (Before → Derailment → Smoke Spread → Response) changes which markers and hazards are visible.
Dashboard Handoff Mock
The simulation starts with a banner showing it was launched from the dashboard with a specific focus object and question. A "Return to Dashboard" button logs a mock payload of findings.
Reconstruction Pipeline Concept
From Photos To Spatial Evidence
The long-term pipeline uses drones, robots, and body cameras to collect overlapping visual data. Photogrammetry or Gaussian splatting reconstructs the physical scene. Vision-language models extract key events, hazards, and objects, which MUMOSA turns into clickable spatial markers with links back to source evidence. The Unreal prototype simulates this pipeline with realistic sample data.
Course Deliverables And Project Status
Completed
Literature Review
Authored research document grounding the redesign in situational awareness, cognitive load, and multimodal crisis-response heuristics.
Completed
Low-Fidelity Team Prototype
The low-fi package included dashboard wireframes, the VR paper prototype, and a usability-testing script for an incident-analysis scenario.
Completed
Revised Dashboard Spec + Axure Prototype
The team finalized the revised 5-screen Q/A workspace flow. I contributed the Axure build spec, design system, and evidence-review interaction model for the digital prototype.
Completed Proof Of Concept
Unreal Spatial Simulation (My Lane)
Built an early Unreal spatial review proof of concept with evidence markers, source-grounded UI behavior, hazard-oriented scene context, and a mock dashboard handoff model.
Completed
Client Report + Final Presentation
Documented the research, paper prototype, electronic prototype, testing approach, lessons learned, and recommendations in a client-facing final report.
Research Grounding
This is the authored research document behind my part of the project. It covers user groups, heuristics, multimodal crisis-response design, and the reasoning that eventually shaped the resolve-phase and VR framing.
The client paper still matters as context, but it is not my portfolio artifact. What belongs in this case study is the bridge from research into the finished low-fidelity prototype and the digital implementation that followed.
Earlier VR Direction Note
Reference Documents
These links are here for context and coursework documentation. The client paper is supporting reference, not presented as my authored portfolio work. The GitHub repository contains the full project working materials.
Outcome So Far
What started as a literature review and a paper VR sketch became a complete client-facing project package: a revised Q/A dashboard direction, an Axure electronic prototype, a final report, and an Unreal proof of concept for spatial evidence review. The strongest through-line is source grounding. Whether the user is reading an AI answer, comparing visual evidence, reconstructing a timeline, or stepping into a 3D scene, the system should make it clear what evidence supports the claim and where uncertainty still exists.