Skip to content

MUMOSA Crisis Response

A graduate client project for the Army Research Laboratory. Our team redesigned ARL's MUMOSA dashboard so analysts can check AI answers against reports, images, timelines and reconstructed scenes after a crisis. I led the research and the spatial prototype, and built the Unreal proof of concept.

My Unreal proof of concept: an evidence marker anchored in the reconstructed scene.
My role
Led research and spatial prototyping; contributed to the dashboard information architecture
Team
Three-person graduate team: Georgi Tsvetanski, Kelly Ehrlich and Kamilah S.
Client
DEVCOM Army Research Laboratory (ARL), via a University of Baltimore interaction design course
Delivered
Literature review, paper prototype, Axure dashboard, Unreal proof of concept, client report
Jump to a section

The brief

MUMOSA (MUlti-MOdal Situation Awareness) is ARL's research dashboard for making sense of a crisis from text reports, images, event structure and 3D reconstructions. Our course team was asked to recommend how its interaction design should evolve.

Analysts ask a question, check the supporting evidence, place events on a timeline, and open a reconstructed scene when location matters. The same flow should later accept new evidence while an incident is still unfolding.

  • Short term

    Help investigators and instructors compare reports, images, events and timelines after a crisis.

  • Long term

    Extend the same structure to incoming evidence, changing timelines, hazard alerts and role-specific views.

  • Our response

    Put questions, sources, conflicts, timelines and spatial context in one evidence workspace.
ARL's published MUMOSA dashboard, our starting point. Figure from Lukin et al., “MUMOSA, Interactive Dashboard for MUlti-MOdal Situation Awareness” (FuturED workshop, 2024).

What I built in Unreal

I turned the paper interaction model into a PC-first Unreal proof of concept. Analysts select evidence in the scene, see its confidence and timeline context, and follow it back to the source.

  • PC first, VR ready

    Keyboard-and-mouse walkthrough first; OpenXR follows once the core interaction is proven.

  • Evidence markers

    Reusable C++ markers store the claim, confidence, timeline state, discrepancies and linked sources.

  • Source-grounded panel

    Selecting a marker opens its interpretation, confidence, timeline context and direct source links.

  • Hazards and timeline states

    Toggleable danger, smoke and uncertainty volumes. A four-state timeline (Before → Derailment → Smoke Spread → Response) changes which markers and hazards show.

  • Dashboard handoff

    The scene opens with a banner naming the dashboard question and focus object it was launched from. "Return to Dashboard" logs a mock payload of findings.
Selecting a marker opens its interpretation while the scene stays visible.
Evidence selection links a spatial marker to its source trail.
How the full pipeline would work (planned, not built)

The proof of concept covers the review layer. This is how a real deployment would get evidence into it: drones, robots and body cameras capture the scene, AI turns it into grounded evidence, and Unreal shows it.

  • Fidelity tiers

    Active response needs answers in minutes, so low-fidelity Gaussian splats (about 5–15 min) show danger zones fast. Investigation uses full photogrammetry (about 30 min to 2+ hours).

  • Raw scans in Nanite

    Nanite renders messy photogrammetry at full detail without retopology. The prototype uses photoscanned Megascans debris to show the quality such a pipeline would give.

  • Query to scene

    A dashboard question goes to the AI, which picks relevant markers and highlights them in Unreal. The prototype mocks this with structured JSON in the shape a real query would return.

Paper prototype

Before any software, I made a low-fidelity VR prototype: a paper headset window, annotated controllers, a sketched crisis site, and evidence notes that appear when the investigator asks to verify a claim. It tested the questions that mattered first. Can people orient themselves? Can they check an AI summary against evidence? Does it stay simple?

The interaction model stayed modest on purpose:

  • Teleport between scene zones instead of navigating menus.
  • A visible "show source" action, so every AI summary leads back to its evidence.
  • LiDAR measurement, zoom and alternate views only for deeper inspection.
  • VR is a review mode paired with the web dashboard, not a replacement for it.
A paper visor frames the crisis scene; hand-drawn controllers map teleporting, evidence reveal and inspection.

  • Scene first

    A panoramic sketch of the site so investigators understand place and hazards before any interface.

  • Evidence in context

    Sticky-note overlays stand in for AI summaries, timestamps and next actions, anchored to the spot being inspected.

  • Testable controls

    Annotated paper controllers to test teleporting, source reveal, LiDAR measurement and zoom/select before writing software.
Three evidence notes anchored to the scene, each a grounded finding with a path back to its source.
The active-fire note sits beside the burning structure instead of in a separate panel.
Scene sketches
Base panorama: street, damaged area, vehicle, civilian, vegetation and active fire.
Low-visibility variant to test orientation when the scene is harder to read.

The team's dashboard redesign

Spatial review only works as one mode of a larger investigation, so the team also reworked the flat-screen dashboard into a five-screen question-and-answer workspace. My part was the information architecture, the Axure build spec, the design system and the evidence-review interaction model.

  • Incident + question entry

    Replaced the role picker with natural-language questions: pick an incident and timeframe, then ask from suggested prompts.

  • Answer workspace

    The AI answer with ranked confidence, text evidence, visual evidence and source metadata, all on one screen.

  • Evidence comparison

    Side-by-side text and image sources with discrepancy callouts, source chains and investigator notes.

  • Timeline + event map

    The most revised screen: a timeline with event nodes, a schema-backed relationship map with a legend, and a detail panel for participants, sources and conflicts.

  • Simulation evidence

    The spatial review screen, renamed from "VR Scene Review" to match ARL's paper: 3D reconstruction with question-driven annotations and linked sources.
Key team decisions and the planning board

  • Information architecture reset

    From a generic event overview to a task flow: Ask → Answer → Compare evidence → Timeline + event map → Simulation evidence.

  • Source-grounded AI

    No AI answer without source cues: confidence badges, ranked evidence scores and discrepancy warnings on every answer card.

  • Timeline critique

    The first timeline had no legend and unclear nodes. The revision added a legend, source-linked nodes, participant roles and a detail panel.

  • Investigator notes

    My teammates carried forward note-taking and saved annotations so findings survive deeper review.
Team planning board from the low-fidelity phase: dashboard ideas and the VR paper-prototype strip, developed in parallel.
The VR cluster of the board: cognitive goals, risks (hardware cost, motion sickness, data streaming) and mitigations.

Research

My literature review grounded the redesign. It covers who uses the system, situational-awareness and cognitive-load heuristics, and multimodal crisis-response design. Three findings shaped the direction:

  • Cognitive load first

    Responders and investigators already juggle too many information streams. The interface should reduce fragmentation and noise.

  • AI claims need sources

    Every AI summary should link to the report, image, event or scene evidence behind it.

  • Post-crisis review first

    Reconstruction and training are the clearest near-term uses for documents, event relationships and 3D scene review.

Outcome

A literature review and a paper sketch became a complete client package. The through-line is source grounding: whether someone reads an AI answer, compares images, rebuilds a timeline or steps into a 3D scene, the system shows what evidence supports the claim and where uncertainty remains.

  • Literature review (mine): situational awareness, cognitive load and multimodal crisis-response heuristics.
  • Low-fidelity team prototype: dashboard wireframes, my VR paper prototype, and a usability-test script.
  • Revised five-screen dashboard and Axure prototype: I contributed the Axure build spec, design system and evidence-review interaction model.
  • Unreal spatial review proof of concept (my lane): evidence markers, source-grounded UI, hazard context and a mock dashboard handoff.
  • Client report and final presentation: research, prototypes, testing approach, lessons and recommendations.
The usability test we planned

The low-fidelity package included a usability script for an incident-analysis scenario:

  • Task 1: Orientation

    Explore the dashboard freely, then explain what you would do first to understand the event.

  • Task 2: Timeline

    Find when the event happened and reconstruct the sequence leading up to it.

  • Task 3: Evidence

    Locate supporting evidence and describe how you would inspect the scene more closely.