FrameAssistant/docs/DESIGN.md

Design

Status: in progress, iteration 1, phase 2 (planning and design). High-level architecture first, then the details for the MVP. Requirements and their decisions (D1–D15): REQUIREMENTS.md. Design decisions start at D16.

Process and threads

• One process started by SteamVR does everything: HA connection, sensors, services, all overlays (D16).
• Several threads inside it for concurrent work. Which work runs in which thread is still open.

Layers

Three layers; dependencies only go downward (D17).

  Presentation      QML (view and controller)
       ▼
  Business logic    entities, services, managers
       ▼
  I/O               HA connection (MQTT), SteamVR, system (network, audio)

• Qt Quick already enforces separating presentation from logic.
• The overlay platform (vrdashboard_core) spans two layers: overlays, rendering and input are presentation infrastructure; SteamVR access for sensors and actions is I/O.

HA channels

• MQTT is the only channel for everything device-related: entities, services, availability (D24).
• MQTT can't show HA as a whole (other entities' states, service calls for them), which the dashboard (DSH-1) needs. The dashboard gets its own channel, separate from MQTT:
  • Q2a, HA web UI in Qt WebEngine: the browser talks to HA itself (HTTP/WebSocket, HA's own login); config: HA URL.
  • Q2b, native QML dashboard: HA's WebSocket API with a token (Q3) as another I/O component; config: HA URL and token.
• Both fit the three layers. The MVP contains the web UI test (DSH-1, D28), which also serves as the spike on WebEngine in the off-screen renderer; Q2/Q3 are decided for DSH-2.
• Known DSH-1 risks: WebEngine must be initialised before the QGuiApplication (created by the platform's base constructor), so the platform needs a small change; WebEngine with our off-screen renderer, especially Vulkan, is untested; text fields inside the page probably don't open the SteamVR keyboard (fallback: log in once in a desktop mode).

Start-up and modes

• Start-up hook (D21): the platform (DashboardApplication in vrdashboard_core) gets a third hook, e.g. virtual bool prepare(QString* error). exec() calls it once mode and app key are known, before the QML engine, the surfaces and VR_Init. On false, exec() ends with exit code 1 and reports the error through the existing path (log and stderr). The default returns true, so other apps built on the template don't change. FrameAssistant loads its configuration there and decides from mode() whether it needs it (CFG-2).

• Modes (D22):

  Mode	Configuration + HA connection	SteamVR
  overlay (default)	yes	required (overlay app)
  --desktop	yes	connects as background app if SteamVR runs; otherwise VR entities are unavailable (SEN-15)
  --desktop-novr	yes	never
  --snapshot, --benchmark	no	no

  • --desktop on the Frame over RDP shows the live UI with real data without wearing the headset. Stop the SteamVR-started instance by hand first: two instances with the same device name would knock each other off the broker.
  • --config <path> loads another configuration file instead of ~/.config/frameassistant/config.json, e.g. on a PC with its own device name (CON-5).

Overlays

Two overlay kinds in the platform (D25):

Overlay	Kind	Visibility	MVP
Main tab (UI-1, later NOT-7, do-not-disturb)	dashboard tab	always there; SteamVR shows/hides it	✓
Notifications	world overlay	the app (notification manager) shows/hides it	✓
Status tab (SRV-6)	dashboard tab	exists only while cards exist (D9)	✗
HA dashboard (DSH-1, web UI test)	dashboard tab	always there	✓

• World overlays (MVP): a new overlay kind next to the dashboard tab (CLAUDE.md, "Planned extension"). The surface gets a kind and a placement; renders only while visible (NFR-3). In the desktop modes (D22) a world overlay is a plain window the app shows and hides.
• Premultiplied alpha: Qt Quick renders premultiplied alpha; SteamVR assumes straight alpha unless VROverlayFlags_IsPremultiplied is set, which the platform doesn't do yet. Fully opaque/transparent pixels are right, semi-transparent ones (antialiasing, shadows) would be too dark. Set the flag in the platform; verify on the device.
• Runtime surfaces (later, SRV-6): creating a surface at runtime repeats the start-up steps for one surface. Destroying one while SteamVR runs is the risk: each surface has its own backend (with Vulkan its own VkDevice), and SteamVR's client keeps objects on devices it got textures from (the reason for the teardown order). Needs a spike on the Frame; ways out: one device shared by all surfaces, or reuse instead of destroy.
• Interactive world overlays (later, NOT-5): the laser hits the whole overlay rectangle, including transparent parts.

Notifications (D26, D27)

• One world overlay for all notifications (D26). QML lays out the visible notifications as cards inside it; transparent areas around them are invisible. No runtime surfaces needed, one renderer and texture (NFR-3). One placement for all; per-notification placement is not required (NOT-2 makes placement configurable, not per notification) and could later become one overlay per placement.
• MVP placement: spawn in front, then stay (D27): about 1 m away, slightly below the line of sight, positioned once when it appears. No continuous tracking. Lazy follow and hand placement come with NOT-2.
• Placement techniques:
  • head-locked: SteamVR keeps the overlay relative to the HMD; runs in the compositor, smooth.
  • spawn in front, then stay: one absolute transform when shown.
  • lazy follow: the app updates the transform continuously from the head pose; CPU cost and jitter if updated too rarely.
• UI guidelines from research: Appendix A.

Object classes

The app is HA-centric, so its building blocks follow HA's concepts (D18). Objects talk to each other through Qt signals and slots (Qt's observer pattern). Senders don't know their receivers, and signals across threads are delivered through the receiver's event loop.

Class	Role	MVP
HA entity	One HA entity, both directions. Entity → HA: state and JSON attributes. HA → entity: commands; HA's MQTT integration turns its own services into them (e.g. button.press, switch.turn_on, number.set_value, notify.send_message). Observes value signals of other objects and publishes them; turns MQTT commands into signals other objects act on. Native entity types are preferred wherever their meaning fits (D23).	✓
HA service	Fallback where no entity type fits (D23): a JSON command on an own MQTT command topic (HA side: the standard mqtt.publish action), e.g. rich notifications or vibration with parameters. Results go back asynchronously, e.g. through an event entity. How the user wraps it in HA (scripts, automations, pyscript) is their choice; FrameAssistant only requires stock HA with MQTT (NFR-6).	depends on the NOT-1 choice
HA device	Owns the MQTT connection, discovery and availability, and owns all entity and service objects. One per headset (CON-5).	✓
I/O wrappers	Data sources and actuators; know nothing about HA. One wrapper object per OpenVR interface used (e.g. IVRSystem, IVRApplications, IVRChaperone, IVRSettings). System wrappers (network, audio) later.	OpenVR ✓, system ✗
Overlays	View and controller of MVC (QML), as dashboard tabs or world overlays. Get their data (the model) from other objects.	✓
Managers	One object per stateful feature that manages and coordinates it, using entities, services and overlays. Notification manager (MVP); status card manager (SRV-6/7) later.	notifications ✓
HA web UI	Encapsulates Qt WebEngine for the dashboard tab (DSH-1; Q2 for DSH-2).	✓ (DSH-1)
Configuration	Settings from the JSON file, loaded once by the app as an immutable value. Each component gets the part it needs through its constructor (D19).	✓
App	The managing object: owns all of the above as components and wires them together.	✓

Entities and services (D23)

• One class per HA entity type on a common base. The base does the discovery part, topics, availability and throttling, so SEN-14, SEN-15 and NFR-7 are handled in one place.

• A concrete entity is an instance with a description (id, name, device class, unit, throttle interval), not a class of its own.

• Our actions map to native types where the meaning fits:

  Action	Type	HA service
  Sensors (SEN-*)	sensor, binary_sensor	—
  Notification (NOT-1)	notify	notify.send_message
  Screenshot, standby, quit app, show boundary (SRV-1/2/3)	button	button.press
  Do-not-disturb (SRV-5)	switch	switch.turn_on/off
  Volume (SRV-3)	number	number.set_value
  Launch app by key (SRV-2)	text	text.set_value
  Attention signal with duration/volume (SRV-1), to check	siren	siren.turn_on
  Feedback (NOT-5, NOT-10)	event	—

• Wiring: the app wires simple entities (I/O wrapper signal ↔ entity). Managers get their entities and services from the HA device and wire them internally. The HA device owns them all (D18).

• MQTT notify drops the title: HA's async_send_message() renders only message into the payload (homeassistant/components/mqtt/notify.py). NOT-1 needs title and text; options (MVP detail): first line is the title, JSON in the message, or an own service.

Notes for later:

• OpenVR delivers all events through one queue; one dispatcher (today VrRuntime) hands them to the wrappers.
• OpenVR is not documented as thread-safe (relevant for the thread decision).

Open

1. High-level points 3–7 (see PROGRESS.md, "Next steps").

Appendix A: notification UI in VR (research)

Collected 2026-10-05 for the notification design.

Studies:

• Rzayev et al., CHI PLAY 2019, 24 participants, tasks incl. gaming; placements head-up display (fixed in view), on-body, floating, in-situ (embedded in the scene). On-body and floating: best understandability, usability and presence. HUD: most intrusive and disturbing; only for urgent notifications.
• Hsieh et al., CHI 2020, 40 participants; displays on the HMD, on the controller, movable panel:
  • Users dislike notifications blocking their focus region (HMD: inflexible location; controller: blocks the work area). Recommendation: let users choose the location from a set of peripheral positions.
  • The more engaged the user, the more disruptive notifications feel. Then rather a small peripheral indicator (e.g. unread count), content later.
  • Don't reuse a modality the VR content already uses (e.g. game vibration hides notification vibration). Urgent: more obtrusive means such as audio.
  • Display time in the study was 12 s; longer is likely more disruptive.
• NotifiVR, Ghosh et al. 2018: haptic alone was the least effective modality.

Platforms and tools:

• Meta Quest design guidelines: avoid head-locked UI; anchor in space or "lazy follow" (follows the head delayed and smoothed); about 1 m away, slightly below the line of sight. Users can set where notifications appear.
• visionOS: content to read goes centred and slightly below the line of sight; periphery for context, centre for focus.
• XSOverlay (common SteamVR overlay tool): popups at a configurable offset from the head, smoothly following; xsoverlay-tweak adds a "leash" mode (stays until the head turns too far away).

Consequences for FrameAssistant:

1. Placement: not rigidly head-locked. Lazy follow/leash, spawn in front then stay, and hand are well supported (matches NOT-2). Head-locked at most for critical notifications (NOT-8).
2. Position: about 1 m, below the line of sight or towards the periphery, not centred; offset configurable.
3. Content: compact (title, short text), fade in/out, timeout around 10 s; longer content in the main tab's list (NOT-7).
4. Several at once: few visible, stacked, newest in front (fits D26).
5. Modalities: sound as the main signal for important notifications (NOT-4/8), vibration only as a supplement.

Decisions

#	Date	Decision	Reason
D16	2026-10-05	One process started by SteamVR does everything; several threads inside it for concurrent work	Meets NFR-2 without extra work, no IPC; the platform already supports several surfaces per process
D17	2026-10-05	Three layers: presentation (QML), business logic, I/O. The overlay platform spans presentation (overlays, rendering, input) and I/O (SteamVR access)	Simple and enough for the start; Qt Quick already enforces separating presentation from logic
D18	2026-10-05	HA-centric object classes: HA entity, HA service, HA device, I/O wrappers, overlays, managers, HA web UI, configuration, app; communication through signals and slots	The app is HA-centric; signals/slots decouple objects and work across threads
D19	2026-10-05	Configuration is injected: the app loads it once as an immutable value and passes each component the part it needs; no singleton	Explicit dependencies, trivial tests with own configs, thread-safe; the app creates all components anyway
D21	2026-10-05	Start-up hook in the platform that may fail (prepare()), before the engine, surfaces and VR_Init; failure → exit code 1	CFG-2 needs a clean abort; loading()/started() can't fail; backward compatible
D22	2026-10-05	Overlay and both desktop modes load the configuration and connect to HA; --desktop also connects to SteamVR as a background app if it runs; --desktop-novr never does; --snapshot/--benchmark stay without configuration; new option --config <path>	Live development over RDP on the Frame with real data; PC development without SteamVR under its own device name
D23	2026-10-05	One class per HA entity type on a common base; concrete entities are described instances. Native entity types (and the HA services the MQTT integration provides for them) are preferred; an own JSON command service only as fallback	Mirrors HA's model; discovery, availability and throttling in one place; actions are visible and selectable in HA without mqtt.publish
D24	2026-10-05	MQTT for everything device-related; the dashboard gets its own channel to HA, chosen with Q2/Q3	MQTT can't provide HA as a whole; both dashboard options fit the layers without affecting the MVP
D25	2026-10-05	Two overlay kinds in the platform: dashboard tabs and world overlays (MVP: one placement, not interactive). Set VROverlayFlags_IsPremultiplied. Runtime surfaces and interactive world overlays later	Notifications need world overlays; Qt Quick renders premultiplied alpha
D26	2026-10-05	One world overlay for all notifications; QML lays out the cards	No runtime surfaces, fewer resources; per-notification placement isn't required
D27	2026-10-05	MVP notification placement: spawn in front (about 1 m, slightly below the line of sight), then stay	Research favours non-head-locked placement; no continuous tracking needed