FrameAssistant/docs/DESIGN.md

# Design

Status: **in progress**, iteration 1, phase 2 (planning and design). High-level architecture
done; next: the details for the MVP. Requirements and their decisions (D1–D15):
[REQUIREMENTS.md](REQUIREMENTS.md). Design decisions start at D16.

## Process and threads

- **One process** started by SteamVR does everything: HA connection, sensors, services, all
  overlays (D16).
- Several threads inside it for concurrent work (D16). **A thread is added only where work can
  block** and would stall rendering and SteamVR events (D29):

  | Thread | Runs |
  |---|---|
  | main | Qt event loop, QML, off-screen rendering of all surfaces, SteamVR event polling and texture submission, **all OpenVR calls** (I/O wrappers), HA device and entities, managers |
  | MQTT | the MQTT client (libmosquitto): DNS lookup, TCP and TLS connect block for seconds when the broker is unreachable |

  - OpenVR stays in the main thread: it isn't documented as thread-safe, and all events come
    through one queue.
  - The boundary is three signals with value arguments (queued connections):

    ```
    main thread                               MQTT thread
    HA device ── publish(topic, payload) ──▶  MQTT client
    HA device ◀── messageReceived(topic, payload) ── MQTT client
    HA device ◀── connectionChanged(state, error) ── MQTT client
    ```

  - The MQTT client lives in its thread via `moveToThread()`; it creates its socket notifiers
    and timers there, after the move.
  - Not needed as own threads: WebEngine (own processes), Qt Multimedia and QtNetwork
    (asynchronous), later D-Bus and `QProcess` for network/audio (asynchronous). Rendering stays
    in the main thread as long as NFR-3 holds.
  - Rules: no direct calls on objects of another thread (signals or
    `QMetaObject::invokeMethod` with `Qt::QueuedConnection`); lambdas connected across threads
    always get a context object; no `BlockingQueuedConnection`; only values, no shared
    pointers, cross the boundary.

## Layers

Three layers; dependencies only go downward (D17).

```
  Presentation      QML (view and controller)
       ▼
  Business logic    entities, services, managers
       ▼
  I/O               HA connection (MQTT), SteamVR, system (network, audio)
```

- Qt Quick already enforces separating presentation from logic.
- The overlay platform (`vrdashboard_core`) spans two layers: overlays, rendering and input
  are presentation infrastructure; SteamVR access for sensors and actions is I/O.

## HA channels

- **MQTT is the only channel for everything device-related:** entities, services, availability
  (D24).
- MQTT can't show HA as a whole (other entities' states, service calls for them), which the
  dashboard (DSH-1) needs. The dashboard gets **its own channel, separate from MQTT**:
  - Q2a, HA web UI in Qt WebEngine: the browser talks to HA itself (HTTP/WebSocket, HA's own
    login); config: HA URL.
  - Q2b, native QML dashboard: HA's WebSocket API with a token (Q3) as another I/O component;
    config: HA URL and token.
- Both fit the three layers. The MVP contains the web UI test (DSH-1, D28), which also serves
  as the spike on WebEngine in the off-screen renderer; Q2/Q3 are decided for DSH-2.
- Known DSH-1 risks: WebEngine must be initialised before the `QGuiApplication` (created by the
  platform's base constructor), so the platform needs a small change; WebEngine with our
  off-screen renderer, especially Vulkan, is untested; text fields inside the page probably
  don't open the SteamVR keyboard (fallback: log in once in a desktop mode).

## Start-up and modes

- **Start-up hook (D21):** the platform (`DashboardApplication` in `vrdashboard_core`) gets a
  third hook, e.g. `virtual bool prepare(QString* error)`. `exec()` calls it once mode and app
  key are known, before the QML engine, the surfaces and `VR_Init`. On `false`, `exec()` ends
  with exit code 1 and reports the error through the existing path (log and stderr). The
  default returns `true`, so other apps built on the template don't change. FrameAssistant
  loads its configuration there and decides from `mode()` whether it needs it (CFG-2).
- **Modes (D22):**

  | Mode | Configuration + HA connection | SteamVR |
  |---|---|---|
  | overlay (default) | yes | required (overlay app) |
  | `--desktop` | yes | connects as background app if SteamVR runs; otherwise VR entities are unavailable (SEN-15) |
  | `--desktop-novr` | yes | never |
  | `--snapshot`, `--benchmark` | no | no |

  - `--desktop` on the Frame over RDP shows the live UI with real data without wearing the
    headset. Stop the SteamVR-started instance by hand first: two instances with the same
    device name would knock each other off the broker.
  - `--config <path>` loads another configuration file instead of
    `~/.config/frameassistant/config.json`, e.g. on a PC with its own device name (CON-5).

## Overlays

Two overlay kinds in the platform (D25):

| Overlay | Kind | Visibility | MVP |
|---|---|---|---|
| Main tab (UI-1, later NOT-7, do-not-disturb) | dashboard tab | always there; SteamVR shows/hides it | ✓ |
| Notifications | world overlay | the app (notification manager) shows/hides it | ✓ |
| Status tab (SRV-6) | dashboard tab | exists only while cards exist (D9) | ✗ |
| HA dashboard (DSH-1, web UI test) | dashboard tab | always there | ✓ |

- **World overlays (MVP):** a new overlay kind next to the dashboard tab (CLAUDE.md, "Planned
  extension"). The surface gets a kind and a placement; renders only while visible (NFR-3).
  In the desktop modes (D22) a world overlay is a plain window the app shows and hides.
- **Premultiplied alpha:** Qt Quick renders premultiplied alpha; SteamVR assumes straight
  alpha unless `VROverlayFlags_IsPremultiplied` is set, which the platform doesn't do yet.
  Fully opaque/transparent pixels are right, semi-transparent ones (antialiasing, shadows)
  would be too dark. Set the flag in the platform; verify on the device.
- **Runtime surfaces (later, SRV-6):** creating a surface at runtime repeats the start-up
  steps for one surface. Destroying one while SteamVR runs is the risk: each surface has its
  own backend (with Vulkan its own `VkDevice`), and SteamVR's client keeps objects on devices
  it got textures from (the reason for the teardown order). Needs a spike on the Frame; ways
  out: one device shared by all surfaces, or reuse instead of destroy.
- **Interactive world overlays (later, NOT-5):** the laser hits the whole overlay rectangle,
  including transparent parts.

### Notifications (D26, D27)

- **One world overlay for all notifications** (D26). QML lays out the visible notifications
  as cards inside it; transparent areas around them are invisible. No runtime surfaces needed,
  one renderer and texture (NFR-3). One placement for all; per-notification placement is not
  required (NOT-2 makes placement configurable, not per notification) and could later become
  one overlay per placement.
- **MVP placement: spawn in front, then stay** (D27): about 1 m away, slightly below the line
  of sight, positioned once when it appears. No continuous tracking. Lazy follow and hand
  placement come with NOT-2.
- Placement techniques:
  - head-locked: SteamVR keeps the overlay relative to the HMD; runs in the compositor, smooth.
  - spawn in front, then stay: one absolute transform when shown.
  - lazy follow: the app updates the transform continuously from the head pose; CPU cost and
    jitter if updated too rarely.
- UI guidelines from research: Appendix A.

## Object classes

The app is HA-centric, so its building blocks follow HA's concepts (D18). Objects talk to each
other through Qt signals and slots (Qt's observer pattern). Senders don't know their receivers,
and signals across threads are delivered through the receiver's event loop.

| Class | Role | MVP |
|---|---|---|
| **HA entity** | One HA entity, both directions. Entity → HA: state and JSON attributes. HA → entity: commands; HA's MQTT integration turns its own services into them (e.g. `button.press`, `switch.turn_on`, `number.set_value`, `notify.send_message`). Observes value signals of other objects and publishes them; turns MQTT commands into signals other objects act on. **Native entity types are preferred** wherever their meaning fits (D23). | ✓ |
| **HA service** | **Fallback** where no entity type fits (D23): a JSON command on an own MQTT command topic (HA side: the standard `mqtt.publish` action), e.g. rich notifications or vibration with parameters. Results go back asynchronously, e.g. through an `event` entity. How the user wraps it in HA (scripts, automations, pyscript) is their choice; FrameAssistant only requires stock HA with MQTT (NFR-6). | depends on the NOT-1 choice |
| **HA device** | Owns the MQTT connection, discovery and availability, and owns all entity and service objects. One per headset (CON-5). | ✓ |
| **I/O wrappers** | Data sources and actuators; know nothing about HA. One wrapper object per OpenVR interface used (e.g. `IVRSystem`, `IVRApplications`, `IVRChaperone`, `IVRSettings`). System wrappers (network, audio) later. | OpenVR ✓, system ✗ |
| **Overlays** | View and controller of MVC (QML), as dashboard tabs or world overlays. Get their data (the model) from other objects. | ✓ |
| **Managers** | One object per stateful feature that manages and coordinates it, using entities, services and overlays. Notification manager (MVP); status card manager (SRV-6/7) later. | notifications ✓ |
| **HA web UI** | Encapsulates Qt WebEngine for the dashboard tab (DSH-1; Q2 for DSH-2). | ✓ (DSH-1) |
| **Configuration** | Settings from the JSON file, loaded once by the app as an immutable value. Each component gets the part it needs through its constructor (D19). | ✓ |
| **App** | The managing object: owns all of the above as components and wires them together. | ✓ |

### Entities and services (D23)

- **One class per HA entity type** on a common base. The base does the discovery part, topics,
  availability and throttling, so SEN-14, SEN-15 and NFR-7 are handled in one place.
- A concrete entity is an **instance with a description** (id, name, device class, unit,
  throttle interval), not a class of its own.
- Our actions map to native types where the meaning fits:

  | Action | Type | HA service |
  |---|---|---|
  | Sensors (SEN-*) | `sensor`, `binary_sensor` | — |
  | Notification (NOT-1) | `notify` | `notify.send_message` |
  | Screenshot, standby, quit app, show boundary (SRV-1/2/3) | `button` | `button.press` |
  | Do-not-disturb (SRV-5) | `switch` | `switch.turn_on/off` |
  | Volume (SRV-3) | `number` | `number.set_value` |
  | Launch app by key (SRV-2) | `text` | `text.set_value` |
  | Attention signal with duration/volume (SRV-1), to check | `siren` | `siren.turn_on` |
  | Feedback (NOT-5, NOT-10) | `event` | — |

- **Wiring:** the app wires simple entities (I/O wrapper signal ↔ entity). Managers get their
  entities and services from the HA device and wire them internally. The HA device owns them
  all (D18).
- **MQTT `notify` drops the title:** HA's `async_send_message()` renders only `message` into
  the payload (`homeassistant/components/mqtt/notify.py`). NOT-1 needs title and text; options
  (MVP detail): first line is the title, JSON in the message, or an own service.

Notes for later:
- OpenVR delivers all events through one queue; one dispatcher (today `VrRuntime`) hands them
  to the wrappers.
- OpenVR is not documented as thread-safe (relevant for the thread decision).

## Open

1. MVP details (see [PROGRESS.md](PROGRESS.md), "Next steps").

## Test strategy

Four levels (D30), implementing NFR-8:

| Level | What | How | Where |
|---|---|---|---|
| 1. Unit tests | configuration (CFG-2), discovery payloads, throttling (SEN-14), availability logic (SEN-15), entities, notification manager | QtTest with **fakes at the boundaries**: a fake MQTT client for the three signals (D29), fake I/O wrappers instead of OpenVR | host and test container, `ctest` |
| 2. Integration tests with a real broker | CON-1–4: connect, last will when killed, reconnect after broker restart, login/TLS | the test starts a local `mosquitto` on a free port (host: installed; test container: add the package) | host and test container, `ctest` |
| 3. UI and rendering | main tab and notification cards, transparency | existing tools: `--snapshot`, `tst_render`, desktop modes | host |
| 4. Manual | against the user's HA: entities appear, no deprecation warnings (NFR-7). On the Frame: sensors, notification in VR and in a game, DSH-1, premultiplied alpha, NFR-3 | checklist per iteration, results documented in `docs/` | user's HA, the Frame |

- OpenVR wrappers stay thin; logic goes into the testable parts behind them. The wrappers
  themselves are only checked on the device (level 4).
- HA itself is tested manually for now (an HA container would be possible but costly);
  revisit later (NFR-8 allows adjusting the strategy).
- The NFR-3 measurement method is a detail (idea: compare SteamVR compositor statistics, e.g.
  missed frames, with and without FrameAssistant).

## Appendix A: notification UI in VR (research)

Collected 2026-10-05 for the notification design.

**Studies:**
- [Rzayev et al., CHI PLAY 2019](https://dl.acm.org/doi/abs/10.1145/3311350.3347190), 24
  participants, tasks incl. gaming; placements head-up display (fixed in view), on-body,
  floating, in-situ (embedded in the scene). On-body and floating: best understandability,
  usability and presence. HUD: most intrusive and disturbing; only for urgent notifications.
- [Hsieh et al., CHI 2020](https://dl.acm.org/doi/10.1145/3313831.3376228), 40 participants;
  displays on the HMD, on the controller, movable panel:
  - Users dislike notifications blocking their focus region (HMD: inflexible location;
    controller: blocks the work area). Recommendation: let users choose the location from a
    set of peripheral positions.
  - The more engaged the user, the more disruptive notifications feel. Then rather a small
    peripheral indicator (e.g. unread count), content later.
  - Don't reuse a modality the VR content already uses (e.g. game vibration hides
    notification vibration). Urgent: more obtrusive means such as audio.
  - Display time in the study was 12 s; longer is likely more disruptive.
- [NotifiVR, Ghosh et al. 2018](https://pubmed.ncbi.nlm.nih.gov/29543163): haptic alone was the
  least effective modality.

**Platforms and tools:**
- [Meta Quest design guidelines](https://developers.meta.com/horizon/design/mr-design-guideline/):
  avoid head-locked UI; anchor in space or "lazy follow" (follows the head delayed and
  smoothed); about 1 m away, slightly below the line of sight. Users can set where
  notifications appear.
- [visionOS](https://developer.apple.com/videos/play/wwdc2023/10078/): content to read goes
  centred and slightly below the line of sight; periphery for context, centre for focus.
- [XSOverlay](https://xsoverlay.vercel.app/settings) (common SteamVR overlay tool): popups at a
  configurable offset from the head, smoothly following;
  [xsoverlay-tweak](https://github.com/chaixshot/xsoverlay-tweak) adds a "leash" mode (stays
  until the head turns too far away).

**Consequences for FrameAssistant:**
1. Placement: not rigidly head-locked. Lazy follow/leash, spawn in front then stay, and hand
   are well supported (matches NOT-2). Head-locked at most for critical notifications (NOT-8).
2. Position: about 1 m, below the line of sight or towards the periphery, not centred; offset
   configurable.
3. Content: compact (title, short text), fade in/out, timeout around 10 s; longer content in
   the main tab's list (NOT-7).
4. Several at once: few visible, stacked, newest in front (fits D26).
5. Modalities: sound as the main signal for important notifications (NOT-4/8), vibration only
   as a supplement.

## Decisions

| # | Date | Decision | Reason |
|---|------|----------|--------|
| D16 | 2026-10-05 | **One process** started by SteamVR does everything; several threads inside it for concurrent work | Meets NFR-2 without extra work, no IPC; the platform already supports several surfaces per process |
| D17 | 2026-10-05 | **Three layers:** presentation (QML), business logic, I/O. The overlay platform spans presentation (overlays, rendering, input) and I/O (SteamVR access) | Simple and enough for the start; Qt Quick already enforces separating presentation from logic |
| D18 | 2026-10-05 | **HA-centric object classes:** HA entity, HA service, HA device, I/O wrappers, overlays, managers, HA web UI, configuration, app; communication through signals and slots | The app is HA-centric; signals/slots decouple objects and work across threads |
| D19 | 2026-10-05 | **Configuration is injected:** the app loads it once as an immutable value and passes each component the part it needs; no singleton | Explicit dependencies, trivial tests with own configs, thread-safe; the app creates all components anyway |
| D21 | 2026-10-05 | Start-up hook in the platform that may fail (`prepare()`), before the engine, surfaces and `VR_Init`; failure → exit code 1 | CFG-2 needs a clean abort; `loading()`/`started()` can't fail; backward compatible |
| D22 | 2026-10-05 | Overlay and both desktop modes load the configuration and connect to HA; `--desktop` also connects to SteamVR as a background app if it runs; `--desktop-novr` never does; `--snapshot`/`--benchmark` stay without configuration; new option `--config <path>` | Live development over RDP on the Frame with real data; PC development without SteamVR under its own device name |
| D23 | 2026-10-05 | **One class per HA entity type** on a common base; concrete entities are described instances. Native entity types (and the HA services the MQTT integration provides for them) are preferred; an own JSON command service only as fallback | Mirrors HA's model; discovery, availability and throttling in one place; actions are visible and selectable in HA without `mqtt.publish` |
| D24 | 2026-10-05 | MQTT for everything device-related; the dashboard gets its own channel to HA, chosen with Q2/Q3 | MQTT can't provide HA as a whole; both dashboard options fit the layers without affecting the MVP |
| D25 | 2026-10-05 | Two overlay kinds in the platform: dashboard tabs and world overlays (MVP: one placement, not interactive). Set `VROverlayFlags_IsPremultiplied`. Runtime surfaces and interactive world overlays later | Notifications need world overlays; Qt Quick renders premultiplied alpha |
| D26 | 2026-10-05 | One world overlay for all notifications; QML lays out the cards | No runtime surfaces, fewer resources; per-notification placement isn't required |
| D27 | 2026-10-05 | MVP notification placement: spawn in front (about 1 m, slightly below the line of sight), then stay | Research favours non-head-locked placement; no continuous tracking needed |
| D29 | 2026-10-05 | Threads: only the MQTT client runs in its own thread; everything else, incl. all OpenVR calls and the HA layer, in the main thread. New threads only for work that can block | Blocking connect must not stall rendering; OpenVR isn't documented as thread-safe; the narrowest possible thread boundary |
| D30 | 2026-10-05 | Test strategy in four levels: unit tests with fakes at the boundaries, integration tests against a local mosquitto, UI/render tools, manual checklist (user's HA, the Frame) | Covers NFR-8; the narrow boundaries (D29, I/O wrappers) make fakes cheap; HA automation deferred |