From 06de5cf1842d0ecd712eb7f52b85ac50565e20d2 Mon Sep 17 00:00:00 2001 From: Matzman Date: Mon, 5 Oct 2026 21:26:38 +0200 Subject: [PATCH] Design: overlays, notification placement, DSH-1 in the MVP D25: dashboard tabs and world overlays; premultiplied-alpha flag. D26: one world overlay for all notifications. D27: MVP placement "spawn in front, then stay". D28: DSH-1 (web UI out-of-the-box test) joins the MVP, full dashboard becomes DSH-2. Add the research on notification UI in VR (DESIGN.md, Appendix A). Co-Authored-By: Claude Opus 5.5 --- docs/DESIGN.md | 111 ++++++++++++++++++++++++++++++++++++++++++- docs/PROGRESS.md | 13 +++-- docs/REQUIREMENTS.md | 8 ++-- 3 files changed, 125 insertions(+), 7 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 299309d..52d588a 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -26,6 +26,23 @@ Three layers; dependencies only go downward (D17). - The overlay platform (`vrdashboard_core`) spans two layers: overlays, rendering and input are presentation infrastructure; SteamVR access for sensors and actions is I/O. +## HA channels + +- **MQTT is the only channel for everything device-related:** entities, services, availability + (D24). +- MQTT can't show HA as a whole (other entities' states, service calls for them), which the + dashboard (DSH-1) needs. The dashboard gets **its own channel, separate from MQTT**: + - Q2a, HA web UI in Qt WebEngine: the browser talks to HA itself (HTTP/WebSocket, HA's own + login); config: HA URL. + - Q2b, native QML dashboard: HA's WebSocket API with a token (Q3) as another I/O component; + config: HA URL and token. +- Both fit the three layers. The MVP contains the web UI test (DSH-1, D28), which also serves + as the spike on WebEngine in the off-screen renderer; Q2/Q3 are decided for DSH-2. +- Known DSH-1 risks: WebEngine must be initialised before the `QGuiApplication` (created by the + platform's base constructor), so the platform needs a small change; WebEngine with our + off-screen renderer, especially Vulkan, is untested; text fields inside the page probably + don't open the SteamVR keyboard (fallback: log in once in a desktop mode). + ## Start-up and modes - **Start-up hook (D21):** the platform (`DashboardApplication` in `vrdashboard_core`) gets a @@ -49,6 +66,49 @@ Three layers; dependencies only go downward (D17). - `--config ` loads another configuration file instead of `~/.config/frameassistant/config.json`, e.g. on a PC with its own device name (CON-5). +## Overlays + +Two overlay kinds in the platform (D25): + +| Overlay | Kind | Visibility | MVP | +|---|---|---|---| +| Main tab (UI-1, later NOT-7, do-not-disturb) | dashboard tab | always there; SteamVR shows/hides it | ✓ | +| Notifications | world overlay | the app (notification manager) shows/hides it | ✓ | +| Status tab (SRV-6) | dashboard tab | exists only while cards exist (D9) | ✗ | +| HA dashboard (DSH-1, web UI test) | dashboard tab | always there | ✓ | + +- **World overlays (MVP):** a new overlay kind next to the dashboard tab (CLAUDE.md, "Planned + extension"). The surface gets a kind and a placement; renders only while visible (NFR-3). + In the desktop modes (D22) a world overlay is a plain window the app shows and hides. +- **Premultiplied alpha:** Qt Quick renders premultiplied alpha; SteamVR assumes straight + alpha unless `VROverlayFlags_IsPremultiplied` is set, which the platform doesn't do yet. + Fully opaque/transparent pixels are right, semi-transparent ones (antialiasing, shadows) + would be too dark. Set the flag in the platform; verify on the device. +- **Runtime surfaces (later, SRV-6):** creating a surface at runtime repeats the start-up + steps for one surface. Destroying one while SteamVR runs is the risk: each surface has its + own backend (with Vulkan its own `VkDevice`), and SteamVR's client keeps objects on devices + it got textures from (the reason for the teardown order). Needs a spike on the Frame; ways + out: one device shared by all surfaces, or reuse instead of destroy. +- **Interactive world overlays (later, NOT-5):** the laser hits the whole overlay rectangle, + including transparent parts. + +### Notifications (D26, D27) + +- **One world overlay for all notifications** (D26). QML lays out the visible notifications + as cards inside it; transparent areas around them are invisible. No runtime surfaces needed, + one renderer and texture (NFR-3). One placement for all; per-notification placement is not + required (NOT-2 makes placement configurable, not per notification) and could later become + one overlay per placement. +- **MVP placement: spawn in front, then stay** (D27): about 1 m away, slightly below the line + of sight, positioned once when it appears. No continuous tracking. Lazy follow and hand + placement come with NOT-2. +- Placement techniques: + - head-locked: SteamVR keeps the overlay relative to the HMD; runs in the compositor, smooth. + - spawn in front, then stay: one absolute transform when shown. + - lazy follow: the app updates the transform continuously from the head pose; CPU cost and + jitter if updated too rarely. +- UI guidelines from research: Appendix A. + ## Object classes The app is HA-centric, so its building blocks follow HA's concepts (D18). Objects talk to each @@ -63,7 +123,7 @@ and signals across threads are delivered through the receiver's event loop. | **I/O wrappers** | Data sources and actuators; know nothing about HA. One wrapper object per OpenVR interface used (e.g. `IVRSystem`, `IVRApplications`, `IVRChaperone`, `IVRSettings`). System wrappers (network, audio) later. | OpenVR ✓, system ✗ | | **Overlays** | View and controller of MVC (QML), as dashboard tabs or world overlays. Get their data (the model) from other objects. | ✓ | | **Managers** | One object per stateful feature that manages and coordinates it, using entities, services and overlays. Notification manager (MVP); status card manager (SRV-6/7) later. | notifications ✓ | -| **HA web UI** | Encapsulates Qt WebEngine for the dashboard (Q2). | ✗ | +| **HA web UI** | Encapsulates Qt WebEngine for the dashboard tab (DSH-1; Q2 for DSH-2). | ✓ (DSH-1) | | **Configuration** | Settings from the JSON file, loaded once by the app as an immutable value. Each component gets the part it needs through its constructor (D19). | ✓ | | **App** | The managing object: owns all of the above as components and wires them together. | ✓ | @@ -102,6 +162,51 @@ Notes for later: 1. High-level points 3–7 (see [PROGRESS.md](PROGRESS.md), "Next steps"). +## Appendix A: notification UI in VR (research) + +Collected 2026-10-05 for the notification design. + +**Studies:** +- [Rzayev et al., CHI PLAY 2019](https://dl.acm.org/doi/abs/10.1145/3311350.3347190), 24 + participants, tasks incl. gaming; placements head-up display (fixed in view), on-body, + floating, in-situ (embedded in the scene). On-body and floating: best understandability, + usability and presence. HUD: most intrusive and disturbing; only for urgent notifications. +- [Hsieh et al., CHI 2020](https://dl.acm.org/doi/10.1145/3313831.3376228), 40 participants; + displays on the HMD, on the controller, movable panel: + - Users dislike notifications blocking their focus region (HMD: inflexible location; + controller: blocks the work area). Recommendation: let users choose the location from a + set of peripheral positions. + - The more engaged the user, the more disruptive notifications feel. Then rather a small + peripheral indicator (e.g. unread count), content later. + - Don't reuse a modality the VR content already uses (e.g. game vibration hides + notification vibration). Urgent: more obtrusive means such as audio. + - Display time in the study was 12 s; longer is likely more disruptive. +- [NotifiVR, Ghosh et al. 2018](https://pubmed.ncbi.nlm.nih.gov/29543163): haptic alone was the + least effective modality. + +**Platforms and tools:** +- [Meta Quest design guidelines](https://developers.meta.com/horizon/design/mr-design-guideline/): + avoid head-locked UI; anchor in space or "lazy follow" (follows the head delayed and + smoothed); about 1 m away, slightly below the line of sight. Users can set where + notifications appear. +- [visionOS](https://developer.apple.com/videos/play/wwdc2023/10078/): content to read goes + centred and slightly below the line of sight; periphery for context, centre for focus. +- [XSOverlay](https://xsoverlay.vercel.app/settings) (common SteamVR overlay tool): popups at a + configurable offset from the head, smoothly following; + [xsoverlay-tweak](https://github.com/chaixshot/xsoverlay-tweak) adds a "leash" mode (stays + until the head turns too far away). + +**Consequences for FrameAssistant:** +1. Placement: not rigidly head-locked. Lazy follow/leash, spawn in front then stay, and hand + are well supported (matches NOT-2). Head-locked at most for critical notifications (NOT-8). +2. Position: about 1 m, below the line of sight or towards the periphery, not centred; offset + configurable. +3. Content: compact (title, short text), fade in/out, timeout around 10 s; longer content in + the main tab's list (NOT-7). +4. Several at once: few visible, stacked, newest in front (fits D26). +5. Modalities: sound as the main signal for important notifications (NOT-4/8), vibration only + as a supplement. + ## Decisions | # | Date | Decision | Reason | @@ -113,3 +218,7 @@ Notes for later: | D21 | 2026-10-05 | Start-up hook in the platform that may fail (`prepare()`), before the engine, surfaces and `VR_Init`; failure → exit code 1 | CFG-2 needs a clean abort; `loading()`/`started()` can't fail; backward compatible | | D22 | 2026-10-05 | Overlay and both desktop modes load the configuration and connect to HA; `--desktop` also connects to SteamVR as a background app if it runs; `--desktop-novr` never does; `--snapshot`/`--benchmark` stay without configuration; new option `--config ` | Live development over RDP on the Frame with real data; PC development without SteamVR under its own device name | | D23 | 2026-10-05 | **One class per HA entity type** on a common base; concrete entities are described instances. Native entity types (and the HA services the MQTT integration provides for them) are preferred; an own JSON command service only as fallback | Mirrors HA's model; discovery, availability and throttling in one place; actions are visible and selectable in HA without `mqtt.publish` | +| D24 | 2026-10-05 | MQTT for everything device-related; the dashboard gets its own channel to HA, chosen with Q2/Q3 | MQTT can't provide HA as a whole; both dashboard options fit the layers without affecting the MVP | +| D25 | 2026-10-05 | Two overlay kinds in the platform: dashboard tabs and world overlays (MVP: one placement, not interactive). Set `VROverlayFlags_IsPremultiplied`. Runtime surfaces and interactive world overlays later | Notifications need world overlays; Qt Quick renders premultiplied alpha | +| D26 | 2026-10-05 | One world overlay for all notifications; QML lays out the cards | No runtime surfaces, fewer resources; per-notification placement isn't required | +| D27 | 2026-10-05 | MVP notification placement: spawn in front (about 1 m, slightly below the line of sight), then stay | Research favours non-head-locked placement; no continuous tracking needed | diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 99b5867..d9749c5 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -10,15 +10,17 @@ CLAUDE.md ("Working rules"). ## Next steps -Phase 2: first the high-level architecture, then the details for the MVP (D8). High-level +Phase 2: first the high-level architecture, then the details for the MVP (D8, plus DSH-1 by +D28). High-level questions, in this order: 1. ~~Process model~~: one process, several threads (D16). 2. ~~Layers~~: presentation, business logic, I/O (D17). 3. ~~How features plug in~~: object classes (D18), configuration injection (D19), one class per HA entity type with native types preferred (D23). -4. HA channels: MQTT only, or prepare for a second one for the dashboard (Q2/Q3)? -5. Overlay kinds: dashboard tabs and world overlays (platform extension). +4. ~~HA channels~~: MQTT for the device, own channel for the dashboard later (D24). +5. Overlay kinds: dashboard tabs and world overlays (D25), one notification overlay (D26), + MVP placement (D27); DSH-1 web UI test in the MVP (D28). 6. Runtime model: which work runs in which thread (D16: several threads). 7. Test strategy at a high level. @@ -40,6 +42,11 @@ configuration, packaging of libmosquitto. present (relevant for Q2/Q3). - libmosquitto ABI check: Debian's 2.0.21 needs at most `OPENSSL_3.0.0`/`GLIBC_2.34`; the Frame has OpenSSL 3.2.1 and glibc 2.39. No libcjson dependency. D14 holds. +- DSH-1 split: web UI out-of-the-box test in the MVP, full dashboard as DSH-2 (D28). +- Overlays: two kinds (D25), one overlay for all notifications (D26), MVP placement "spawn in + front, then stay" (D27). Researched notification UI in VR (DESIGN.md, Appendix A). Found: + premultiplied-alpha flag missing in the platform. +- HA channels: MQTT for everything device-related; dashboard channel decided with Q2/Q3 (D24). - Entities: one class per HA entity type, native types preferred, own service only as fallback (D23). Found: MQTT `notify` drops the title. - Start-up hook `prepare()` in the platform (D21); modes, `--desktop-novr` and `--config` diff --git a/docs/REQUIREMENTS.md b/docs/REQUIREMENTS.md index 442b344..738c523 100644 --- a/docs/REQUIREMENTS.md +++ b/docs/REQUIREMENTS.md @@ -103,7 +103,8 @@ FrameAssistant. | ID | Requirement | Prio | MVP | Acceptance criteria | |----|-------------|------|-----|---------------------| -| DSH-1 | A dashboard tab to control HA entities in VR (approach: Q2, Q3). | S | | Defined after design exploration. | +| DSH-1 | A dashboard tab shows HA's web UI (Qt WebEngine) at a configured URL, as a "does it work out of the box" test. | M | ✓ | The page loads in VR and can be operated with the laser; whatever doesn't work out of the box is documented and moved to a later iteration. | +| DSH-2 | The full dashboard to control HA entities in VR: everything beyond DSH-1, e.g. login comfort, opening a given HA view (SRV-4), possibly a native approach (Q2, Q3). | S | | Defined after design exploration. | ### Non-functional (NFR) @@ -161,6 +162,7 @@ FrameAssistant. | D14 | 2026-10-05 | MQTT client: **libmosquitto** from Debian's package in the build container, shipped in the release package like `libopenvr_api.so` | Not on the Frame; Qt MQTT would need the exact Qt version (Appendix B) | | D15 | 2026-10-05 | HA MQTT discovery: **own small layer on Qt's JSON classes**, using device-based discovery (one `/device//config` topic with all entities under `cmps`) | No suitable library (Appendix B); HA recommends device-based discovery for multi-entity devices (NFR-7) | | D20 | 2026-10-05 | New requirement UI-1 (status display in the main tab), MVP | Helps testing on the device; little effort | +| D28 | 2026-10-05 | DSH-1 split: DSH-1 (MVP) = HA web UI in a tab as an out-of-the-box test; DSH-2 (Should) = the full dashboard. The MVP (D8) now also contains DSH-1 | The user wants to see early whether the simple variant works | Design decisions (D16 onward) are in [DESIGN.md](DESIGN.md). @@ -271,8 +273,8 @@ use Qt's private API, which requires the exact Qt version. | Sounds, audio from URL | NOT-4, SRV-8 | Qt Multimedia + QtNetwork | ✓ `libQt6Multimedia`, FFmpeg plugin | no | | HTTPS (downloads, WebSocket) | SRV-8, Q3 | Qt TLS plugin | ✓ `libqopensslbackend.so` | no | | Translations | NFR-4 | `QTranslator`, `lupdate`/`lrelease` | ✓ (build-time tools) | no | -| Dashboard, HA web UI | DSH-1, Q2 | Qt WebEngine (Quick) | ✓ `libQt6WebEngineCore`, `libQt6WebEngineQuick`, `libQt6WebView` | no; works with our off-screen renderer and Vulkan? Needs a spike | -| Dashboard, native UI | DSH-1, Q2/Q3 | HA WebSocket API via Qt WebSockets | ✓ `libQt6WebSockets` | no | +| Dashboard, HA web UI | DSH-1, DSH-2, Q2 | Qt WebEngine (Quick) | ✓ `libQt6WebEngineCore`, `libQt6WebEngineQuick`, `libQt6WebView` | no; works with our off-screen renderer and Vulkan? Needs a spike | +| Dashboard, native UI | DSH-2, Q2/Q3 | HA WebSocket API via Qt WebSockets | ✓ `libQt6WebSockets` | no | | Haptics, fade, apps | SRV-1/2 | OpenVR | ✓ | no | | Tests | NFR-8 | QtTest | ✓ (host only) | no |