Files
FrameAssistant/docs/PROGRESS.md
T
matzmanandClaude Opus 5.5 6e1bed79f0 Wrap up the session: iteration 2 implemented, phase 4 next
PROGRESS.md: current state, next session, open suggestions, log entry.
DESIGN.md and CLAUDE.md status lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:50:06 +02:00

346 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Progress
Where the project stands, and what happened in each session. Newest first. The process is in
CLAUDE.md ("Working rules").
## Current state
- **Implementation iteration 2, phase 4 (testing), not started yet.** Requirements (see
"Next steps", D41) and design (DESIGN.md, "Iteration 2: MQTT and HA device", D42–D45)
approved; implementation (phase 3) done and committed on 2026-10-06.
- **Iteration 1 closed.** DSH-1 had a negative result: HA's web UI via Qt WebEngine doesn't
work on the Frame, so the "HA Dashboard" tab and the WebEngine dependency were removed
(D40). The rest of iteration 1 (platform hooks, settings file, `--config`, log file for
SteamVR-started instances) works on the Frame with the system-Qt package (3.7 MB).
- **Frame test results of iteration 1 (2026-10-05):**
| # | Check | Result |
|---|---|---|
| 1 | Started by SteamVR: process runs, log file from the manifest's `--log-file`, settings loaded, rendering with Vulkan (Turnip, Adreno 750) | ✓ |
| 2 | In VR: tab "Frame Assistant" visible and usable | ✓ |
| 3 | Settings without `device.name`, `./install.sh` → app exits, log says "Setting 'device.name' is required" | skipped (no device access), postponed |
| 4 | Settings restored, `./install.sh` → runs again | skipped (no device access), postponed |
The error itself is covered off the device by `tst_config`. Checks 3 and 4 run in the next
session on the Frame (iteration 2 testing needs it too).
- **Implementation (phase 3) done:** steps 1–7 of DESIGN.md. **Next: phase 4 (testing)**
with the manual checklist below, from the PC against the user's HA and broker; then on the
Frame.
- **Next session:** run the checklist below with the user (points 1–9 from the PC against
the user's HA and broker, 10–12 on the Frame), record the results, then phase 5
(assessment of iteration 2), then iteration 3 (OpenVR wrappers, sensors).
- **Open suggestions (not done, need the user's decision):**
- Strip the debug info from the package binary (11 of its 12 MB; package 5.0 MB instead of
about 3.7 MB before iteration 2).
- Package README.txt: drop "Home Assistant URL" from the set-up line (`homeassistant.url`
isn't used since D40).
- Iteration 3 decides whether the dummy sensor "Uptime" is removed from HA or kept as a
diagnostic entity (D41).
- **Manual checklist for iteration 2 (phase 4):**
| # | Check | Where | Result |
|---|---|---|---|
| 1 | Settings file for the PC (own device name, the user's broker), `--desktop-novr --config …`: tab shows "Connected" | PC | open |
| 2 | HA: device appears with the "Uptime" sensor (diagnostic), value rises every 10 s; HA log has no MQTT discovery warnings (NFR-7); `~` base topic accepted | PC + HA | open |
| 3 | Close the window → HA shows the entities unavailable at once (graceful exit) | PC + HA | open |
| 4 | `kill -9` → unavailable (last will) | PC + HA | open |
| 5 | Broker restart → tab "Not connected" + error, then "Connected"; HA values current again | PC + HA | open |
| 6 | HA restart → device republished after the birth message | PC + HA | open |
| 7 | Two settings files with different device names → two HA devices | PC + HA | open |
| 8 | Login and, if the broker offers it, TLS against the user's broker | PC + HA | open |
| 9 | `--ha-remove --config …` → device gone from HA | PC + HA | open |
| 10 | Package on the Frame: starts with SteamVR, connects, tab shows the status; "Quit app" → unavailable | Frame | open |
| 11 | `uninstall.sh` on the Frame removes the device from HA; `install.sh` brings it back | Frame | open |
| 12 | Iteration 1 checks 3 and 4 (settings errors when started by SteamVR) | Frame | open |
## Next steps
Requirements and design cover the whole MVP; it is **implemented in four small iterations**
(D37), each with implementation, testing and assessment:
| Iteration | Content | Platform change | Checked by |
|---|---|---|---|
| 1 | ~~DSH-1: HA web UI in a dashboard tab~~ (failed, removed: D40); configuration (D34) with start-up hook and `--config`; log file for SteamVR-started instances | start-up hook (D21), app command-line options (D38) | DSH-1 result documented; app starts on the Frame with the settings file, settings errors in the log |
| 2 | MQTT client (own thread), HA device with discovery and availability, libmosquitto in the package, status display UI-1 | modes (D22) | device appears in HA, online/offline right, reconnect works |
| 3 | OpenVR wrappers and the 13 sensor entities | `--desktop` with optional SteamVR (D22) | sensor values in HA on the Frame |
| 4 | notification manager, `notify` entity, world overlay | world overlay, premultiplied flag (D25) | notification from HA in VR, also in a game |
### Requirements of implementation iteration 2: MQTT, HA device, UI-1
Approved 2026-10-06. Goal: FrameAssistant connects to the user's broker, appears in HA as a
device that goes online and offline correctly, and the main tab shows the connection status.
No real sensors yet: a **dummy sensor** stands in for them (D41).
- **Dummy sensor "Uptime":** a `sensor` with the seconds since start, `entity_category:
diagnostic`, updated every 10 s. In the code a generic test entity whose value is set from
outside: the app feeds it a timer, unit tests set values directly. Ships in iteration 2
builds; iteration 3 decides whether it is removed from HA (component with only `p` once) or
kept as a diagnostic entity.
| Item | Acceptance criterion |
|---|---|
| CON-1, CFG-3 | Connects without login, with login, and with TLS (self-signed CA); connection state and errors are logged |
| CON-2 | The device with the dummy sensor appears in HA via device-based discovery, without configuration in HA |
| CON-3 | Online while running; normal exit and `kill -9` make it unavailable |
| CON-4 | After a broker restart it reconnects with backoff and republishes discovery and states; also republishes after HA's birth message |
| CON-5 | Two settings files with different device names give two HA devices |
| SEN-14 | The dummy sensor's updates are throttled |
| UI-1 | The main tab shows device name, broker, connected or not, and the last error, and follows a broker stop and start |
| Modes (D22) | Overlay, `--desktop` and the new `--desktop-novr` load the settings and connect; in iteration 2 the two desktop modes behave alike (neither uses SteamVR yet) |
| `--ha-remove` (D32) | Clears all retained topics, after which the device is gone from HA; `uninstall.sh` calls it |
| D35 | libmosquitto is in both packages; `package-release.sh`'s symbol-version check passes |
| Tests (D30) | Unit tests with a fake MQTT client; integration tests against a local mosquitto (connect, last will, reconnect, login, TLS) |
| Manual | CON, UI and `--ha-remove` checks from the PC against the user's HA (possible because nothing needs SteamVR). On the Frame: the package with libmosquitto runs; the postponed checks 3 and 4 of iteration 1 |
To find out during the iteration: where the Frame keeps its system certificates (default
for `mqtt.tls.ca_file`).
### Plan for implementation iteration 1: DSH-1 and configuration
Goal: a second dashboard tab **"HA Dashboard"** shows HA's web UI at the URL from the
configuration; what works out of the box is tested and documented. The main tab stays
"Frame Assistant".
Platform (`vrdashboard_core`):
1. Start-up hook `prepare()` (D21).
2. App-specific command-line options (D38): the base constructor takes an optional list of
extra `QCommandLineOption`s, adds them to parsing and `--help`, and exposes their values.
App:
3. `QtWebEngineQuick::initialize()` in `main.cpp` before the app object (D39); no platform
change.
4. Configuration (`src/config/`): immutable value plus `load(path, &error)`, complete per D34;
path from `--config`, else `$XDG_CONFIG_HOME` or `~/.config/frameassistant/config.json`;
loaded in `Dashboard::prepare()` (overlay and desktop modes).
5. HA web UI (`src/ha/HaWebUi`, D18): holds the URL (injected), exposed to QML as a singleton
owned by `Dashboard`; a named, persistent WebEngine profile under
`~/.local/share/frameassistant/` so the HA login survives restarts (D39).
6. New surface `dashboard` ("HA Dashboard"), QML root `HaDashboard.qml` with a `WebEngineView`;
without a URL it shows a hint.
7. Build: `find_package(Qt6 … WebEngineQuick)`; `qt6-webengine-dev` (native and `:arm64`) in
the containers; a new static library for app logic (configuration, HaWebUi) that tests link.
Tests (D30):
8. `tst_config`: valid file, missing required fields, invalid JSON, wrong types, defaults,
derived and explicit `ha_unique_id`, unknown keys, `--config`.
9. Platform: `tst_surfaces` extended for `prepare()` (success/abort) and extra options.
10. Manual, results documented: host `--desktop`; Frame desktop mode over RDP; Frame in VR with
Vulkan and OpenGL: page loads, laser click, scrolling, SteamVR keyboard in the page's input
fields, login kept after restart; CPU/GPU while the tab is open.
11. `clang-format` clean, `ctest` green; `container-test.sh` only if Vulkan code is touched.
Docs:
12. README (configuration file, `--config`, new tab), ARCHITECTURE.md and CLAUDE.md (new hook,
app options), THIRD_PARTY_NOTICES (Qt WebEngine from the system), STEAM-FRAME.md (results),
PROGRESS.md.
Known limitation: the bundled-Qt package does not support WebEngine in this iteration.
## Log
### 2026-10-06: iteration 2 implemented
- Phase 1 (requirements, D41), phase 2 (design, D42–D44) and phase 3 (implementation in
seven steps, D45) of iteration 2, each approved by the user; details in the entries below.
The user allowed continuing between the steps of phase 3 without asking.
- Commits `cd62e57` (requirements) to `6cb3091` (step 7). All 13 automated tests pass in
`container-test.sh`; phase 4 (manual tests against HA and on the Frame) is next.
- Session ended here at the user's request.
### 2026-10-06: iteration 1 closed
- Frame checks 3 and 4 (settings error when started by SteamVR) skipped: the user had no
access to the device. Postponed to the next session on the Frame; listed in CLAUDE.md
under "Not yet verified on a Steam Frame".
- Phase 5 (assessment) of iteration 1: the user agreed to close it with checks 3 and 4
postponed and to start iteration 2.
- Requirements of iteration 2 approved (see "Next steps"). User's choices: a dummy sensor
instead of a real one (D41); TLS, `--desktop-novr` and `--ha-remove` in this iteration.
Since nothing in iteration 2 needs SteamVR (`--desktop` doesn't call `VR_Init`), the
MQTT/HA checks run from the PC; the Frame only for the package.
- Design of iteration 2 written (DESIGN.md). User's decisions: libmosquitto's own thread
(D42), app commands in the platform for `--ha-remove` (D43), committed TLS test
certificates (D44). Design approved; phase 3 started.
- Step 1: `cmake/Mosquitto.cmake` (pkg-config; `FRAMEASSISTANT_SHIP_MOSQUITTO=ON` in all
package presets installs `lib/libmosquitto.so.1` and Debian's copyright file, which already
contains the EPL-2.0/EDL-1.0 texts); `libmosquitto-dev(:arm64)` in the build containers,
`mosquitto` in the test container; THIRD_PARTY_NOTICES entry. Checked: host configure finds
2.1.2; `container-test.sh` 8/8, no Vulkan validation messages; the cross build stages
2.0.21 (aarch64) and the copyright file. The binary doesn't need the library yet (nothing
calls it; `--as-needed`), which changes in step 2. On the host, the GUI tests can't run in
this session (no display).
- User: no need to ask for permission to continue between the steps of this phase.
- Step 2: `MqttClient` interface, `MosquittoClient`, `tst_mqtt` (12 cases against a local
mosquitto). Reading libmosquitto 2.0.21's source showed that `mosquitto_loop_start()`
doesn't fit (blocking DNS in the caller, silent thread end on TLS/protocol errors); the
user chose an own `std::thread` (D45, amends D42). Findings: a failed first connect is
retried; libmosquitto loses the DNS error code; `strerror()` is localised (German on the
dev host), so errors use `strerrordesc_np()`. Test pitfall: `QTRY_*` evaluates its
expression once more at the end, so it must not consume data (`readAll…()`). Checked:
host 3 × 12/12; `container-test.sh` 9/9 (Debian's libmosquitto 2.0.21), no Vulkan
validation messages; cross build: private Qt symbols only `QVulkan*`, at most
`GLIBC_2.34`; the app binary links libmosquitto only from step 4 on (first use).
- Step 3: `HaEntity` (description, discovery component, throttling, availability),
`HaSensor`, `HaDevice` (discovery payload, publish all after connect and after HA's birth
message with random delay, last will, `removeFromHa()`), `FakeMqttClient` for tests.
`tst_haentity` 11/11, `tst_hadevice` 14/14 on the host.
- Step 4: platform mode `DesktopNoVr` (`--desktop-novr`, behaves like `--desktop` for now;
`tst_surfaces` `surfaces_desktop_novr`); `Dashboard` creates client and device in
`loading()`, connects in `started()`, feeds the dummy sensor every 10 s and disconnects
gracefully on `aboutToQuit`; `ConnectionStatus` QML singleton and the status card in
`Main.qml` (UI-1). Checked:
- end to end against a local mosquitto (`--desktop-novr`, watched with `mosquitto_sub`):
discovery, `online`, state and availability in the designed order; broker restart →
"connection lost", "connection refused", reconnect and full republish; `kill -9` → will
`offline`. (mosquitto also sends the will when it shuts down itself.)
- screenshots under Xvfb: unconfigured (snapshot), connected, not connected with last
error.
- `container-test.sh` 12/12, no Vulkan validation messages; cross build links
`libmosquitto.so.1`, at most `GLIBC_2.34`, private Qt symbols only `QVulkan*`.
- Not checked yet: the graceful exit through `aboutToQuit` in the app (only in
`tst_mqtt`); it needs a real window close or "Quit app" in SteamVR (manual test).
- Step 5: TLS in `MosquittoClient` (`ca_file` or the system's CAs, client certificate,
checks before connecting); test certificates in `tests/data/` (D44); 9 TLS cases in
`tst_mqtt`. Findings:
- **SIGPIPE crash:** with TLS, a write to a connection the broker closed killed the
process (OpenSSL uses `write()`). Fixed by blocking SIGPIPE in the client thread.
- libmosquitto's `tls_set()` needs a CA file or directory, so with system CAs only
`MOSQ_OPT_TLS_USE_OS_CERTS` is set, and a client certificate needs `ca_file`
(documented, reported as a settings error).
- A failed handshake returns only "TLS error"/"Protocol error"; the reason is in
libmosquitto's log, which is now appended.
- Checked: host 3 × 21/21; `container-test.sh` 12/12 (libmosquitto 2.0.21), no Vulkan
validation messages.
- Still open: which CA store the Frame's OpenSSL uses by default (for `ca_file` empty);
check on the device.
- Step 6: app commands in the platform (D43: second option list, `QCoreApplication`, mode
`Command`, hook `runCommand()`; `tst_surfaces` `surfaces_command`); `--ha-remove` in the
app; `uninstall.sh` stops the app, runs `--ha-remove` (failure doesn't stop it) and
unregisters. Checked end to end against a local mosquitto, without a display: retained
discovery, availability, state and entity availability before, none after (exit 0);
broker down → exit 1 after 10 s; missing settings → exit 1.
- Step 7: `package-release.sh` checks the symbol versions of every shipped ELF file against
the Frame (glibc 2.39, OpenSSL 3.2.1; D35); a test with a too low limit names the files
and refuses the package. Building the bundled package showed two problems, both fixed:
the check read version *definitions* too (false positives from libssl), and the bundler
shipped Debian's OpenSSL 3.5 for libmosquitto (now excluded; SteamOS provides OpenSSL).
Checked:
- system-Qt package 5.0 MB (`lib/libmosquitto.so.1`, its copyright file; the binary is
12 MB, 11 MB of it debug info); bundled package 45 MB, no OpenSSL, libmosquitto once;
both need at most `GLIBC_2.38`, `OPENSSL_3.0.0`.
- generated `uninstall.sh` with a stub binary and a stub `pkill`: stop, `--ha-remove`,
unregister; a failing `--ha-remove` prints the warning and still unregisters.
- `container-test.sh` 13/13, no Vulkan validation messages.
### 2026-10-05: requirements approved
- User approved REQUIREMENTS.md including the priorities (D13). Phase 2 started.
- Q4 decided: libmosquitto, shipped in the package (D14). Qt MQTT ruled out: private Qt API
requires the exact Qt version. Checked on the Frame: no MQTT library installed; libssl/
libcrypto 3 and libcjson 1 present.
- HA discovery: own small layer on QJson, device-based discovery (D15); hacpp and the
embedded libraries rejected.
- Checked on the Frame which Qt modules, plugins and audio/network tools exist; library list
per requirement in REQUIREMENTS.md Appendix C. Notable: Qt WebEngine and Qt WebSockets are
present (relevant for Q2/Q3).
- libmosquitto ABI check: Debian's 2.0.21 needs at most `OPENSSL_3.0.0`/`GLIBC_2.34`; the
Frame has OpenSSL 3.2.1 and glibc 2.39. No libcjson dependency. D14 holds.
- Frame test of the remaining iteration 1: started by SteamVR with log file and settings,
Vulkan rendering, tab usable in VR (checks 1, 2 ✓); checks 3, 4 left for the next session.
- DSH-1 closed (D40): the bundling experiment failed (sandbox, ICU resources, Vulkan abort,
empty page with OpenGL and in a desktop window; details in STEAM-FRAME.md). At the user's
decision the "HA Dashboard" tab, `HaWebUi`, the WebEngine build dependency and the
bundler's WebEngine support were removed (reverts of the corresponding commits). Qt WebView
is no alternative (WebEngine backend on Linux). `homeassistant.url` stays reserved for
DSH-2, where a native dashboard is the candidate.
- Phase 4 (testing on the Frame), first findings:
- `deploy-to-frame.sh` exited silently without a bundled package in `dist/` (fixed).
- The system-Qt package doesn't start: the Frame's Qt WebEngine is 6.7.0 (rest of Qt 6.8.0)
and built against FFmpeg 6, which the Frame lacks (STEAM-FRAME.md). DSH-1's out-of-the-box
variant therefore fails on the Frame; and since the app links WebEngine, the whole app
fails.
- At the user's request, experiment B: the bundled package now ships Qt WebEngine (Debian
6.8.2, own FFmpeg inside) with its helper process and resources: 141 MB archive, 352 MB
unpacked. Needs `GLIBC_2.38` and `GLIBCXX_3.4.32` from the Frame. Test pending.
- Other options noted: load WebEngine only at runtime so the app survives without it; a
native dashboard (DSH-2, Q2b). Qt WebView is no alternative (on Linux its backend is
WebEngine; native views are separate windows, not rendered into the texture).
- Phase 3 of implementation iteration 1:
- Platform: `prepare()` hook and app-specific command-line options, tested in
`tst_surfaces` (incl. `surfaces_prepare_fail`).
- Settings file (`src/config/`, new library `frameassistant_lib`), `--config`; `tst_config`
(37 cases). `config.example.json` in the package; README.txt explains the set-up.
- "HA Dashboard" tab: `WebEngineView`, `HaWebUi` with a persistent profile, WebEngine
initialisation in `main()`. Host: desktop mode loads a local test page, profile data
lands in `~/.local/share/frameassistant/webengine`; snapshot renders both tabs.
- Build containers with Qt WebEngine. Cross build: the aarch64 binary needs no private Qt
symbols, at most `Qt_6.8` and `GLIBC_2.34`. Container tests 8/8 (incl. Vulkan with
validation layers, no messages).
- Snags: moc misses classes in files with raw strings containing `//` (CLAUDE.md);
generated QML type registration includes headers by file name; Qt ≥ 6.9 hints at
`WebEngineProfilePrototype` (not in the Frame's 6.8). Deviations: `HaWebUi` lives in the
app, not the library; the password warning needs a password in the file.
- Log file for SteamVR-started instances: the manifest passes `--log-file
logs/frame-assistant.log` (user's choice among: default log path in the app, manifest
argument, document only).
- Not verified yet: WebEngine with a URL in the off-screen pipeline (Frame, phase 4); the
bundled package (not supported for WebEngine in this iteration).
- Plan for implementation iteration 1 approved; phase 3 started.
- Plan for implementation iteration 1 written; new tab "HA Dashboard" (D38, D39).
- MVP implemented in four iterations; DSH-1 first, at the user's wish (D37).
- NFR-3 measurement method, manual in the MVP (D36). MVP details done.
- Shipping libmosquitto (D35).
- Configuration file format (D34).
- MVP notification: `notify` entity with first line as title; JSON command later (D33).
- HA MQTT interface: ids (D31), topics, retain, availability (D32), MVP entity list.
- Test strategy in four levels (D30). High-level architecture done.
- Threads: MQTT client in its own thread, everything else incl. OpenVR in the main thread (D29).
- DSH-1 split: web UI out-of-the-box test in the MVP, full dashboard as DSH-2 (D28).
- Overlays: two kinds (D25), one overlay for all notifications (D26), MVP placement "spawn in
front, then stay" (D27). Researched notification UI in VR (DESIGN.md, Appendix A). Found:
premultiplied-alpha flag missing in the platform.
- HA channels: MQTT for everything device-related; dashboard channel decided with Q2/Q3 (D24).
- Entities: one class per HA entity type, native types preferred, own service only as fallback
(D23). Found: MQTT `notify` drops the title.
- Start-up hook `prepare()` in the platform (D21); modes, `--desktop-novr` and `--config`
(D22).
- New requirement UI-1: status display in the main tab (D20).
- The user installed `mosquitto` on the dev host, so native builds can use libmosquitto.
- Started docs/DESIGN.md: one process (D16), three layers (D17), HA-centric object classes
(D18). HA side: FrameAssistant needs only stock HA; how the user wraps commands (scripts,
pyscript) is up to them (NFR-6 unchanged).
- Agreed to settle the high-level architecture first (list under "Next steps"), before
component details.
- Stay with the Debian build container: an Arch container wouldn't match SteamOS's own package
versions (private repos, rolling Arch) and has no arm64 multiarch for cross builds.
### 2026-10-04/05: project set-up and requirements
- Copied the vrdashboard template. Removed what the app doesn't need:
- `examples/` (diagnostics app) and the `VRDASH_BUILD_EXAMPLES` option
- the Frame test package (`scripts/package-frame-test.sh`, `docs/FRAME-TESTING.md`,
screenshots)
- the demo UI (`src/qml/DemoPage.qml`)
`Main.qml` shows a placeholder card.
- Named the app **FrameAssistant**, "Steam Frame companion for Home Assistant":
- CMake project `frameassistant`, executable `frame-assistant`
- app key `net.matzman.frameassistant`, tab "Frame Assistant", QML module `FrameAssistant`
Names rejected:
- "Steam…" and "…Home Assistant": trademarks
- "HASS…": the user dislikes it, because "Hass" is German for hate.
- Agreed working rules (in CLAUDE.md):
- explicit permission for everything; ask instead of assuming
- iterative five-phase cycle; an informal "ok" approves a phase
- document everything in Markdown in `docs/`
- `git init`. Baseline commit `4a5b090`.
- Requirements engineering (several question rounds with the user). The results are in
REQUIREMENTS.md:
- decisions D1–D12
- 51 requirements
- research on OpenVR data sources (Appendix A)
- Checks: `native-debug` builds, `ctest` passes 6/6 (Vulkan surface test skipped, no Vulkan on
this host), `--snapshot` renders.