FrameAssistant/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Working rules (agreed with the user, always apply)

• Proper software engineering: follow established procedures and processes, not ad-hoc hacking.
• Nothing without explicit permission. Propose first, then act only on the user's go-ahead. Permission covers only what was asked: no extra changes, renames, rewrites or "while I'm at it" fixes. Suggest those instead.
• Ask, don't assume. When a requirement, decision or detail is unclear, ask for clarification before proceeding. Don't fill gaps with your own defaults.
• Fast, iterative development cycle. Each iteration runs through these phases in order, and each phase ends with the user's approval before the next starts:
  1. Requirements engineering: what is needed and why, with acceptance criteria.
  2. Planning and design: how it will be built, alternatives, trade-offs.
  3. Implementation: only what the approved design covers.
  4. Testing: verify against the acceptance criteria and report the results faithfully.
  5. Assessment and decision: review the outcome together and decide how to proceed.
• Keep iterations small, so each cycle stays fast.
• An informal "ok" from the user is enough to approve a phase.
• Document everything: requirements, designs, decisions, test results. Documentation is direct and concise, but complete. It is Markdown and lives in docs/. Whenever something changes, also update the supporting documents (README, CLAUDE.md, other docs) to match.

What this is

FrameAssistant, "Steam Frame companion for Home Assistant" (in the spirit of HASS.Agent): it reports headset/environment sensors to Home Assistant, offers services Home Assistant can call (e.g. notifications in the headset), and shows a Home Assistant dashboard as a SteamVR dashboard tab. Status: the overlay platform works; the Home Assistant parts are being built (iteration 1, settings file, is done; the web UI tab was dropped, see REQUIREMENTS.md D40; implementation iteration 2: MQTT and HA device).

Start every session by reading docs/PROGRESS.md (current state, next steps, log).

User-facing docs: README.md (quick start, development, CLI), docs/REQUIREMENTS.md (requirements, open questions, decisions), docs/DESIGN.md (design of the HA parts and its decisions), docs/PROGRESS.md, docs/ARCHITECTURE.md, docs/STEAM-FRAME.md. Keep them in sync with changes. This file holds what a developer needs on top of that.

Platform: a SteamVR dashboard overlay, built from the author's vrdashboard template, targeting the Steam Frame (aarch64, SteamOS) and x86_64 Linux. Stack: C++20, Qt 6 Quick (QML, ≥ 6.5) and OpenVR IVROverlay. Two render backends behind one interface, chosen at runtime with --graphics-api vulkan|opengl.

• Naming: the app is FrameAssistant (project(frameassistant), executable frame-assistant, app key net.matzman.frameassistant, tab "Frame Assistant"). The overlay platform keeps its template names (vrdashboard_core, VRDASH_*, vrdash_*, DashboardApplication); don't rename them. Avoid "Steam", "Home Assistant" and "HASS" in product names (trademarks; the user dislikes "HASS").
• Keep it simple: no FetchContent dependency, no plugin system.
• Vulkan is the default where it is compiled in: on the Frame it is about 1.9× faster than OpenGL, which runs there through zink.
• Without an explicit --graphics-api, a Vulkan start-up failure falls back to OpenGL (startBackend() in src/app/DashboardApplication.cpp). With an explicit choice there is no fallback.

Commands

cmake --preset native-debug && cmake --build --preset native-debug   # system Qt
ctest --preset native-debug                                          # all tests
build/native-debug/bin/tst_inputbridge clickUsesGlCoordinates        # one QtTest function
build/native-debug/bin/frame-assistant --desktop       # UI preview in a window, no SteamVR
build/native-debug/bin/frame-assistant --desktop-novr  # same, and never SteamVR (from iteration 3 on the difference)
build/native-debug/bin/frame-assistant --snapshot x.png  # one frame via the real offscreen path
scripts/container-build.sh cross    # x86_64 → aarch64 against the Frame's system Qt (default)
scripts/container-build.sh native   # same for the host arch (use on the Frame)
scripts/container-build.sh cross-bundle   # fallback: self-contained, Qt bundled (also native-bundle)
scripts/container-build.sh cross shell    # debug inside the cross container
scripts/container-test.sh           # all tests in Debian incl. Vulkan + validation layers
VK_DRIVER=lvp scripts/container-test.sh  # same on lavapipe (software Vulkan)
build/native-debug/bin/frame-assistant --graphics-api vulkan --benchmark 600  # frame ms incl. GPU

• Use --snapshot and then read the PNGs to check UI or rendering changes without a headset. With several surfaces it writes <name>-<id>.png per surface (-0 for an empty id). Snapshot and benchmark don't load the settings.
• Overlay and both desktop modes need a settings file and connect to its MQTT broker; pass a test one with --config <path> (e.g. mosquitto -p 18883 and "mqtt": {"host": "127.0.0.1", "port": 18883}).
• tst_render runs once per backend (VRDASH_TEST_API, separate CTest entries, because the Qt graphics API is process-global). It QSKIPs without a GPU context or when Vulkan isn't compiled in.
• tst_inputbridge runs with QT_QPA_PLATFORM=offscreen.
• The Vulkan backend is optional (VRDASH_VULKAN=AUTO|ON|OFF). It needs the Vulkan headers and a Qt built with Vulkan. Hosts without headers build OpenGL only.
• Any Vulkan change: run scripts/container-test.sh. It must report "No Vulkan validation messages". VRDASH_VULKAN_VALIDATION=1 enables the layers in any run.
• Cross bundles can be smoke-tested on x86 with qemu-aarch64 plus Xvfb, inside a Debian container that has the arm64 X11/GL runtime libraries and no Qt.

Code style

• C++ follows the Google C++ style, enforced by .clang-format (BasedOnStyle: Google): 2-space indent, 80 columns, opening braces on the same line everywhere (functions too), int* p, sorted include blocks.
  • Run clang-format -i $(find src tests -name '*.cpp' -o -name '*.h'). clang-format --dry-run --Werror … must be clean.
  • Lines may exceed 80 columns only for user-visible strings (log/error messages).
  • Never let clang-format split a trailing comment: put the comment on its own line above.
  • src/AppConfig.h.in is formatted by hand.
• Naming stays Qt style, not Google naming (CamelCase classes, camelCase methods, m_ members, k constants), so overrides, signals and QML-visible names match Qt's APIs.
• QML, CMake and shell scripts keep their usual conventions (QML: 4 spaces).

Architecture

The data flow is QML → off-screen texture → SteamVR overlay. SteamVR overlay events flow back into Qt events.

• src/app/DashboardApplication is the base class of every app (in vrdashboard_core): CLI, logging, SteamVR connection, backend selection, surfaces, controller navigation, and the modes (overlay, --desktop, --desktop-novr, --snapshot, --benchmark, manifest install, app commands).

  • Inheritance for the app, components for the surfaces (the user's decision, after trying composition): the app derives from DashboardApplication. In its constructor it configures (addSurface, optional setAppKey/setDisplayName) and passes its own command-line options and its commands (two QCommandLineOption lists, D38, D43) to the base constructor. It overrides the hooks prepare(QString*) (first; mode and app key known, no engine/surfaces/SteamVR yet; returning false ends exec() with exit code 1), loading() (backends + shared engine exist, QML not loaded), started() (everything runs) and runCommand(name) (mode Command, after prepare(), nothing else exists). In the hooks, protected accessors give the running state: mode(), appKey(), appName(), isOptionSet(), optionValue(), engine(), surfaces(), surface() (primary), surface(id), vrRuntime(), navigation(). The app is src/Dashboard.h/.cpp.
  • The base constructor creates the QGuiApplication (QCoreApplication for manifest and app commands: decided from the raw argv before parsing), so derived members can be QObjects with timers.
  • All QML and surfaces are torn down at the end of exec(), while the derived object and its members (e.g. QML singletons) still exist.

• src/app/Surface is one view: its QML root (from the app's module), renderer, render backend and overlay. All surfaces share one QQmlEngine.

  • It's a component: configured via SurfaceConfig (id, qmlRootType, displayName, textureSize, thumbnail), with signals visibilityChanged/activated. Never subclassed.
  • Overlay key = <app key>.<id>, or the app key itself for an empty id. The app key (manifest, IdentifyApplication) is one per process; --app-key/--app-name override it.
  • Backends: the first surface decides the API (incl. the Vulkan→OpenGL fallback); the others use the same API, because QQuickWindow::setGraphicsApi is process-wide.
  • Controller navigation sends keys to the active surface: the one last opened or pointed at (Surface::activated), if visible; else any visible one.
  • vr/Overlay is the overlay base (input, visibility, frames). DashboardOverlay is the only kind so far; a WorldOverlay would be another subclass (see "Planned extension").
  • vrdash_add_dashboard() (cmake/Dashboard.cmake) defines a dashboard executable: QML module, bytecode-only QML, OpenVR lib, install, registration for Qt bundling. The QML module's URI (QML_URI) is written down only there: a generated source file registers it as the app's default module (DashboardApplication::setDefaultQmlModule at static-init time). setQmlModule() exists only to load from another module.

• render/RenderBackend.h is the only place graphics-API code may live:

  • context/device setup, texture allocation, layout handling, vr::Texture_t + UV bounds for SetOverlayTexture, and CPU readback.
  • The header documents the call order (initialize → configureWindow → attachWindow → createRenderTarget → per frame beginFrame/finishFrame → detachWindow).
  • createRenderBackend() is the factory.
  • QQuickWindow::setGraphicsApi() must run before any QQuickWindow exists (see startBackend() in DashboardApplication.cpp).
  • In overlay mode the backend is initialized after VR_Init, because Vulkan needs VrRuntime::renderInterop(): the instance/device extensions SteamVR requires and the HMD's VkPhysicalDevice.

• VulkanRenderBackend:

  • Qt Quick creates the VkDevice; the backend owns the VkImage.
  • OpenVR requires TRANSFER_SRC_OPTIMAL plus TRANSFER_SRC|SAMPLED usage. Qt leaves the target in COLOR_ATTACHMENT_OPTIMAL, so finishFrame()/beginFrame() transition between the two with small synchronous submits. QQuickRenderControl offscreen frames already wait for the GPU.
  • Everything created on Qt's device must be freed in detachWindow(), before QQuickRenderControl destroys the device.
  • The instance requests API version ≤ 1.3. QVulkanInstance's 1.0 default made Qt itself trigger validation errors.

• render/OffscreenQuickRenderer is API-agnostic:

  • Drives QQuickRenderControl and renders on demand (on scene change or animation), capped at --max-fps.
  • Pacing uses an absolute schedule from frame starts. Waiting a fixed interval after each frame cost about 10% on the Frame and about 50% under load; tst_render pacingHoldsFrameRateWithSlowFrames guards this.
  • Each frame goes to a FrameSink callback while the context is current, which OpenGL SetOverlayTexture requires. Surface sets it to its overlay.

• vr/VrRuntime handles VR_Init(VRApplication_Overlay) and polls events on a Qt timer. It routes per-overlay events to handlers registered by overlay handle.

  • VR_Shutdown must run before the graphics device is destroyed (openvr.h: "Be sure to call OpenVR_Shutdown before destroying these resources").
  • Order in runOverlay() (DashboardApplication.cpp): the surfaces (renderers, backends) are members and die after it returns; VrRuntime is a local, so it shuts down first; an OverlayGuard declared after it destroys the overlays before that. Otherwise SteamVR's client frees its query pool on a dead VkDevice at exit, which crashed on the Frame.

• vr/DashboardOverlay is CreateDashboardOverlay plus the tab thumbnail. Rendering follows VREvent_OverlayShown/Hidden, not the dashboard's open/closed state. Users can float or pin dashboard overlays so they stay visible in games; tying rendering to the dashboard would freeze them there.

• input/OverlayInputBridge turns OpenVR overlay events into Qt events, so QML never knows about VR:

  • Mouse coordinates arrive in GL convention (y from the bottom) and must be flipped.
  • Smooth scroll becomes QWheelEvent (with a sub-pixel accumulator).
  • When a text item gets active focus, it opens the SteamVR keyboard (KeyboardFlag_Minimal: characters stream in; \b and \n mean Backspace and Return).
  • The offscreen window gets an explicit FocusIn event (in OffscreenQuickRenderer), otherwise activeFocusItem stays null.

• input/ControllerNavigation turns SteamVR Input actions (resources/input/actions.json, default bindings for frame_controller) into Tab/Backtab/arrow/Space/Escape key events. Qt Quick's own focus chain does the rest; no QML glue is needed.

• QML singletons owned by the app: keep them as members of the app class. A singleton must not be default-constructible, or QML builds its own instance instead of calling create() (e.g. give the constructor a required QObject* parent and init it {nullptr}).

• App code layout: logic without SteamVR or rendering goes into the static library frameassistant_lib (src/config/, …), which the app and its tests link. QML types (QML_ELEMENT) go into QML_SOURCES of vrdash_add_dashboard(): they must be part of the QML module. The generated type registration includes their headers by file name only, so their directory is on the app's include path.

• Settings (src/config/Config) are loaded once in Dashboard::prepare() (overlay and both desktop modes) and passed on as an immutable value (DESIGN.md, D19).

• MQTT and HA (DESIGN.md, "Iteration 2"): Dashboard::loading() creates MosquittoClient (src/mqtt/) and HaDevice (src/ha/), started() connects; aboutToQuit disconnects gracefully (publishes "offline"). The HA layer only knows the MqttClient interface; tests use tests/FakeMqttClient.h. MosquittoClient's callbacks run in its own std::thread and may only emit signals with copied values (D45).

• Error texts are English regardless of the locale: strerror() is translated (German on the dev host), so use strerrordesc_np().

• QTRY_* evaluates its expression once more at the end: never consume data in it (e.g. readAllStandardOutput()); accumulate into a variable instead.

• No Qt WebEngine (REQUIREMENTS.md, D40): the Frame's system WebEngine can't be loaded, and bundling our own failed (STEAM-FRAME.md). Don't link it: the whole app would fail to start on the Frame.

• Check Frame libraries by loading, not by file: before relying on a system library, check ldd <lib> | grep "not found" and that its package version matches the rest of Qt (pacman -Q | grep qt6-).

• moc and raw string literals: a raw string containing // (e.g. a URL) makes moc treat the rest as a comment; it then misses Q_OBJECT classes in that file (link error "vtable for …"). In moc'ed files, write such text as normal string literals (tst_config uses JSON with ' quotes and converts them).

• Publish per-frame values (frame stats etc.) to QML at most about once per second. Binding per-frame values in QML re-triggers the on-demand renderer forever.

• The executable is frame-assistant (VRDASH_EXECUTABLE_NAME), never vrdashboard. SteamVR identifies processes by executable name. With vrdashboard, the Frame's SteamVR treated our process as its own dashboard (watchdog, "VRDashboard process failed to start 10 times"). CMake rejects names of SteamVR processes.

• The version is in project() in the root CMakeLists.txt (currently 0.1.0); release packages are named after it.

• App identity (app key, name, executable name, texture size) lives in CMake cache variables in the root CMakeLists.txt and is generated into AppConfig.h.

Planned extension: world overlay (not implemented)

Several surfaces per process exist (tested by tests/tst_surfaces). Still to come, for a notification app among others: a WorldOverlay (subclass of vr/Overlay, created via CreateOverlay) with configurable placement and optional interactivity. Placement modes: room, HUD, hand, and "spawn in front then stay". Interactivity uses VROverlayFlags_MakeOverlaysInteractiveIfVisible, which takes the system laser while visible. SurfaceConfig would get a kind (dashboard/world) plus placement; the app shows and hides world surfaces itself. Keep the teardown rule: overlays, then SteamVR, then backends.

Build layout and deployment

• The build tree mirrors the install layout (bin/, lib/, share/frameassistant/). The binary finds its data at <exe>/../share/frameassistant, and libopenvr_api.so through RPATH $ORIGIN/../lib.
• manifest.vrmanifest uses paths relative to itself, so --install-manifest works from both the build tree and an install.
• The manifest passes --log-file logs/<exe>.log (arguments), so SteamVR-started instances log to the install folder (or build/<preset>/logs/ from the build tree).
• libmosquitto (MQTT, D14/D35) comes from pkg-config (cmake/Mosquitto.cmake). The Frame has none, so FRAMEASSISTANT_SHIP_MOSQUITTO=ON (all package presets) installs it into lib/ as libmosquitto.so.1, with Debian's copyright file (it contains the EPL-2.0/EDL-1.0 texts), and adds it to VRDASH_BUNDLE_EXCLUDE. Include it after BundleQt, which sets that list.
• cmake/OpenVR.cmake downloads only openvr.h, LICENSE and the matching bin/linux64|linuxarm64/libopenvr_api.so into third_party/openvr-<ver>/ (gitignored).
• Deploying to the Frame uses its system Qt by default. Bundling Qt is the fallback for when a SteamOS update ships a Qt that no longer fits.
• VRDASH_DEPLOY_QT=ON (presets native-bundle and cross-arm64-bundle) runs cmake/BundleQt.cmake and bundle-install.cmake.in at install time:
  • qmlimportscanner collects the QML modules.
  • file(GET_RUNTIME_DEPENDENCIES) with the target's objdump collects the libraries.
  • patchelf sets relative RPATHs, and the script writes bin/qt.conf.
  • Qt's own qt_generate_deploy_* is deliberately not used: it skips cross-compiled Linux targets, and with a /usr Qt it would bundle glibc.
• VRDASH_BUNDLE_EXCLUDE must keep excluding the GPU/display stack (libGL/EGL/GLX, libdrm/gbm, Vulkan, Wayland, X11/xcb, libffi, glibc, libstdc++). These must come from SteamOS, or GPU access breaks.
• Release packages (scripts/package-release.sh [--bundle-qt]) contain only the app, install.sh (install/update: stops a running instance, registers, launches), uninstall.sh and README.txt. deploy-to-frame.sh rsyncs the newest one to the Frame and runs its install.sh.
• Licenses: MIT (LICENSE); THIRD_PARTY_NOTICES.md and OpenVR's license are installed to share/doc/frameassistant/. Bundled builds also get the Debian copyright files of every bundled library plus the referenced common license texts (collected via dpkg -S in bundle-install.cmake.in, after resolving paths: trixie registers /usr/lib paths).
• Qt bundling runs once for all dashboards (vrdash_bundle_qt() at the end of the root CMakeLists.txt). Bundling per executable made the second run see each library twice.
• System-Qt builds (the default; presets cross-arm64 and native-systemqt, container-build.sh cross|native): no Qt bundle, so the package is about 3 MB instead of 46 MB.
  • VRDASH_QML_BYTECODE_ONLY=ON keeps qmlcachegen's version-locked C++ out of the binary.
  • The binary then needs only stable Qt_6 symbols, qt_version_tag@Qt_6.8 (Qt ≥ 6.8), and Qt's Vulkan function wrappers. Those are tagged Qt_6_PRIVATE_API but generated from the Vulkan spec.
  • Verified: the Debian-built binary (Qt 6.8.2) runs on Arch's Qt 6.8.0 (Arch Linux Archive snapshot 2024-10-20, image archlinux:base-20241020.0.271562), with both backends.
  • Check with objdump -T bin/frame-assistant | grep Qt_6_PRIVATE_API: only QVulkan* may appear.
• -mno-direct-extern-access is added whenever the compiler supports it (root CMakeLists.txt).
  • Arch/SteamOS build Qt with protected symbols, so consumers must not use copy relocations. Debian's Qt doesn't pass that flag on.
  • Without it, an x86-64 Debian build fails against Arch's Qt (GNU_PROPERTY_1_NEEDED_INDIRECT_EXTERN_ACCESS). aarch64 GCC has no such flag, and arm64 PIE never uses copy relocations.
• The Frame's /etc/pacman.conf points to private Valve repositories (URLs marked "DO_NOT_SHARE"). Never put those URLs into this repo or into scripts.
• Cross builds use Debian multiarch:
  • host (amd64) Qt tools via QT_HOST_PATH=/usr, target Qt from /usr/lib/aarch64-linux-gnu.
  • Cross presets turn tests off.

Verified on the Steam Frame

• Over SSH, run.sh opengl finds DISPLAY=:0, VR_Init succeeds, and the bundled Qt starts (2026-10-04, SteamOS holo 0.4.3, system Qt 6.8.0).
• OpenGL on the Frame runs through zink (GL on Vulkan) on Turnip/Adreno 750. There is no native GL driver, so the Vulkan backend avoids a translation layer there.
• Both backends work in SteamVR: display orientation (no texture flip needed for either backend), laser, scroll direction, keyboard and D-pad navigation (SteamVR Input actions arrive while the dashboard is open) are all OK.
• Floating works: a floated/pinned dashboard tab keeps rendering in games and stays interactive.
• Started by SteamVR itself (install.sh --launch, auto-start), the app finds a working display connection; no QT_QPA_PLATFORM override is needed.
• The system-Qt package works on the Frame (2026-10-04): the Debian-built binary (Qt 6.8.2) runs on SteamOS's Qt 6.8.0 with both backends. The package contains no Qt, so it can only have used the system Qt.
• Vulkan now exits cleanly via "Quit app" (the OpenVR shutdown-order fix is confirmed).
• Install/launch works end to end (2026-10-04): install.sh registers both test overlays, SteamVR knows both keys, --launch starts them, they auto-start again after a reboot, and uninstall.sh unregisters them.
• SteamVR on the Frame reads the program path from binary_path_linux_arm (its own manifests in /opt/steamvr use it). Entries with only binary_path_linux are skipped ("must specify binary_path for launch_type binary"), and the key then counts as unknown. resources/CMakeLists.txt picks the key from the target architecture.
• SteamVR on the Frame refuses VRApplication_Utility. VR_Init() always queries IVRSystem, which fails with VRInitError_Init_NotAvailableToUtilityApps (123). Manifest commands therefore use VRApplication_Background, which needs a running SteamVR and never starts it. Don't switch back to Utility.
• benchmark.sh on the Frame (600 frames, 1600×1000, incl. GPU wait): Vulkan 1.01 ms avg (982 fps), OpenGL/zink 1.89 ms avg (528 fps), so Vulkan is about 1.9× faster.
• After the pacing fix both run at 60 fps in VR. The (since removed) diagnostics example's frame time (Vulkan about 2.4 ms, GL about 1.2 ms) isn't comparable, because GL leaves out its asynchronous GPU work.
• Before the pacing fix, Vulkan showed 55 fps and GL 60. Cause: the render time was added to the frame interval, and GL's frame time leaves out its asynchronous GPU work.

Not yet verified on a Steam Frame

Check these when testing on the device, and update this list:

• scripts/deploy-to-frame.sh and the release package (install.sh install/update, uninstall.sh).
• The restructured app code (base class DashboardApplication, Surface, Overlay base, active-surface navigation) has only been tested off the device so far.
• Several dashboard tabs from one process (works in tst_surfaces, not tried in SteamVR).
• Settings errors when started by SteamVR: the app exits and the log names the error (e.g. missing device.name); covered by tst_config off the device (iteration 1 checks 3, 4).