FrameAssistant/docs/ARCHITECTURE.md

Architecture

A dashboard is one process. It renders QML scenes off-screen into textures and hands them to SteamVR as overlays. Input flows the other way: SteamVR reports laser, scrolling, keyboard and controller actions, and the app turns them into ordinary Qt events. The QML never knows it is in VR, which is why it also runs in a plain desktop window (--desktop).

Structure

flowchart LR
    app["<b>Your app</b><br/>class Dashboard : DashboardApplication<br/>members, loading() / started()"]
    subgraph base["DashboardApplication"]
        direction TB
        engine["QQmlEngine<br/>(shared by all surfaces)"]
        runtime["VrRuntime<br/>SteamVR connection"]
        nav["ControllerNavigation"]
    end
    subgraph surfaces["Surfaces: one per view"]
        direction TB
        s1["<b>Surface 'main'</b><br/>QML root → Renderer → RenderBackend<br/>DashboardOverlay + InputBridge"]
        s2["<b>Surface 'settings'</b> (optional)<br/>same parts"]
    end
    steamvr(["SteamVR"])
    app -- "inherits" --> base
    base -- "creates, drives" --> surfaces
    surfaces -- "textures" --> steamvr
    steamvr -- "laser, scroll, keyboard" --> surfaces
    steamvr -- "controller actions" --> nav
    nav -- "keys to the active surface" --> surfaces

DashboardApplication (src/app/DashboardApplication.*) is the base class of every app. It owns everything that isn't app-specific: command line, logging, the SteamVR connection, choosing the render backend, the surfaces, controller navigation, the auxiliary modes, and the order in which all of that is created and destroyed.

Your app derives from it:

• Constructor: configuration (addSurface(), optionally setAppKey(), setDisplayName(), setQuickStyle(), setQmlModule()). The app's own command-line options go to the base constructor as a list of QCommandLineOptions; they are parsed and listed in --help with the built-in ones, and read with isOptionSet()/optionValue().
• Hooks:
  • prepare(QString* error): first, once mode and app key are known, before the QML engine, the surfaces and SteamVR exist (not for manifest commands). Returning false ends exec() with the error and exit code 1, e.g. for invalid settings.
  • loading(): the shared QML engine, the surfaces and their render backends exist; QML isn't loaded yet. Set up what the QML needs here.
  • started(): everything runs. Connect to surfaces and input here.
• Accessors: mode(), appKey(), appName(), isOptionSet(), optionValue(), engine(), surfaces(), surface() (the first), surface(id), vrRuntime(), navigation().
• Lifetime: the base constructor creates the QGuiApplication, so your members can be QObjects with timers. All QML and surfaces are torn down at the end of exec(), while your object and its members still exist.

Surface (src/app/Surface.*) is one view: a QML root, its renderer, its render backend and its overlay. It's a component you configure (SurfaceConfig: id, QML root type, tab name, texture size, icon) and connect to (visibilityChanged, activated), never subclass.

• All surfaces share one QML engine, so models and singletons are visible in every view.
• Overlay key: <app key>.<id>, or the app key itself for an empty id.
• A surface renders only while its overlay is visible.

Render layer (src/render/):

• OffscreenQuickRenderer drives Qt Quick through QQuickRenderControl, without a window.
  • It renders on demand (scene change or animation), capped at --max-fps, on an absolute schedule so slow frames don't lower the frame rate.
• RenderBackend is the only place with graphics-API code: device setup, texture, hand-off description for SteamVR, readback.
  • VulkanRenderBackend (default): Qt creates the device on the GPU SteamVR names, with the extensions SteamVR requires. The backend owns the image and moves it to TRANSFER_SRC_OPTIMAL after each frame, the layout OpenVR wants, and back before the next.
  • OpenGLRenderBackend (fallback): on the Frame it runs through zink on Vulkan.

VR layer (src/vr/):

• VrRuntime: VR_Init as overlay app, polls SteamVR events and routes them by overlay handle. It also reports SteamVR's Vulkan requirements, and registers and launches manifests.
• Overlay: base of all overlays. It handles input routing, visibility and frame submission.
  • DashboardOverlay is the dashboard tab with its icon. A world overlay would be another subclass (planned).

Input layer (src/input/):

• OverlayInputBridge (one per overlay):
  • Laser: turned into mouse events, with the y-axis flipped (OpenVR counts from the bottom).
  • Scrolling: thumbstick scrolling becomes wheel events.
  • Keyboard: opens the SteamVR keyboard when a text field gets focus, and turns typed characters into key events.
• ControllerNavigation (one per app): SteamVR Input actions (resources/input/, rebindable in SteamVR) become Tab/arrow/Space/Escape for the active surface, i.e. the one last opened or pointed at. Qt Quick's own focus handling does the rest.

One frame

sequenceDiagram
    participant QML as QML scene
    participant R as OffscreenQuickRenderer
    participant B as RenderBackend (Vulkan)
    participant O as Overlay
    participant S as SteamVR
    QML->>R: scene changed / animation tick
    Note over R: wait for the next slot (≤ max-fps)
    R->>B: beginFrame (image back to COLOR_ATTACHMENT)
    R->>R: polish, sync, render (Qt Quick)
    R->>B: finishFrame (image → TRANSFER_SRC, GPU done)
    R->>O: frame sink: texture + bounds
    O->>S: SetOverlayTexture (SteamVR copies it)

Input

sequenceDiagram
    participant S as SteamVR
    participant V as VrRuntime
    participant I as OverlayInputBridge
    participant Q as QML (QQuickWindow)
    S->>V: overlay event (mouse, scroll, keyboard)
    V->>I: routed by overlay handle
    I->>Q: QMouseEvent / QWheelEvent / QKeyEvent
    Q-->>I: text field got focus
    I->>S: ShowKeyboardForOverlay

Start and shutdown (overlay mode)

1. Start:
   1. Call prepare(); stop with exit code 1 if it fails.
   2. Connect to SteamVR (VR_Init).
   3. Create one render backend per surface. The first surface decides the API, including a fallback to OpenGL; the others use the same API, because Qt's graphics API is process-wide.
   4. Call loading().
   5. Load each surface's QML.
   6. Create the overlays.
   7. Set up controller navigation.
   8. Call started(), then run the event loop.
2. Shutdown runs in reverse, with one rule found on the Frame: overlays first, then SteamVR, then the graphics devices. OpenVR requires shutting down before the devices it got textures from are destroyed. Otherwise its client library frees objects on a dead VkDevice at exit, which crashed on the Frame. runOverlay() encodes this order through the declaration order of its objects.

Building blocks in CMake

• vrdash_add_dashboard() (cmake/Dashboard.cmake) turns sources plus QML files into a dashboard executable. It covers:
  • the QML module, whose QML_URI becomes the app's default module through a generated source file
  • bytecode-only QML for system-Qt builds
  • libopenvr_api.so next to the executable
  • install rules and registration for Qt bundling
  • the check against SteamVR process names
• cmake/Mosquitto.cmake (app, not platform) finds libmosquitto through pkg-config. With FRAMEASSISTANT_SHIP_MOSQUITTO=ON (all package presets) it installs the library into lib/ under its soname plus Debian's copyright file, and excludes it from the Qt bundler.
• App identity (root CMakeLists.txt): project() name and version, VRDASH_APP_KEY, VRDASH_DISPLAY_NAME, VRDASH_EXECUTABLE_NAME, texture size. CMake generates from them AppConfig.h and the SteamVR manifest, with binary_path_linux_arm for ARM builds and binary_path_linux for x86_64.

Planned: world overlays

For e.g. a notification app: a WorldOverlay (another Overlay subclass, created with CreateOverlay) that the app shows and hides itself.

• Placement modes: room, HUD, hand, "spawn in front, then stay".
• Optionally operable: while visible, it takes the system laser (VROverlayFlags_MakeOverlaysInteractiveIfVisible).

The surface structure is ready for it: SurfaceConfig would gain a kind and a placement.

FrameAssistant on top

The app itself (src/Dashboard.*, src/config/, src/ha/, src/qml/) builds on this platform; its design (Home Assistant connection, entities, overlays, threads) is in DESIGN.md.