Embedding Real-Time Rendering Engines in Declarative UI Frameworks

#Abstract

Compositing a real-time rendering engine into a declarative user interface framework involves three independent lifecycles, owned by the operating system, the interface framework and the engine, which no party coordinates. This report characterises the resulting failure modes, proposes an ownership model that resolves them, and reports the cost of running a second renderer as measured on one class of mobile hardware and one engine version.

#Motivation

Applications that require registered three-dimensional content, including augmented reality, product configuration and training simulation, need an engine. Applications that require conventional interface behaviour need the platform's own controls. Building either inside the other produces a known set of deficiencies: interface elements drawn by an engine ignore accessibility settings and platform conventions, while an interface framework cannot perform the tracking and rendering an engine provides [1] [2].

#The three lifecycles

The core difficulty is that three parties may independently invalidate state the others hold.

1. The operating system may destroy and recreate the hosting context at times of its choosing, including configuration change and memory pressure.
2. The interface framework may dispose the view while the hosting context persists, and may rebuild it for reasons unrelated to whether the engine should continue.
3. The engine holds a rendering context, audio focus and possibly a camera, any of which the operating system may reclaim independently.

A correct integration defines behaviour for every combination. The combinations that produce defects are the infrequent ones, which is why the resulting failures are timing dependent and why reproductions are the dominant cost in triage.

1host context   view        engine       correct behaviour
2------------   ---------   ----------   ------------------------------------------
3alive          attached    running      steady state
4alive          detached    running      keep running, unbind surface  (list scroll)
5alive          attached    paused       resume, rebind surface
6recreated      attached    running      rebind surface, keep scene   (rotation)
7recreated      detached    paused       do nothing until reattached
8destroyed      any         any          tear down; scene is not recoverable
Copy

The row that gets implemented wrongly most often is the second. A view detaching feels like a signal to stop, and stopping is cheap to write and looks correct in testing, because a test scrolls the view away and back within a second and the reinitialisation is not noticed. In use, the same code destroys and rebuilds tracking state every time a user scrolls, which is the fourth row's cost paid on the second row's frequency.

#Why tracking state changes the calculus

An ordinary embedded view can be destroyed and recreated with no consequence beyond a frame of blank space. An engine performing camera-based tracking cannot, and this is the reason the ownership model below is worth its complexity rather than being over-engineering.

Tracking systems build a map of the environment incrementally and estimate pose against it, and the map is the accumulated product of everything the camera has seen since initialisation [3] [4]. Destroying the engine discards it. What the user experiences is not a brief blank: it is content that was correctly registered to the world becoming unregistered, drifting, and needing them to move the device around again to recover [5].

Recreating a button costs a frame. Recreating a tracking session costs the user's confidence that the content is attached to the world, which is the entire premise of the application.

This asymmetry is why pause and resume, rather than create and destroy, is the operation the integration must make cheap and obvious. An API where destroying is the easy path will be used that way.

#Ownership model

Three rules, each of which corresponds to a decision being placed with the component that owns it [6].

One owner for the engine. A single object holds the engine and is solely responsible for pause, resume and teardown. Views attach and detach. They never own it, because a view is disposed for reasons that do not imply the engine should stop.

No assumption of visibility. Engine code that runs on resume rebuilds anything bound to a rendering surface rather than trusting that its previous surface remains valid.

Explicit state. The integration exposes whether the engine is created, paused or destroyed. This is not a convenience. A defect report that includes the state is worth an order of magnitude more than one reporting that the view went black.

#Input and accessibility, which the lifecycle discussion tends to omit

Two boundaries besides the lifecycle need deciding, and both were underestimated in the first implementation.

Input arbitration. A gesture arriving over the engine surface may belong to the engine, a rotate on a model, or to the host framework, a scroll of the list containing it. Neither system can decide alone, because the engine knows whether the touch hit an object and the framework knows what the enclosing layout expects. The workable arrangement is that the framework offers the gesture to the engine first, the engine performs a hit test and claims it or declines, and the framework proceeds normally on a decline. Getting this wrong produces the two symptoms every such integration exhibits at some point: a view that cannot be scrolled because the engine swallows everything, or a model that cannot be rotated because the scroll wins.

Accessibility. An engine surface is opaque to the platform's assistive technologies. Nothing inside it is announced, focusable or navigable, which means an application that draws its controls inside the engine is inaccessible in a way that no amount of care within the engine repairs. This is the strongest argument for the design rule stated below about implementing interface elements in the host framework, and it is a stronger argument than the performance one, because the performance cost is a degradation and this is an exclusion.

The residual case is content with no host-framework equivalent, such as a label attached to a three-dimensional object. The best available answer is exposing a parallel accessibility tree describing what the scene contains, which is more work than it sounds and which the reference implementation does not do.

#Measured cost

On mid-range mobile hardware, running a modest scene at 60 frames per second inside a declarative interface:

• Resident memory increased by approximately the engine's own footprint plus scene assets. The two runtimes do not share an allocator, so nothing is pooled between them.
• Cold start increased by roughly one and a half seconds, dominated by engine initialisation rather than by view creation.
• Scrolling a list containing the engine view was visibly worse than the same list without it, because the compositor synchronises two surfaces.

Three design consequences follow: keep the engine surface large and stationary, avoid repeated creation and destruction in favour of pausing, and implement all interface elements in the host framework.

#Build integration, which is where most of the support burden went

The lifecycle model is the interesting part of the problem and the build is where the time actually goes, so a report that omits it would misrepresent the work.

The engine and the interface framework each have their own build system, their own toolchain versions and their own opinions about the shape of the output. Producing a single application means exporting the engine as a library artefact and then persuading the host build to consume it, and every element of that is version-sensitive: the engine version determines the artefact layout, the host framework version determines how libraries are consumed, and the platform toolchain version constrains both. A combination that works is a point in a three-dimensional space, and the space is mostly empty.

Two consequences shaped the project more than any design decision. The first is that most support requests were build failures rather than runtime defects, by a wide margin, and answering them required reproducing a combination of versions rather than reasoning about behaviour. The second is that automating the export step, so that the engine artefact is produced by the host build rather than by a developer following instructions, removed more support burden than any feature. A practitioner write-up covers the configuration in the detail this section does not [7].

The general point is one this project taught me and I have seen since: for an integration library, the installation path is part of the interface. A library that is correct and cannot be built is not usable, and effort spent making the first step reliable competes directly with effort spent on the library's actual subject. The split here was heavily toward the former and that was the right allocation.

#Deployment evidence

The reference implementation has been used in production for location-based augmented reality at public scale [8] and in a consumer mobile game with substantial engine content [9]. A practitioner write-up covers the build configuration in more detail [7].

#Limitations

The measurements are from one hardware class and one engine version, and both platforms have since changed their compositing mechanisms. The numbers should be read as an order of magnitude rather than a benchmark.

The ownership model does not resolve the case where the host application's configuration determines whether the context is recreated, since that is outside the integration's control. The only available response is documenting supported configurations, which is a mitigation rather than a solution.

The deployment evidence establishes that the approach works at scale and does not isolate the contribution of the ownership model. Both products involved substantial engineering beyond this integration, and neither was run against a controlled alternative. What can be said honestly is that applications of this shape shipped and were used by large numbers of people, which is worth more than a benchmark and less than a comparison.

The state table is derived from defects encountered rather than from an exhaustive enumeration of the product space. Three lifecycles with several states each admit more combinations than the six rows shown, and the omitted ones are omitted because nobody reported them, which is not the same as their being impossible. A model checker over the combined state machine would settle it and would be a reasonable piece of work for anyone maintaining such an integration.

The accessibility discussion identifies a gap and does not close it. No implementation of the parallel accessibility tree exists here, and I am not aware of a satisfactory general treatment of assistive technology for registered three-dimensional content, which suggests the difficulty is real rather than neglected.