Research

MiniConv on the device. CALF on the server.

StandardRL is the hardware realisation of a distributed reinforcement-learning stack: an edge Policy Unit runs on the device, compact visual features can be produced by MiniConv, and larger models can be trained and served through CALF over real networks.

The research stack is deliberately split. StandardRL observes the host computer through its ordinary display signal and acts through ordinary HID input. The device runs a Debian-based operating system on a Raspberry Pi Compute Module platform, giving it enough local compute to preprocess observations and run small edge policies. More expensive policy components remain on the server, where CALF can train and deploy them under communication conditions that resemble real networked execution.

This split is not an implementation detail. It is the central systems idea: keep the low-latency physical interface near the host computer, while allowing learned policies, larger models, monitoring tools, and training infrastructure to live where compute is available.

Policy units and effects

The StandardRL design follows the Policy Unit framing developed in the Effects work. A policy unit is a discrete callable component in a larger policy graph. It can be invoked, can return, and can be reasoned about as an explicit part of the agent's action structure rather than as an opaque monolithic controller.

For StandardRL, that abstraction maps naturally onto hardware. The device-side unit is responsible for observation capture, immediate preprocessing, and time-critical interaction with the host. Server-side units can handle larger neural models, longer-horizon reasoning, or task-specific policy heads. The division gives the project a clean way to discuss accountability, delegation, latency, and fallback behaviour.

Policy unit diagram from the Effects chapter
The Effects chapter formalises Policy Units as explicit components in a policy graph. StandardRL uses the same idea to separate edge execution from server-side policy computation.

MiniConv edge inference

MiniConv addresses the edge-compute problem. It is an open-source library for tiny visual encoders that can be compiled into OpenGL fragment shaders, allowing small convolutional encoders to run on low-power devices using the graphics stack that is already present.

In the StandardRL architecture, MiniConv is used on the device to run the edge Policy Unit. The device can transform a captured screen into a compact feature representation close to the host computer, reducing the amount of data that needs to cross the network before a server-side policy head or larger model is consulted.

The MiniConv paper evaluates this approach on constrained edge platforms, including Raspberry Pi-class hardware. It shows the practical trade-off: when bandwidth is poor or network conditions are unstable, sending compact features from an edge encoder can be preferable to streaming full observations to a remote policy.

MiniConv fragment shader input and output
MiniConv maps visual encoding onto fragment-shader passes, making small convolutional encoders practical on embedded graphics hardware.
Mapping CNN layers to fragment shader passes in MiniConv
The layer mapping from the Models chapter illustrates why MiniConv is useful for a device that must process pixels locally before networked policy execution.

MiniConv paperMiniConv sourceModels chapter

A runtime of modules

On the device, the policy is assembled from small, swappable modules rather than baked into a single binary. Encoders, network heads, gym interfaces, packet inspectors, and input testers are installed, started, and stopped independently — so researchers can reshape the running policy graph without reflashing hardware.

This is what makes the edge-versus-server question a knob rather than a rebuild: move a module on-device for latency, or off-device for capacity, and measure the difference.

The on-device module manager listing installable units: Demo App, Gym, Inspector, Neural Network, and Input Tester
The device's module manager. Each unit — from a Gym interface to a configurable PyTorch network or a packet inspector — can be installed and run independently, letting the policy graph be reshaped in place.

CALF networked policies

CALF, the Communication-Aware Learning Framework, handles the other half of the split. Standard reinforcement-learning experiments often assume that observations and actions are exchanged without delay. A physical StandardRL deployment cannot make that assumption: communication can be delayed, jittered, bandwidth-limited, or interrupted.

CALF is used to train and evaluate policies under those communication effects. In this project, the heavier models and server-side policy units run through CALF infrastructure, while the device keeps the physical interface local. The result is a cleaner separation between fast embodied interaction and larger remote computation.

Under CALF, a policy is treated as a distributed system rather than a single zero-latency process: units become networked services that can be routed and analysed under concrete deployment modes, instead of only under idealised simulator conditions.

CALF deployment modes diagram
CALF describes policies as distributed systems rather than assuming a single zero-latency process.
CALF CartPole network-aware training results
Network-aware training is the relevant experimental idea for StandardRL: the device must remain useful when compute is not entirely local.

CALF paperSystems chapter

The cost of the split, measured

The split is only worthwhile if the numbers support it. The MiniConv work reports platform-level timing on Raspberry Pi-class devices and breaks the decision loop into its parts: local encoding, feature transfer, and server inference. Those measurements set realistic expectations for a device that sits between a normal host computer and a potentially remote learning system.

MiniConv frame duration results on Raspberry Pi Zero 2 W
MiniConv evaluates the cost of small visual encoders on constrained Raspberry Pi hardware, setting realistic expectations for edge Policy Unit latency.
Decision latency breakdown for MiniConv split-policy inference
The latency breakdown shows the point of the StandardRL split: local feature extraction can reduce network pressure before server-side policy execution.

BrowserEnv alignment

BrowserEnv provides the simulation-side training setting for computer interaction: a live browser runs inside an isolated container, the agent observes rendered pixels, and actions are delivered as low-level keyboard and mouse events. The Realisations chapter emphasises that this observation-action interface is structurally aligned with the physical device.

BrowserEnv architecture for computer-interaction policy training
BrowserEnv trains against the same kind of boundary that StandardRL exposes physically: pixels in, input events out.

Runtime split

  1. CaptureThe host display is observed through the StandardRL video path.
  2. EncodeMiniConv runs locally on the device to produce compact visual features or an edge policy decision.
  3. RouteCALF and the server-side runtime receive features, state, or calls from the edge Policy Unit.
  4. InferLarger models run on the server when the task needs more capacity than the device should carry.
  5. ActThe selected action returns to the device and is emitted as ordinary HID input.

This keeps the StandardRL device simple enough to be physically plausible while retaining a path to richer models. It also gives experiments a meaningful variable: which parts of the policy should be local, which should be remote, and how much performance is lost or gained as the network changes?

Research lineage

EnvCraft generates validated browser-playable environments. BrowserEnv provides a proxy computer-use setting. MiniConv moves visual encoding toward the edge. CALF makes distributed policies trainable under real communication effects. StandardRL closes the loop by giving that software stack a physical endpoint on an ordinary computer.

MiniConv and CALF are open-source research components, and StandardRL's own device software is intended to follow as it stabilises.

MiniConv on GitHubCALF paperChapter 9: Realisations