Letting a model drive the bench
Agents
A model writes a plugin quickly. What it cannot do, today, is see the device. That gap is where the time goes: coordinates guessed from a screenshot, taps that silently miss, and a crash nobody can explain because the device keeps no symbols.
What already exists
| Piece | What it gives you |
|---|---|
addin remote (MIT) |
the live screen, touch injection, MIDI, and a JSON-RPC MCP endpoint on 127.0.0.1:6720 —
screenshot, tap, drag, scroll,
wait_for_screen, play_notes, device_status, list_files |
addin commander |
the loaded plugins with their skins, MPC’s transport and parameter changes over HTTP/WS — no tapping required |
addin filebrowser |
browse the device drives in a browser (deploy without scp acrobatics) |
addin usb-audio |
the MPC as a USB audio interface — worth knowing before you assume “no loopback” forever |
addin skipback |
records the main output all the time, saves the last 30–60 s |
mpc-store.sh, Plugin Manager, MPC Link |
installing, updating, and getting files from a computer into the device |
patches: drum-pad-layout, drive-exec, button-remap |
what a patch can change without touching a plugin at all |
What a development agent actually needs
From one real port — a plugin built, tested, deployed, driven and twice crashed — the missing pieces are not “more tools”, they are named handles and evidence:
Say the name, not the coordinate
The skin already declares every widget and its bounds. A helper that parses the installed
TUI.json can answer “where is the RESONANCE knob, what is its value” and tap it by
name. MPC’s own screens (plugin list, dialogs) are fixed-pitch lists: record their geometry once,
with a human watching, and replay it with assertions instead of re-deriving pixels every session.
Every action returns evidence
Before/after screenshot hash, the changed rectangle, the parameter value read back. “The tap landed and nothing happened” has to be a visible result, not a story the model tells itself.
Health and crash facts
MPC pid, whether the .so is still mapped, the /data/sentry tail,
audio-worker CPU from /proc, xruns. That is what turned “it crashed twice” into a
rule in the device page.
Safe staging
Snapshot and restore of MPC.settings, the plugin folder and the project; a single
“remove and re-insert” verb; a watchdog that restarts acvs when a plugin took it down.
Proposal: a development bench as a plugin
A resident addin (plus a thin host CLI) that turns the loop into verbs:
screen.describe, screen.act(name), param.set(instance, id, value),
plugin.reset, device.health, crash.explain. It composes the
remote and commander addins instead of reimplementing them, speaks JSON-RPC
on 127.0.0.1 with the same names as its MCP tools, and every answer carries
{ok, before, after, evidence[]}.
First release, read-only: describe a screen, act on one binding, report health and the plugin’s own diagnostics. Writes come second, behind snapshot/rollback.
Do not build these
- Audio capture through the plugin. The render goes to the DAC; there is no loopback. A plugin can report its own levels — that is the honest path.
- Writing
MPC.settingswhile MPC runs. Stop first or lose the settings file. - Hardware-button injection. No virtual RawMIDI port; the remote addin reports
zero buttons. Anything behind
MENUstays a human step. - Opening
/dev/driinside an addin. Whoever opens the card first becomes DRM master and MPC dies on the spot. - Threads that outlive what started them. That is the crash this whole site keeps coming back to.
When the agent is the one laying out the screen
Skin work is where an agent burns an afternoon fastest, because the feedback loop is the
device. This week it did exactly that: switches that did not react were “fixed” five times —
bigger art, padded hit boxes, a different widget — and every attempt needed a rebuild, a deploy
and a human finger. The cause was one number that had been written down all along: a widget is
drawn 86 px above its cy, so in a 1280×628 area nothing below
art y ≈ 542 has a reachable touch box. The switches sat at cy=658.
Every “fix” changed the thing that was not broken.
The loop that converges, in this order:
- Read the layout skill first (
.opencode/skills/mpc-ui-builder). It carries the offsets, the canvas budget, and which widget to use for which kind of value. tools/skin_check.py "<skin>/Plugin Skins"— reportsEDGE(box past 1280×628),TOUCH(two touch boxes overlapping),QLINK,OPTSper page, in about a second.gen_vst.pyprints the same findings after every build. A skin with any finding is not deployed.studio.py previewfor the picture, including the variant per stateful widget — that is the only way to see a pressed switch without touching the device.- Only then build and deploy. A skin needs remove/re-insert; a
.soneeds anacvsrestart. Removing and re-inserting a plugin does not reload a new library.
Two facts that cost the most, and neither is about taste: a toggle never
fired on a 2-option enum on this OS build (enum_h did, immediately), and
MPC acts on release, not on touch-down. If a control must show its state, it has
to latch.
How to drive it today, without the bench
ssh -N -L 6720:127.0.0.1:6720 root@<ip> # the addin listens on the device only
curl -s localhost:6720/info # {"width":1280,"height":800,"touch":"/dev/input/event0"}
# screenshot → tap → screenshot; assert with a hash or a region diff, never by hope
# and read /proc/<pid>/maps when a deploy “did nothing”