Riddle: An Interactive E-Ink Diary Powered by Vision LLMs

Riddle is an open-source project that turns the reMarkable Paper Pro into a magical diary reminiscent of Tom Riddle's diary from Harry Potter. Write with the pen, rest it, and after ~2.8 seconds of idle, the page is committed as a PNG and sent to a vision language model (LLM). The LLM's reply streams back sentence-by-sentence, rendered in a cursive font (Dancing Script) that animates stroke by stroke, then fades away. No screen glow, no keyboard—just ink on paper.

How It Works

The app is written in Rust and consists of four main components:

  • Pen input: Reads raw evdev events with 4096-level pressure and hardware event rate.
  • Ink surface: Renders strokes using the Zhang-Suen thinning algorithm to skeletonize handwriting into single-pixel paths for animated replay.
  • Oracle: A resident LLM process that receives the committed page as a PNG and streams a reply.
  • Display backends: Two options—qtfb (windowed inside xochitl via AppLoad) and quill (full takeover, stopping xochitl and driving the vendor e-ink engine directly for lowest latency).

Gestures

  • Write and rest the pen → diary drinks ink, Tom replies.
  • Flip the marker → erase.
  • Draw a large ? → summon built-in guide.
  • Tap five fingers → leave diary.
  • Power button → page turns to "The diary sleeps.", then suspends; press again to wake exactly where you were.

The Oracle: Vision LLM Backends

Two backends are available, chosen at startup:

Option A — OpenAI-compatible API Set an environment variable and the app talks directly to any /chat/completions endpoint. Works with OpenAI, OpenRouter, Groq, or a local server. Example config:

export RIDDLE_OPENAI_KEY="sk-..."
export RIDDLE_OPENAI_BASE="https://api.openai.com/v1"
export RIDDLE_OPENAI_MODEL="gpt-4o-mini"

Measured ~0.9–1.1 seconds to first ink on-device using this path.

Option B — pi If you already run pi, Riddle uses a resident pi --mode rpc process to avoid extra startup overhead.

Installation Requirements

  • reMarkable Paper Pro (model ferrari, aarch64) with OS 3.26–3.27.
  • Developer mode enabled (use remagic to set up).
  • For windowed mode: xovi + AppLoad.
  • For takeover mode: reMarkable SDK toolchain (~/rm-sdk-3.26) and libqsgepaper.so pulled from your own device (proprietary, not distributed).

Building and Deploying

Windowed (AppLoad/qtfb)

cd riddle
cargo build --release --target aarch64-unknown-linux-gnu

Copy the binary and manifest to /home/root/xovi/exthome/appload/riddle/.

Takeover (instant ink)

cd quill && ./build.sh          # pulls libqsgepaper.so from device
cd ../riddle && ./build-takeover.sh

Deploy libquill.so to /home/root/quill/ and riddle-takeover to /home/root/riddle/riddle. Launch via AppLoad; it stops xochitl, runs the diary, and always restores xochitl on exit. If something wedges: ssh root@10.11.99.1 'systemctl start xochitl'.

Technical Details

  • Pen latency: The takeover backend (quill) drives the vendor e-ink engine directly, bypassing xochitl for minimum latency. A companion demo scribble shows pen-to-glass latency.
  • Handwriting synthesis: Rasterized strokes are skeletonized via Zhang-Suen thinning, then traced into animated stroke sequences.
  • Font: Replies use Dancing Script (SIL OFL 1.1), rendered as animated strokes.
  • Security: Runs as root; modifies the device. The vendor UI is stopped in takeover mode. SSH access must be kept working as an escape hatch.

Why It Matters

Riddle demonstrates a novel interaction paradigm: using an e-ink tablet as an AI-powered notebook that reads and writes physically. It's a technical showcase of low-latency e-ink display driving, Rust-based system programming on ARM64, and integration with vision LLMs. For developers, it's a hands-on example of:

  • Direct e-ink engine manipulation via libqsgepaper.so interposition.
  • Real-time handwriting processing with thinning and stroke replay.
  • Streaming LLM responses on constrained hardware (the reMarkable Paper Pro runs a Linux-based OS).

Risks

The project warns: "This modifies your device. It runs as root, stops the vendor UI, and drives the e-ink engine directly. It has only been tested on a reMarkable Paper Pro (ferrari, aarch64, OS 3.26–3.27)." Use at your own risk. Keep SSH access working before installing anything.

License

MIT for all code in the repository. Vendor libraries (libqsgepaper.so, Qt) are not included and must come from your own device/SDK.

Next Steps

If you have a reMarkable Paper Pro and want to try Riddle:

  1. Enable developer mode using remagic.
  2. Install xovi + AppLoad for windowed mode, or set up the SDK for takeover.
  3. Download the prebuilt bundle from the latest release or build from source.
  4. Set your OpenAI API key or leave empty to use pi.
  5. Launch via AppLoad and start writing.