Live2DPet-Enhanced: Giving a Desktop Pet a Brain That Watches Your Screen

Live2DPet-Enhanced is an Electron-based AI desktop pet. The Live2D character that lives on your desktop isn’t just there to look cute: it periodically takes screenshots and detects the active window, hands the image to a Vision-capable LLM (OpenAI-compatible, so anything like OpenRouter works), the model figures out what you’re doing, and then it starts a conversation with you through a speech bubble. The character is transparent, borderless, always on top, and its eyes follow the mouse. Screenshots only pass through memory on their way to the API and are never written to disk. Your desktop has a lot of sensitive stuff on it, so the less that gets stored the better, and this rule was locked in from the start. Repo: github.com/dwgx/Live2DPet-Enhanced.

Keyframe visual memory is the key to keeping costs down. Stuffing every screenshot into the context blows up both tokens and money, so it samples first, then lets the VLM pick a few representative frames to keep.

Voice runs on VOICEVOX, a local Japanese TTS, with koffi calling the VOICEVOX Core dll directly over FFI; replies are translated to Japanese before being spoken. To guard against crashes there’s a three-tier fallback: use TTS if it works, drop to a default voice if it doesn’t, and go silent if even that fails, so conversation never gets blocked by audio.

It works without a model too. Hot-import a .model3.json to auto-map parameters, expressions, and motions, or point it at an image folder as the character, tagging idle/talking/expression states so it switches on AI emotion triggers. The persona uses a JSON template with variable substitution for things like {{petName}}. API keys are encrypted with AES-256-GCM before they hit disk.

One trap I hit: when running from source, don’t use npx electron ., since it fights with ELECTRON_RUN_AS_NODE. Just use node launch.js.

Early on I also built a whole intelligence-enhancement pipeline: automatic search, knowledge organization, activity memory, VLM scene extraction, all meant to make it smarter. But it was too heavy, burned too much money, and the payoff wasn’t obvious, so the whole thing was disabled in v2.0 with the skeleton left in place. Vision models really do cost money, so don’t set the detection interval too tight.