Kiran’s notes 👨🏻‍💻

2025-06-18-Wed

25 min read

18:23: The third and final piece of my Hackberry Pi CM5 arrived yesterday so I put it together. By order of arrival:

  1. NVME 2242 drive with 512 GB from discount brand EVM.
  2. Zitao’s Hackberry Pi CM5 in black with a Q10 keyboard, from Elecrow.
  3. Raspberry Pi CM5, model CM5108032 (1 = WiFi, 08 = 8 GB RAM, 032 = 32 GB eMMC), after two failed attempts to buy a CM5 Lite 16 GB (CM5116000) from Element14 and Mouser. Lite is preferred as the eMMC models don’t have SD card support, eMMC is not particularly fast, 32 GB isn’t much, and the storage is not user-replaceable but will degrade with use. eMMC only makes sense for embedded firmware.

This is my second Hackberry Pi. The last one was the RPi4 model with a BlackBerry 9900 keyboard, and it was preceded by a half-hearted attempt to make one with an RPi4, a 4.3” touchscreen, and no keyboard because I didn’t know how to add one. I’ve also setup Termux from scratch over three devices: a OnePlus 9 Pro phone (since deceased), a Boox Mini C colour e-ink Android tablet, and a OnePlus Open foldable phone with Termux:X11 running an XFCE desktop.

I’ve enumerated these because they have common features: small screens that don’t fit desktop windows or even CLI apps producing wide output, and they have terrible keyboards – either an Android soft keyboard that is heavily dependent on autocorrect and not great for code and CLI, or a thumb keyboard that’s even slower to use these days because the layout and the button ergonomics are both unfamiliar.

Since these devices are necessarily handheld (small screens), they aren’t usable with external keyboards. CLI shortcuts – say Vi or Emacs-style navigation – evolved on the unergonomic hardware of decades ago and they’ve lasted because they’re efficient, but not on these devices where the entrypoint : is buried in layers and there’s no Esc key at all to backtrack. Need Ctrl+A to move the cursor home? It’s easier to touch the screen and move the cursor, except I can’t because that’s not a thing for terminal emulators. The terminal ecosystem hasn’t worked out the protocols for touch interfaces. It’s not even settled for mouse events, and we’ve had those things on every desk for decades.

So I’ve spent a lot of time obsessing over any minor improvements to CLI ergonomics, filing tickets pleading with terminal maintainers, trying out anything that can make it marginally better. An example: if I’ve just listed a directory and there’s a file I want to look at:

  1. First, to list the dir, type: ls /some/path
  2. Second, to view the file: Up (chorded) to recall this command, Ctrl+A to move to start of line, Right (chorded) to skip the l, then type es to change the ls into less, Ctrl+E to return to end of line, type / to specify directory context, then Tab multiple times to select the file again, and finally Enter.

Handheld keyboards soft or hard are typically missing all of these keys: Ctrl, Alt, Meta, Esc, Tab, Delete, Up, Down, Left, Right, PgUp and PgDn. Even the common chord modifiers Ctrl and Alt may themselves be buried behind other layer shifting keys. There’s no “short” in “shortcut” here. I’ve made a shell function named l that serves as an “ls or less” based on the type so I can skip the left side of the edit and just append the filename (for half the effort):

unalias l
function l {
    if [ "$#" -ne 1 ]; then
        # If not sole arg, ls
        ls -h "$@"
    elif [ -f "$1" ]; then
        # If sole arg is a file, cat
        cat "$1"
    else
        # Fallback, ls
        ls -h "$1"
    fi
}

I settled on cat over less so the content remains on the screen, but my cat is aliased to bat which behaves like less where appropriate. This works, but:

  • I keep forgetting to use it and must amend with Up (recall), Ctrl+A (go to start), Alt+D (delete word), and then type l to switch to this command, but
  • Then I find I’m on a different machine today where this function isn’t present, so I’m amending again to switch to less, and
  • I can’t reliably push this function into every shell I ssh or su into, so where’s the scope for habit formation?

What is even the point of this tedious input when the filename is right there on a touchscreen, within reach of my thumb? Why can’t I simply tap on it?

Turns out terminal emulators do indeed support hyperlinks and some ls replacements like lsd can make each filename a hyperlink to the file path, but it ends there. Tapping on the link won’t insert it into the command prompt. It may invoke an OS-level handler instead. Why can’t I tap and drag it into the prompt?

(For one, the terminal emulator doesn’t reliably know the shell’s current working directory and has to supply an absolute path defeating the many advantages of relative paths, but what if the link and the prompt are on different hosts? if the link is from ssh ls, it’s not a local path for the prompt that follows. But so what? It’s the user’s problem. Aside: the shell’s bracketed paste handler can parse the file: scheme and rewrite from absolute to relative. I should investigate that. Can bracketing have metadata so it doesn’t mess with clipboard paste?)

So if this cooperative ls+prompt isn’t happening, maybe there are TUI apps for it? There’s the ancient GNU Midnight Commander, inspired by the even more ancient Norton Commander on DOS, but it turns the screen blue, tries to render two side-by-side panes on a narrow screen, and uses F1–F10 for navigation, which are totally non-existent on these soft/thumb keyboards and disappearing from laptops too. Besides, if a terminal emulator can’t be sure what’s happening inside a shell, MC can’t either. Their shell integration is a fragile hack that falls apart very easily (see also Warp Terminal), so I went looking for modern implementations, nnn etc, until I found one I liked. Yazi. It even has nice shell integration in the correct direction: jump from the prompt into yazi to pick the path/file and insert it, like fzf tab completion but way better. Nice. I will use the ™ mark henceforth to mark these recognizable inflection points.

I spent hours setting up Yazi in Termux on my phone. It has a plugin manager and its own store (just GitHub repos), but when I updated after a few months the plugin API had changed and Yazi wouldn’t start and the plugin manager turned out to be even more fragile, breaking in new ways while I attempted to upgrade the plugins, but I’m getting ahead of myself here. Dial back to when I had Yazi installed and configured just right. Nice™. But only on one device. I also had Termux on another device I wanted to reproduce this setup on, and on my macOS laptop, and on the Hackberry Pi. That’s three different distributions with different package managers and folder layouts (eg, Termux can’t use #!/usr/bin/env), and this Nice™ setup was a fragile combo of packaged zsh+yazi, git-cloned oh-my-zsh, custom scripts within omz, yazi config, and again git-cloned yazi plugins via its plugin manager. Those plugins in turn depended on package manager-installed tools like bat (syntax highlighted files) and glow (rendered Markdown)). How do I reproduce this fragile coalition on another device when I don’t even remember the contents?

I started off reproducing on my laptop where I have the nicest keyboard and the lowest pain of typing tedium, but even there I failed by distraction. Now I have Termux Nice™ that’s not on the laptop and macOS Nice™ that’s not on the phone. Which is how I went a few months without upgrading yazi before finding it bafflingly broken.

And which is why the final part’s arrival for the Hackberry Pi CM5 yesterday was a moment of dread. I will have to Make Config™ again. Config portability has to be a solved problem, right? There are more dotfile managers than I have fingers. Figuring out which one to start with has repeatedly put me off them, but this time it felt like It’s About Time™, so I asked on the socials and read all the Reddit opinions and picked the one with the most GitHub stars and the fewest complaints, Chezmoi. Incredibly enough, runner up Mackup has a warning that it will delete all your config (instead of, you know, saving it), because it uses symlinks and macOS Sequoia believes a symlinked preference file is a sign of malware and should be deleted immediately.

Symlinks for config. Why? It’s not the only dotfile manager to use this method. Chezmoi doesn’t, but after setting it up I think I can skip ahead to the conclusion: all of these tools are crap. They are to config management what GST is for ease of business: a system designed for government accountants to have a grand dashboard of the economy, but by constraining the user into allowed behaviour. Pushing boundaries? Can’t accommodate that.

I’ve arrived to this moment through a simple quest: I have devices with small screens and terrible keyboards that negate the efficiencies of command-line interfaces, but they also have great touchscreens that aren’t being harnessed. This isn’t new: Android is Linux but not Linux Desktop because it pursued this exact trade-off to create Mobile as a distinct category of hardware and software from Desktop.

I don’t have such ambitions. I just want a pocket command line, and I’m cycling through hardware (phones and cyberdecks) to find one that will make home in my pocket. As I cycle, I want to build on the tinkering with the last device instead of a fresh setup each time. This should be within the ambit of a dotfile manager, but it’s not because the problem space isn’t dotfiles alone. It’s a familiar Linux command-line, but the underlying stack is all variable: the CPU architecture, the OS, the package managers (plural) and their repositories (which also vary by OS+arch), and the dependency graph between them.

What good is a dotfile manager if the tool that uses the dotfile isn’t also managed? What’s the value in “one line” dotfile deployment (with templates!) if those templates depend on a password manager that must first be installed without a common method (different package managers), and the password manager in turn depends on credential storage (like Keepass with any synced file storage) which also has to be installed and authorised (like Syncthing, with the credentials folder shared from any of the previous devices). This is a lot of manual setup before you get to the convenience of a “one line” deployment on a new device, and that convenience is also dependent on you having written a template that covers all the gotchas from the differences in the environment. We’ve all learnt to write our hashbangs as #!/usr/bin/env python3 because hardcoding a Python path is a Major Oops™. You must always ask for the Python from the currently active environment, but if you do this on Termux, you’ll have another gotcha because there’s no /usr/bin/env on Android. It’s Linux without POSIX paths. Termux has an LD_PRELOAD lib to specifically rewrite hashbang paths, but that doesn’t catch every instance.

When your gotcha-mitigation tactic is the cause of the gotcha, have you really mitigated it? If your gotcha-proof dotfile template is suitable for a “one line” deployment, is the device even new? It’s just one more of something familiar, and therein lies the assumption of whom dotfile management is for:

  1. You’ve got a config nailed down and you want to replicate it over a number of devices. This number of justifies the effort of writing a template to account for differences between these devices.
  2. You’re going through a periodic upgrade cycle and want to carry over config without the cruft that comes with full backup-and-restore. Maybe you’re an experimental upgrader and the next device has a different arch/OS/distro, so you also have the incentive to write templated config. The goal is to get two different devices to have mostly identical config. At this point the dotfile manager is not your aide. You are its aide, doing the labour of writing it a perfect template so it can boast about the convenience of one-line deployment.

I’m not either of these. I’m trying to extract usability from a limited device, and when I find Nice™ I won’t recognise it immediately, and later I will only have a vague idea of how I got there. Install something, link it to something else that came via some other package manager, and do it in the correct order or it will fail. This is a long, long way from precision gotcha-proof templated dotfile config. Each step in this setup is a Maybe This Will Work™, needing confirmation and correction before proceeding.

Even for a stable baseline, dotfile tooling feels inadequate. For instance, I like zsh, oh-my-zsh and powerlevel10k. They’re showing their age, but they work for me and I don’t have headspace to evaluate better choices, so I just want these in my baseline config on every device. That means:

  1. Ensure zsh is installed. It’s the default shell on macOS but not elsewhere, so the first part is invoking the correct package manager. Recent ones:
    • macOS: brew install
    • Termux: pkg install
    • Alpine Termux: apk add
    • Arch Termux: pacman -S
    • Debian/Ubuntu: sudo apt install (first to need sudo in this list)
  2. Switch shell using chsh, but it wants the full path and that’s not always /usr/bin/zsh (Termux says hello again), so that’s another templated list. Also ensure Zsh is an allowed shell in /etc/shells (skip for Termux).
  3. Install oh-my-zsh. It uses the fashionably insecure URL-piped-into-a shell method with your choice of wget or curl (because one of them is typically missing). Piping from a URL is either okay because you trust this one, or you’re frowning already because you know what happens in audit. In any case, the main action is just a git clone of the omz repository followed by a template .zshrc pointing to it.
  4. Install powerlevel10k. This may be packaged (brew install) or another git clone, each needing a different invocation from .zshrc.
  5. Probably no need to configure p10k because we want a standard config, but it makes a cached script for faster runtime and surely that’s dependent on the version installed? Does p10k configure take all the answers via CLI flags? IDK™.
  6. And almost there but not quite. The point of oh-my-zsh is the extensibility, so I’ll want another plugin or two and a custom script to use them. In the grand wisdom of oh-my-zsh’s makers, this is organized as a series of nested git repositories that are not aware of each other, with your custom scripts in the middle of this nesting that are also not managed by git, that we will now attempt to place in the care of our dotfile manager.

Zsh plugins? I’ll throw in my pet peeve here for the illustration. A terminal emulator is not a text editor. You can’t select text with a mouse and manipulate it with the keyboard. The terminal can do copy and paste but not cut and paste. There’s no cut. How often have you invoked something as <command> <filename> <options> only to find this particular command is fussy and wants the options before the filename? This is a simple four-step transpose in a text editor – select, cut, locate, paste – doable with either a mouse or a keyboard exclusively. A terminal can’t do the middle two, so it becomes select (mouse only), copy (mouse or keyboard), backspace (keyboard only), locate (keyboard only), and paste (mouse or keyboard). This is tedious. It may be faster to type it all out again.

Enter zsh-shift-select, a plugin that adds a select mode accessible with shift+movement keys. Now I can shift+move to select text, cut using a shell shortcut key, and paste elsewhere. The selection here is only known to the prompt and not the terminal, so you can’t use your mouse at all, which makes this not as fast or elegant as a text editor, but way better than before.

zsh-shift-select even chooses keybindings based on the detected host, since macOS key bindings are typically Command key-based, while Linux follows Windows style with Ctrl keys. The plugin picks the appropriate keybindings based on the current host. Nice™.

Except, another gotcha: the terminal and the shell can be on different hosts! This is what happens when you ssh somewhere. The plugin can identify the shell’s host but not the terminal’s. Kitty and Ghostty users: you’re allowed to smile now if you been frustrated by broken ssh and made the leap from there to discover what a terminfo is.

If host-inappropriate keybindings are a gotcha to be mitigated, as this plugin did, the gotcha-mitigation is itself the cause of the next gotcha, for the shell host isn’t necessarily the terminal host. I felt tempted to help here, for I had recently discovered terminfo. I could write superior host-detection logic! But head won over heart. The only way to not get lost down this rabbit hole was by adopting universal keybindings. Support all the navigation keybindings on both macOS and Linux. What matters is that familiar keybindings should work as expected. Unfamiliar bindings exhibiting unfamiliar behaviour are not a concern. I made a bindkey.zsh and dropped it in ~/.oh-my-zsh/custom across my devices.

I tell you this story because my gotcha-mitigation was the cause of another gotcha that we’ll presently get to.

Present day: new device that needs config, so I’ve cleared my day to Finally Figure Out™ dotfile management starting with the leading candidate, Chezmoi. I want zsh+omz+p10k+custom on all my devices. Can Chezmoi handle this for me? Why sure, here you go directly into the deep end, all you need are some preparatory config files in two different formats where you tell Chezmoi what the task is. You want Chezmoi to manage one file? Sure. Deploy an external git repo? Sure. Manage your config file inside a git-ignored folder of that git repo? No Can Do™. Chezmoi uses git to manage your files, but doesnt approve of git directly managing your files. There’s even a documentation page for Chezmoi with oh-my-zsh that tells you to not use git. I gasped and said No Thanks™.

This isn’t Chezmoi’s fault. There’s a lot happening here, so let’s take this slower.

As I’ve enumerated above, the deployment has many steps, the first of which is obtaining zsh itself via an abstraction layer for the package managers. But all of these package managers were born as abstraction layers for the previous generation of package managers, so I will recognise this Rabbit Hole™ for what it is and refuse to enter. No automation here.

Next, oh-my-zsh. Here we must examine the nested folder structure to spot the pattern to put a ™ symbol on. If my ASCII art is any good:

~
└ .oh-my-zsh (git repo)
  └ custom (git ignored)
    ├ bindkey.zsh
    ├ update.zsh
    ├ plugins
    │ └ zsh-shift-select (git repo)
    └ themes
      └ powerlevel10k (git repo)

Here we have git repositories nested within a git repository, but none of them are aware of each other, so they can’t be updated atomically. My Cope™ is in update.zsh:

source "$ZSH_CUSTOM/colors.zsh"
 
omzu () {
        find ~/.oh-my-zsh -type d -name .git -exec bash -c 'echo -e ${Color_BIGreen}\~/$(realpath "$(dirname "{}")" --relative-to="$HOME")${Color_Off}' \; -execdir git pull \;
}

Any grey-haired programmer can tell you a heartwarming tale of how they got their grey hair, and it’s guaranteed some flamboyant variant of this: if you create interdependencies between code, config and data, three domains that have independent lifecycles, you too will get grey hair.

Apparently this is too profound because there’s an entire side of tech that has embraced YOLO into a philosophy of This Is The Way™. Fork my git repo, put your data in it and mash it all together. If my repo breaks, yours will break too, taking down all your data with it. Oh, you thought you were going to use my code to solve your problem? My code is your problem. This Is The Way!

Some weeks ago I tried upgrading Quartz, the tool I use to publish a selected part my Obsidian vault as the public website you’re currently reading. I thought I had been careful in preventing their code from contaminating my config and data, but this meant the fixes I had made to their code (awaiting upstream acceptance) came undone with the upgrade. If I pushed the merge, auto-deploy would have happily taken down my website, so now I was locked out of data edits until I fixed the code first.

Last year I took the bait to upgrade from Vim to Neovim, on the appeal of syntax highlighting using a language server instead of regex. I tried Helix first but it didn’t stick because the keybindings are not Vim, and Vim or Vi are everywhere. Helix doesn’t have plugins and that’s very, very appealing, but I reluctantly moved on to Neovim, trying to make sense of its many competing plugin ecosystems. I tried one and it gave me a headache, so I nuked the config and tried another, repeating until I had something working. I didn’t keep notes during this time so I have no idea what my final setup is, but it was certified Works For Me™ when I emerged from that rabbit hole. Fast forward to today: to edit any file, I have to wait out this splash screen of error messages. They used to cover the entire screen but I’ve heroically dismissed them – no clue how – and these remaining few are also complaining about something I have no clue about. What exactly is failing to install, and why should I care if the editor still works?

Whichever of the competing plugin ecosystems I deployed this from – I’m not even sure which – had a deploy script that mashed together multiple git repositories and config files to make them dependent on each other, and now something is incompatible with something else but this incompatibility is unexpected and has no error handler to inform the user, so I get these cryptic messages that I can… apparently just ignore?

We’ve had generations of grey-hair graduates who channelled their pain to give us data integrity, atomicity, ACID commits, journaled filesystems, CAP theorem compromises, semantic versioning, API contracts and more, and the lesson we’ve learnt is not that cross-domain dependencies are weak links needing extra safeguards, but that having them fail is a rite-of-passage everyone must experience?

If I’m keeping track, this is now Oh-my-zsh, Neovim, Yazi, Quartz, and even Obsidian’s plugin ecosystem. The appeal of plugins, a promise that there will be more to whatever this is today, could go in any direction. More gain, more pain, make a toss, YOLO, This Is The Way™. Just pray that if it blows up, it happens when it still has your attention. I now realise I’ve unconsciously learnt to clear my day before an npm update, because it will be an interesting day. It could end in victory or defeat but never ho hum.

Anyway, where were we? I wanted Chezmoi to manage oh-my-zsh and they made an entire page about their disapproval, and I felt insulted enough to try anyway, pointedly asking it to manage a git repo and not an archive. Then I tried adding one of my custom scripts and Chezmoi… segfaulted. It wasn’t a principled refusal to manage a file inside a git repo, it just didn’t know how? The crashes wouldn’t stop until I removed omz from management, so I figured I’d start with the inner git repos first.

This time in VS Code, where some AI assistant kicked in and added the git repo URLs automatically. Nice™, I was hesitantly going to admit, until I noticed the URL was different. I use zdharma-continuum/fast-syntax-hightlighting, but the AI was suggesting zdharma/fast-syntax-hightlighting. Was I on some less popular fork? Should I switch? I looked and immediately recalled this one from a furious Reddit conspiracy thread about a particular zsh plugin ecosystem.

Whatever the veracity, this is a real risk when you’re git-pulling random code into a user account that has all of your data and credentials with no access control restrictions whatsoever. This is the other great divergence between Desktop and Mobile. Desktop environments have all given up on multi-user account security. Everything is now in one common superuser account with no internal access controls, across all the major operating systems. Mobile OSes have gone the other way, restricting every app to a distinct user account. (This is also why Termux has such interesting gotchas: it’s not a window into the phone’s OS, but a sandbox compressing a copy of the OS into a single user account.)

I’m using the version of the plugin that I had previously vetted (only basis that Reddit thread, mind) and AI was now slipping in the version I had explicitly avoided? Woah, woah, woah!™

I mean, the dubious source even speaks for itself:

If AI is only representing the collective hive mind from all the existing config it has swallowed, this meant two things:

  1. A lot more people are using the dubious version and the AI-accelerated botnet apocalypse is imminent, but also
  2. If I play along and let it autocomplete more, maybe it’ll suggest other popular plugins that I should be using? I can do all my discovery here instead of trawling Reddit and random forums.

I tabbed my way through a full list of recommendations, then commented them all out because I have no appetite for Russian Roulette today.

Back on track with Chezmoi. Will it accept this little offering? A list of zsh plugin git repos and my custom scripts? Accepted, committed, pushed, and pulled to deploy on another device that already had oh-my-zsh installed – only to break zsh there. I had missed one tiny detail: the plugins were not activated because that happens in ~/.zshrc, which I hadn’t yet placed in Chezmoi’s care because it looked like it needed templating, and my custom keybinding script was depending on the zsh-shift-select plugin having done some setup first. My script, written for gotcha-mitigation, caused this new gotcha because (a) it had no defensive assertions to confirm the correct environment, and (b) if there’s a way to express an explicit dependency, I’m not aware of it. (The shell command source isn’t it, for it expresses “include” not “requires”).

This is my user error, of course, but I have to ask again: what is the point of dotfile management if it takes this much care and attention to dependency-graphs to make the config reproducible? I could have just copied the files over myself to multiple devices, sorting out issues along the way with concise mental notes to “do this before doing that”. This is far easier than learning the esoteric syntax and data model of the tool, and then hoping for playback without a hitch. If your target device doesn’t have undoable state (like a filesystem snapshot), how do you even test that the managed config is deployable?

I will finish putting oh-my-zsh in Chezmoi by relocating $ZSH_CUSTOM outside ~/.oh-my-zsh as seems prudent (and necessary), but how do I get Neovim and Yazi under management without first becoming an expert on how Neovim and Yazi work, so I can tell Chezmoi exactly how to manage them? It’s all uphill here.

To be clear, dotfile management tools do have their use as I’ve stated above, but the opportunity space here is much larger and there doesn’t seem to be good tooling for it. I’ll try to articulate:

  1. When tinkering across devices, I realise I want some setup from another device, but until that point it hadn’t registered as worth reproducing.
  2. I don’t recall how I set that up because it came together over a period of time.
  3. I do not want to clone that device’s entire state, just that part of it, but I’m unsure of the boundaries and dependencies.
  4. I may be able to identify what needs to be reproduced if I have a delta against a prior state, because looking at it through this filter will improve recall – same as how a folder is useful to look at some files in isolation from all other files.
  5. Therefore the game is in delta production, like a git diff but across domains.

Deltas:

  1. Filesystem state diffed against a snapshot, where the snapshot can be from a filesystem feature like an overlay fs, against a backup, or against a reference ISO image. This will be comprehensive but overwhelming, and will need additional abstraction deltas.
  2. Shell prompt history to associate commands with filesystem changes, making for easier identification of relevant changes.
  3. Package manager deltas. Files deployed by a package manager can be ignored and replaced with just need the name and version of the package.
  4. If there are nested levels of package managers, reconstruct that graph too.
  5. For git repos, commit hashes will do, but please don’t miss untracked files and nested repos.
  6. Popular apps with in-built package management should also be covered because they will have relevant metadata in their remote repositories and not the local filesystem.

2025-06-19-Thu 17:24: Time to stop because it’s not Wednesday anymore. This is a “daily” update written over two days, though I did take a break to sleep at night – I don’t want my grey hairs from dotfile management. To be continued another day.

Graph View