July 03, 2018

In setting out to write up a little status report of what I’ve been up to recently, I found myself thinking of military acronyms.

Combat, not surprisingly, engenders a dark cynicism, leading to some amusing abbreviations for situation reports. A few examples:

• SNAFU (Situation Normal - All Fucked Up)
• TARFUN (Things Are Really Fucked Up Now)
• FUBAR (Fucked Up Beyond All Recognition)¹

No doubt there are many more.

I suspect that programmers generally share the same attitude to planning - and the sense of how much any plan matches reality - that most troops on the ground in the thick of it do.

This may explain why, when setting out to describe my current situation, an acronym of my own came to mind: SNAIL.

The good news is that it’s is not (quite) so negative as the miliary ones.

It is though quite descriptive of the state I normally find myself in - and also the pace at which things tend to move as a result.

SNAIL: Situation Normal, All Inter-Linked.

Focus (Ooh Look, A Unicorn)

I’m kind of between projects at the moment, taking some time out and investigating new avenues. As a result, I’ve been tinkering with quite a few things. This could be regarded as a lack of focus (but is a quite deliberate choice on my part).

I’m interested in Swift - not because I think it’s the new shiny or is going to solve all the world’s problems - but because it’s evolving at a healthy pace and seems to have an interestingly open design process. We weren’t really in a position to use it on Sketch², so since moving on I’ve been eager to get some experience with it.

For largely political reasons, though also fuelled by some frustration with Apple’s desktop hardware, I’m also interested in Linux. Or, more accurately, in platforms-that-are-more-open-than-macOS-and-dont-totally-suck-by-not-being-macOS.

Meanwhile, in a very background sort of way, I’ve started tinkering with a game. The code name is “Problem”. Possibly as in, “Houston, we have a…”.

This could be regarded as: Project #1: Problem

I used some Apple-only technology to quickly bootstrap things - which was the right decision, but probably in the long term I want to replace them with something I have more control over. Especially given the Linux interest.

Linux

As I said above, I’m interested in Swift, and interested in Linux.

Working with Swift on multiple platforms brings a few challenges. The absence of Xcode on Linux being an obvious one.

For everything but Swift/Objective-C coding, my editor of choice has long been Atom, so I started wondering how feasible it would be to use it instead of Xcode for swift.

Xcode is more than just a text editor of course - it’s an IDE with a deep understanding of Swift, which allows it to do syntax highlighting, show warnings and errors in context, let the user set breakpoints and symbollically debug, and so on. So I got interested in working out how to integrate those features into Atom.

Leading to Project #2: atom-ide-swift.

This is very much a work in progress, and at a very basic stage right now.

It’s useable to edit and build code now, however. I know: I’m using it.

It could do with a lot more work however, and better integration with Atom’s new IDE features. There are some third party projects which would really help to flesh it out, and as if I didn’t have enough distractions of my own, I’ve been sorely tempted to get involved with them.

Build Actual Things

As well as it’s editing and interactive debugging features, Xcode is also a sophisticated build system, which lots of support for bundling non-code products and resources, running additional tools, and so on.

Because of the two fairly orthogonal interests of Swift and Linux, I’m also interested in the Swift Package Manager, which is a way to specify and build Swift modules, and has the advantage of being platform-neutral³.

Working with Swift on Linux has lead me to the realisation that SPM does a fraction of what Xcode does, and in particular it’s missing pretty much everything that you’d need to actually build a large product with bundled resources or a sophisticated build chain.

Trying to explore ways of fixing this has lead to another interesting diversion.

Hence Project #3: Builder.

I wanted to work on this on Linux as well as macOS, so, unsurprisingly, I needed the Atom Swift stuff. Thus: linkage!

I’ll probably need this, or something like it, if I ever want to really make my game, use Swift to make it, and have it build on Linux. More linkage!

Too Many Projects Already

Meanwhile, shifting around between machines and projects also reminded me of how painful it was to try to keep track of which projects were where, and make sure that the version on each machine was up to date.

In days of yore, my old habit - which predated the existence of things like git / github - was to keep absolutely everything important on every machine⁴. I’d arrange things in a nice reverse-domain folder hierarchy, so that as long as you knew the location of the root, everything else had a logical place.

Going through various machines trying to remove a lot of projects that were now obsolete - and should theoretically be backed up in git anyway - turned out to be a pain when I realised that I needed to do a lot of manual checking that things were in fact correctly pushed and didn’t have half-finished local changes outstanding.

I concluded that, at a time when anything I do pretty much must be in git as a matter of course, it makes no sense any more to keep a load of old or parked projects on my machines. And if I was going to have fewer projects on my machine, having a complicated folder structure made no sense either.

What would be a lot easier would be some sort of management tool - kind of like a cross-platform package manager for projects - which let me “install” the projects I needed into a known location (maybe ~/Projects/), when I needed them.

With a tool like this, I could add just the projects I was currently working on, keep them up to date across machines.

When it came to removing projects from my machine, a tool could automatically perform checks first to ensure that there were no local-only changes and that it was safe to just throw away the local copy.

I could also easily bootstrap a machine by just installing the tool, then giving it a list of projects to install.

Having the tool know which projects were “installed” would have other advantages. It opens up the possibility of automatically fetching updates for all projects, automatically pushing any changes, adding command-line support for quickly navigating to them, and so on.

An Aside: My Scripting Environment

Peripherally related to this, working across both platforms (and having to set up a new Linux box) brought home to me the state of the cobbled together set of scripts, aliases, symbolic links, and the like which I’ve been carrying around and evolving for the last twenty or more years - most of which live in a single (and now rather monolithic) git repo.

Many of these scripts and tools are quite minor, but add little quality-of-life things to my setup: easy ways of navigating to projects, a standard shell prompt, various bits of auto-completion support, aliases for connecting back to known machines in my setup, ways of sharing settings across machines for things like git, atom, and so on - often by symbollically linking files or folders from the place that an app expects them, to another place in my scripts repo.

Having all of this dumped into a monolithic repo was feeling unweildy, especially when a lot of the repo’s content was now obsolete, or only appropriate to one platform.

Some of these scripts also rely on hooking into bashrc, or placing symbolic links to places like ~/. or /usr/local/bin in a way that was also a bit inflexible and hard to manage. I had a “bootstrap” script that theoretically set everything up right, but it was a bit one-shot and hard to test.

It occurred to me that what I really needed was to be able to split these utilities into individual git repos, install/remove them easily, and have some hooks execute when they were installed/removed. The install hook could manage the creation of symbolic links, and generally hook into bash, git, or whatever. The remove hook could do the reverse, making sure that each individual bit of support could be uninstalled without leaving a mess behind.

This Is Sounding Familiar

At some point I realised that an easy way to add/remove projects (based in git repos), and an easy way to add/remove my personal tools (also based in git repos), was basically the same problem.

Being able to run hooks whilst doing so, and being able to choose where the local copy ended up (somewhere obvious like ~/Projects/ for the former, somewhere more hidden like /usr/local/share/ for the latter) was a feature that could be useful in both cases, and was in any case a per-package kind of setting.

So in summary, I needed a kind of package manager like Homebrew or npm, that wasn’t tied to a particular tools ecosystem or OS platform.

Thus was born Project #4: xpkg.

I wanted to work on this on both platforms, hence I needed the Atom Swift stuff for it. I also wanted to build some features into it that required a more complex build system than SPM is capable of. Thus I needed Builder.

Four Projects Doesn’t Sound To Bad

So all of this is really just a long winded way of talking what I’m currently working on / playing with.

The game is on hold for now (mostly because of the time I’m spending in Linux), but I plan to get back to it when I’ve built out some infrastructure. At the very least, I intend to modularise it in a way that allows most of the code to be built and tested across both platforms.

The Atom/Swift stuff needs work, but I’m kind of using it, so evolving it slowly.

I’m nibbling away at various aspects of Builder, mostly as a result of using it.

The thing I’m spending more time on right now is xpkg - which of course uses Builder for it’s build, and which (in Linux at least), I’m using Atom to work on.

These four projects are up at the top of a pyramid, but inevitably there’s more clutter underneath.

Builder itself isn’t really one project anyway, it’s a small cluster of projects. It’s deliberately modular, with the idea that you can pull in other tools that you need for your build process - tools which themselves are just Swift packages.

In addition, both Builder and xpkg are command line tools that share some common requirements. From my time working on tools for Bohemian (which were written in Python), I got familiar with Docopt. I like it as a solution for the command line, so wanted to use it in Swift - but it turned out to need a bit of work to build on Linux. I’m starting to see some other common command-line patterns appearing in both xpkg and Builder, which I think are going to get split out into a separate package.

My old legacy scripts and tools need splitting up to work with xpkg. Most of them are in private repos right now, but a few - such as a generic way of hooking into the bash startup process, may be of more general use.

I also have a common Logging pattern that I tend to use as one of the “Hello World”-style test projects whenever I change language. Pretty much all of the above are using that.

I have plans to add more tools to Builder, and/or more tools installable with xpkg, to simplify some other repeating patterns, such as setting up Travis to test Swift modules cross-platform.

So there’s plenty to keep me busy. And surprise, surprise, it’s a case of: situation normal - all inter-linked.

It’s the way I like it. From a casual perspective, things are moving forward so slowly as to almost appear static. I’m building an interconnected set of things though, and hopefully the eventual payoff will result in much faster progress for some future projects.

---