Scripting projects
22nd September, 2020
Over my career the move from monolithic applications to micro-services has come with a number of benefits. In my experience some of the best features are that they can reduce overall complexity, enforce design discipline, scale much better to larger applications (and teams) and allow for more appropriate technical choices. One of the problems they bring is that the developer’s environment can get a lot more complex. Each service you require to run the app is an additional bit of complexity as each service needs certain dependencies installing, needs the correct environment variables, the correct branch etc. Then once you’ve got them running you need to be able to access each one so you can make the necessary change for your work - you might even need to be able make changes to multiple services ones in one go.
As a new developer working on one of these applications can be a bit of a nightmare. You often need a decent amount of the services running every time you want to start work. You need the services started, you might need the tests running as well and then you’ll need a terminal in the right directory as well. For years I’ve been doing this all manually - log on in the morning and spend a couple of minutes getting everything started or if its a specific laptop just for this project then hope everything survived the wake from sleep and hope I don’t need to restart it ever! As you can see this was never sustainable especially when I started working on a mature project that pushed this much further than I have seen before.
For this mature project to run the public facing website required a dozen services and a multi-day set up process using internal tools to import the necessary data. This was also running on my laptop so when I wanted to stop working on the project and work on home projects in the evening I needed to able to stop all of these services and then start them all when I started work the next morning.
To solve this problem I chatted firstly with the existing developers and they were using a
combination of iterm and screen.
All the solutions were very custom to their individual developer and contained many caveats so I
couldn’t take them wholesale and just get going. So I needed to build my own. From a technology the
iterm solution was not ideal, the scripting seemed very complex so might work for the current
project but would need a custom solution for each project. screen looked much better but I have
been listening to a lot of podcasts from people in the ruby community
who have heavily been advocating tmux which I have noodled with
in the past (for adding tabs to alacritty). So if I’m
going to have to learn a tool I think its best I use the one I am starting from a better position
especially as I didn’t have a lot of time to do this.
So I ended creating a pretty small but useful script for the project that would start all the
services and bring them up in their own tabs and two panes for each application - one running it the
other just in the correct directory for git commands, running tests etc. There is also a kill
command to bring it all back down again so that there is nothing left running when I have finished
for the day. This solution has worked well enough and I’ve really enjoyed living without any iterm
tabs and just living in tmux. So much so that I’ve also started creating similar scripts for all
my projects. Even if they don’t require multiple services, just having a nice consistent way of
going into the project and running a single command its all up and running is really nice. This
space needs a lot of work there are a few things that don’t work well yet.
- Starting a dozen or more services starts them ALL at the same time which means that your CPU is overwhelmed for those first couple of minutes
- A proper way of defining shutdown commands for each of the services rather than just killing everything seems necessary
- Some sort of priority order would also be nice but I would hope most services should be pretty resilient.
- Integration of existing tooling like docker compose etc would be nice so that you don’t have to define too much for each app.
- Allowing a user to customise what additional things every app gets or defining some sort of standard layout would be nice as people work in different ways and this is defining the services and how to run the tools not forcing a workflow.
Here are my current versions