If you are like me chances are you will have many projects (or experiments) going on at the same time and accumulate new ideas or things to research on a daily basis. In this post I describe the tools and systems I use to help me manage that process - all of them are free or very low cost and can be hosted on your home network if you don't like the idea of keeping work in progress on the public cloud.
Every year over the Christmas break I tend to perform a cleanup of my internal infrastructure and this year was no different, as my setup has been working well for a number of years now it seemed like a good opportunity to share my experience with you - if it's working well for me chances are it will work well for you too.
This post is not meant to be an installation guide, the details of how to set up the individual components are easily found on the internet, I'm simply presenting the tools and applications that I use as well as the reasons I selected them rather than something else. While my setup is possibly overkill for a single person it would definitely be suited to a group of people who work on projects together, a small business or a Makerspace.
GitLab - Source Code and Assets
Git is probably the de-facto standard for managing source code and other project assets. Pretty much anyone who has done any development is aware of GitHub, a site that allows you to create an publicly share repositories.
I keep pretty much everything I work on in Git - management scripts, configuration files, throw away pieces of experimental code and full projects. I like the idea that everything is stored in one place and that I can come back to it at any time and see the full history of what I worked on. I also prefer to keep what I'm working on out of the public eye until it reaches a project read stage at which point I clone the repository to my GitHub account. Event at that stage I prefer to do additional work on the project locally and then push it to the public repository once it has been tested and cleaned up a bit.
Setting up a local Git server isn't a trivial task but it's not overly complicated either. Helper tools like Gitolite can add simple web based management of your repositories and allow you to view your code and contents without cloning the entire repository.
Although I used a Gitolite based server for quite a few years I migrated to a system called GitLab in 2013. GitLab provides a full web based management system with many of the features found in GitHub running on your own server and completely under your control. The GitLab project provides a number of different versions to chose from - the free Community Edition, a paid for Enterprise Edition as well as externally hosted services. The Community Edition provides all the features you are likely to need:
- User and group management for access control.
- Repository management (creation, deletion, archiving).
- Access to the repository over SSH and HTTP/HTTPS connections.
- Public key management for SSH access.
- Per-project wiki pages for documentation.
- Per-project issue management system.
- Continuous integration support (trigger a build when new code is committed).
- Support for Webhooks and a REST API to integrate with other tools.
While earlier versions of GitLab were somewhat complex to set up the new Omnibus distributions are provided through the various Linux package manager systems and can be downloaded, configured and running in under two minutes. I have been running my GitLab instance on a headless Debian system (installation instructions) and it has worked well without missing a beat.
Confluence - Notes and Documents
Like most makers I am learning as I go - this process tends to generate a lot of notes, links to references and accumlated datasheets, reference designs and other documentation. The best way to manage this is using a Wiki - everything is kept in a single place, external documents can be attached to your own notes for local reference and things can easily be cross referenced.
While GitLab includes Wiki functionality it is on a per project level - keeping notes that are universal or related to multiple projects can be a bit painful. Last year I installed a system called Confluence which provides an Enterprise Level Wiki implementation.
The company I work for has been using Confluence for a number of years to manage product design, documentation and production processes - as a result they make a lot of use of the features the product provides - workflow control, external integrations and custom macros. For a small installation like mine a lot of these features go unused, they simply aren't applicable.
Having a single large Wiki has it's advantages - when I do a search for something like FT232R the results will include all pages that reference it even they are related to a separate project, the Wiki system provided by GitLab made this difficult to do as each set of pages was in an isolated silo.
A Wiki doesn't impose or enforce a structure to the documentation so it is up to you how you want to organise things. For myself I have set up separate trees for components, projects, ideas and instructions which seems to work well.
For electronics projects I keep a copy of the schematic and board layout in the Wiki as well which has proved very useful when it comes time to actually build up the project (especially if I am building a board that I designed a while ago). I have an old plasma TV I rescued from being discarded set up in my garage with a Raspberry Pi attached to it - I just bring up the board layout in Confluence to guide my soldering, a lot simpler than printing it out for reference.
Jira - Tasks and Goals
I only recently installed Jira for personal use, like Confluence it has been in use at the company I work for for a number of years so I was familiar with it already. Jira is another product by Atlassian and has the same pricing model as Confluence - a self hosted server license will cost you a one off fee of $10.
I have been using Trello to manage tasks for a long time and it has worked well. The main reason for changing to Jira for me was to use the GitLab integration support and to have a bit more control over things like software releases and version tracking.
Trello uses a Kanban style process - tasks progress a series of columns from left to right, generally Proposed -> Accepted -> Doing -> Done but you can use whatever sequence makes sense for your workflow. Because I use the system to manage personal projects and it can be difficult to schedule time in advance to work on tasks I don't impose any time constraints or set explicit priorities. The goal is to keep the Doing column relatively short - I aim for a maximum of 4 tasks at any given time, which one I work on when time becomes available is determined by what resources I have at hand.
For most people (and small groups) Trello will be more than enough to keep on top of outstanding tasks but it was not quite meeting my needs. Over the past year or so I've moved to breaking things down into much smaller tasks, ideally something that I can complete in under an hour (the process is similar to the Pomodoro Technique). I've found this reduces the number of tasks that get stalled or forgotten in one of the Kanban columns (and it has a positive psychological effect - although there are more tasks in total I am regularly moving things to the Done column).
The large number of tasks was overwhelming my Trello board and making it difficult to sort through them to decide what to queue up next. Jira supports the Kanban model and had a number of useful features I wanted to use as well:
- A REST API and support for Webhooks that I could use to integrate with other services.
- GitLab has built in support for Jira integration - mentioning a Jira task in a commit message will automatically add a comment on that task with a link back to the commit. Using a phrase like 'Closes JIRA-101' in a commit message will automatically mark the task as done for you.
- I'm working on projects (like the IotWeb HTTP/Websocket server for Windows 10 IoT and the upcoming SensHub MQTT rules engine) that have specific release versions and I need to be able to associate tasks with a particular release. Jira has the concept of an Epic that encapsulates this without changing my workflow.
There is a lot of functionality in Jira that I am not making use of and I am finding more useful things I can do with it every day. A recent addition I made was to use the task query API to find tasks that haven't been modified for over two weeks and send a notification by Pushbullet to prompt me to either discard them or move them up the list. Another useful function is showing a graphical representation of how fast I am moving through tasks - if I am not closing off tasks rapidly enough I can go and split them into smaller ones which can then be finished off in smaller time slices.
Home Server Hardware
If you are running these servers on your own network you are obviously going to need something to run them on - my preference has been to use headless Debian based servers and run each application on it's own physical machine. Up until recently I had been using small Celeron based Mini-PCs (single core, 1Gb RAM and 500Gb HDDs) to do the job and they handled it admirably.
Unfortunately I had a hard disk failure on one of these in early November, when I was repairing it I found the others were starting to report the occasional disk error as well. Given they are all over 5 years old it was time to look at replacing the hardware.
I understand that people will suggest using a Raspberry Pi as the server platform (they are small, cheap and consume very little power) and at first glance it probably seems like a good idea. These applications are very disk intensive though (all of them use something like MySQL or PostgreSQL for data storage) and the SD card based disks on the Raspberry Pi really aren't suited for that sort of workload.
A friend was upgrading his desktop system from an older Intel Core i5 so I took that off his hands, boosted the memory to 16Gb, added a pair of 2Tb hard disks and installed Xubuntu along with VirtualBox. Each of the three apps got it's own VM with 1Gb of RAM and 200Gb of disk allocated (I gave GitLab 500Gb of disk, I have a lot of repositories in there). The VMs are configured to automatically start on boot so they come back up if the server restarts.
The performance is reasonably snappy (I am the only user of the system though so there is not much of a load on the system) and could be improved by allocating more memory if needed. I haven't found a good remote management system for VirtualBox yet but it does have a rich command line interface so most of the management tasks can be scripted or performed remotely over SSH.
Backups are a lot easier now - I run a weekly cron job that takes a copy of the VDI (Virtual Disk Image) files for each of the VMs and stores it on a file server, this captures not only the application data but all the configuration files and other tweaks made to the system (which should all be documented on your Confluence server of course :P). In the event of a catastrophic failure I can always bring up the backup VDI files on any machine that supports VirtualBox - getting back up and running in a very short time.
Another solution may be to run the servers inside Docker containers, all of the applications listed in this article support this mode of operation. From what I have read so far though this would be a little more difficult to set up and manage - if you are already familiar with the technology it may be a good alternative for you.
Finally you will probably want to allow access to your servers from outside your home network. A simple way to do this is to set up a reverse proxy to handle all incoming requests at a single point and dispatch them to the appropriate server. All you need to do then is use port forwarding on your router to direct the incoming HTTP traffic to your proxy.
I added a few extra steps to this, I registered a domain name (I use DynDNS for registration and DNS hosting) and generated a SSL certificate with Let's Encrypt to ensure all traffic to and from the servers wasn't in plain text. Using NGINX as the reverse proxy lets you do name based request routing so https://jira.mydomain.com will connect to the Jira server, https://gitlab.mydomain.com will connect to the GitLab server, etc.
I also wanted to be able to refer to the servers by the same URL both inside and outside my home network which was a little bit trickier to set up. Because mydomain.com is being resolved to the external IP address of my router trying to connect to it from inside the network doesn't work. My solution was to run my own DNS server internally and set it up as an authorative server for mydomain.com that resolved jira.mydomain.com to the correct internal address of the NGINX server. Finally you need to configure your router to use your DNS server instead of the ones provided by your ISP.
These components (NGINX and DNS) need a server to run on as well - another VM will do the trick or, as in my case, an old Rasberry Pi Model B is more than good enough to do the job.
So there you have it - that is the infrastructure backing The Garage Lab and the Sensaura Project. Pretty much everything I do from electronics projects to blog posts is managed, stored and documented using those applications; all of them are sitting on a single physical server stashed in the back of my garage. Having these tools at my disposal has made keeping track of (and more importantly, completing) tasks a lot easier than it used to be.
As I mentioned at the start of the post this was not intended to be an installation guide, if you are interested in setting up something similar for yourself or your organisation and have some questions about the setup details please feel free to ask them in the comments below - I will do what I can to help or at least point you in the right direction to get the information.