This series of articles (The Truisms) is aimed at imparting some of the things we find to hold true no matter the project one undertakes in the space carved out by our interests in Open Source and Free Software engineering. The first article in the series was If you don't know why you're doing it, you shouldn't be doing it
Previously we talked about how if you no longer want to use a project you wrote, you should give it to someone else who does use it and wants to maintain it. This brings us to the last thing I want to cover in this little series we have been enjoying for the past few months -- namely that since everything is changing (some things slowly, some quickly) if your projects stand still they will simply cease achieving their goals.
As with all rules, there may exist exceptions -- some software out there has remained unchanged for decades. Some people consider it "done", "perfect", "finished", while others simply think of it as "good enough". However, I consider such projects "stagnant", "dead", "uninteresting". Even if your project is, right now, perfect; it will not remain so.
Everything is changing around you. The operating systems your project is meant to run on will be changing, both at the fundamentals such as the kernel or filesystem, or simply in terms of the UIs people grow to expect. Baserock (a project I am privileged enough to be paid to enjoy working on) builds an entire linux-based operating system and no single component of the stuff we build has failed to be updated in at least some way in the last six weeks. Software which doesn't react to this ever-changing landscape will fail to work eventually.
In addition, people's needs change and evolve. Expectations of usability, security, overall functionality, documentation, (and a million other things) change over time, both in terms of individual users, and in terms of the overall userbase you attract to your project. If you have no plan to deal with issues these changes expose, no interest in fixing, maintaining, extending your project, then it will eventually be overtaken by a new upstart of a project which has the energy of interested developers behind it. Don't let this happen to your project while you still care about it.
A few years ago I wrote a program to extract entropic values from the noise in
sound card inputs. I wrote this to get some entropy into /dev/random for a
device I had which otherwise had a tendency to block because it had no
randomness to draw upon for creating secure keys for ssh etc. This program
did its job and worked fine for several years, but recently the APIs it was
using turned out to be sub-optimal and were removed. Since I had no plan to
maintain this software it simply stopped working and someone else had to do the
work to fix it again. I realised that I wasn't using the program and that's
why I had no maintenance plan, so I passed maintainership along to the person
who fixed it. They may at least be more proactive about dealing with
deprecated APIs. Interestingly I did still use the program, but on such an
old system (that wasn't upgradeable in any useful sense) that I'd never noticed
the issue. It wasn't that I was neglecting the program, simply that I had no
plan for maintaining it on modern systems.
So, in conclusion, go to your projects list, which by now should be honed to only projects you have a reason to work on, that you know what you're doing to achieve your goals, that are in revision control and are pushed to a server, that are well tested and are backed up to protect against loss. All your projects should have ways to measure their performance and of course you should be actively using them yourself. So with all these limits in mind, your projects list should be shiny, it should be a glorious testament to your healthy and wholehearted commitment to quality, and as such you should very easily be able to write down what you will do to ensure that your projects keep working as time goes by. Your homework is, quite simply, to keep on top of that list, keep it at the high quality it has now reached, and continue writing new stuff which satisfies you, demonstrates your capabilities, and keeps on working.
Go forth, and engineer the future (so I don't have to).
When you start publishing free software, and gain users, you will inevitably receive some bug reports. This is a good thing. It means other people care about your software enough to tell you when they have a problem.
If you're lucky, your user is reporting a bug that is easily reproduced, and is an actual bug, and you can quickly find and fix the problem. Your user will be very pleasantly suprised to find the fix and the new release when they wake up in the morning, after you've spent the evening dealing with the bug report.
Sometimes, however, things don't go smoothly.
- The bug report might be very unclear, or even wrong.
- It might be difficult to reproduce the problem (if you can't reproduce it, how do you know you've fixed it?).
- The user might be very hostile. They wanted to use your program so much to achieve some goal they have, and your program failed to help them, and this frustrates them a lot, and they take it out on you.
- You might be busy with other things, and not have time to deal with a bug report.
- The bug might be really hard to fix.
Bug reports are, more than anything, about dealing with people. The key to dealing with people is clear, courteous, and prompt communication. If you don't have time to fix the problem, or you need more information, respond as soon as you can. If people know you're busy, they're happier to wait than if they hear nothing.
- Respond quickly, even if it is just to say you're busy and will have a closer look as soon as you can.
- Be courteous, even in the face of provocation.
- If you need more information, be as clear as possible about what you want and how they can get the information you need.
If you get a lot of bug reports, you may want to set up a bug tracking system of some sort. There are lots to choose from. As long as you and your users are happy to use it, it doesn't matter what you choose. However, you may want to wait until the volume of bugs warrants the extra complexity of a tracking system. Or not. Experiment and see what you like.
If you get a lot of users, but few co-developers, you may end up in a situation where new bugs are reported faster than you can deal with them. This is not a good situation to be in. There are a few things you can do:
- Raise the quality of your software. Have fewer reasons for people to report bugs.
- Get people to help you, even if it's just triaging the incoming bug reports.
- Let the bug reports pile up and ignore them and do something else instead.
Various projects have, over time, done one or more of the above. Sometimes the situation becomes better, sometimes it gets even worse. Don't get into this state. Stay on top of your incoming flow of bug reports.
On the other hand, if you do deal with bugs well enough, it is a constant source of pleasant interaction with people who use your software and like it, and you, enough to report problems in it.
Recently there has been an upsurge in interest in the security of systems attached to the Internet. A long while ago, firewalls were considered the state of the art in security. They weren't, even at the time, but that was the general understanding of the populace… that if you have a firewall in place, you're safe. Then came the era of computer viruses enveigling their way into your computer from across the globe and no longer were firewalls the panacea they had been believed to be. Instead we grew a multiplicity of virus scanners to find and eliminate these beasties. As early as 2002, an article on Symantec was written by Paul Schmehl which referenced a variety of sources all calling for something better than the detect and eradicate approach of those scanners. So began the era of "holistic" security approaches including intrusion detection systems, generic detection, threat mitigation etc which have been the mainstay of the "protection" market for some time.
Recently, however, with the move to putting more and more of our personally and commercially sensitive data in a location we have no control over, in the hands of someone whose only possible motivation can be to use the data they hold for their commercial gain, the security of our information is threatened less by the virus attack on our personal computers (an attack vector much more lucratively monetised by DDOS-for-hire and spam-for-hire botnet providers) and much more at risk from attacks on the security protocols put in place to protect the data.
Note, by security protocols I do not mean cryptography. Cryptography is about keeping information secret in the face of mathematical attack. Security (in this instance) is much more about keeping information safe in the face of an attacker exploiting the myriad weaker links between you and your precious precious ones and zeroes. Recently there was a lovely talk given by Peter Gutmann of the University of Auckland entitled Crypto won't save you either wherein he detailed a variety of situations in which cryptography was not broken, but bypassed. Adi Shamir, in the early 2000s, gave a Turing Lecture alongside Rivest and Adleman. Shamir provided the 'status report' on cryptography and among his presentation where three slides which various people have felt worth repeating in various places.
Even more recently, Shubham Shah posted about how he bypassed two-factor auth on various high profile sites and service providers. His article very effectively demonstrates why security is everyone's problem. His attack did not need to touch the cryptography typically employed to protect your bits as they fly across the intertubes. He did not need to perform complicated mathematical operations to determine how to bypass that cryptography. He just made a call at the same time as clicking a button on a web page.
The cryptography itself in all of the above examples was never under attack. While there have been recent stirrings about the possibility of a breakthrough in mathematically attacking the discrete logarithm problem, the majority of effective attacks of late have not been near the mathematics. Security of your data bits cannot be left entirely to mathematics you do not understand. Unfortunately nor can it be left to systems you might previously have thought secure such as the voicemail of your mobile telephone. Security costs, both in money and in convenience, and we're all about keeping costs down in our brave new world. As such many of us don't enable things like the two factor authentication which Shubham Shah bypassed, let alone take care to reduce the attack vectors provided by relying on external means such as voicemail etc.
Cryptography is the problem of the software authors who write the product you rely on. Security is their problem too (designing a protocol for access which will easily authenticate and authorise you while easily detecting and denying imposters is a toughie and can't be done without them) but it's also your problem. And the problem of anyone you share your data with, and the problem of anyone you rely on to keep your data safe. But most importantly it is your problem -- and that's a problem we're unlikely to solve any time soon.
The basic command line utilities in a GNU/Linux system tend to come from the GNU project:
These provide the usual set of tools expected in a tradtional Unix-like environment. These utilities, plus the usual scripting languages (shell -- bash, perl, python) are usually enough. They have certainly served the Unix command line users for decades.
Occasionally someone notices a need for a tool missing from the usual set. In 2007 Joey Hess started a project, moreutils, to collect such new tools in one place.
The moreutils collection currently contains the following programs (quoted from the home page):
- chronic: runs a command quietly unless it fails
- combine: combine the lines in two files using boolean operations
- ifdata: get network interface info without parsing ifconfig output
- ifne: run a program if the standard input is not empty
- isutf8: check if a file or standard input is utf-8
- lckdo: execute a program with a lock held
- mispipe: pipe two commands, returning the exit status of the first
- parallel: run multiple jobs at once
- pee: tee standard input to pipes
- sponge: soak up standard input and write to a file
- ts: timestamp standard input
- vidir: edit a directory in your text editor
- vipe: insert a text editor into a pipe
- zrun: automatically uncompress arguments to command
The most fun of these is sponge. It reads its standard input
completely, and when it has read everything, it writes it to an output
file. The output file may be an input of whatever produces the data
sponge reads, but sponge won't overwrite the file until it's safe
to do so.
$ cat foo.txt
foo
$ (cat foo.txt; echo bar) | sponge foo.txt
$ cat foo.txt
foo
bar
$
ts timestamps its input. This can be used for timestamping log
output from programs that don't do that themselves.
$ ts < foo.txt
May 19 19:34:06 foo
May 19 19:34:06 bar
$
Have a look at the other programs in the moreutils home page. It's
packaged as moreutils in Debian, and other Linux distributions.