Daniel Silverstone Psst, can you keep a secret?

Human society is based, in part, on keeping things secret. Our society (as it is) would fail horribly if everything was publically known. We rely on keeping some information secret to protect our private content. For example, we often protect access services with secrets we call passwords (though if they are simple words then it's unlikely they're very secure). We also use things called cryptographic keys which are large complicated-to-work-with numbers which computers can use to secure information.

If you've been following since we started Yakking then you've probably got some passwords and some keys of your own. Your keys might be things like SSH identities or GnuPG keys. Your passwords will protect things like your computer login, your social media accounts, etc.

As computer users, we have so many of these secrets to look after that you're unlikely to be relying on your own fallible memory. As such you're likely already getting your computer to remember them for you. If you're doing this semi-well then you're protecting all the remembered credentials with some password(s) and/or key(s).

There are many ways of looking after credentials, generating passwords, measuring the quality of passwords, handling keys, policies for retaining and changing credentials, etc. Over the next few articles we'll discuss a number of these points and hopefully you'll all feel a little more secure as a result.

Posted Wed Sep 20 12:00:11 2017 Tags:
Richard Maw Time - Rendering

Time-stamps are great for computers. They are just any other number that they can perform arithmetic on.

Unfortunately humans don't think of time as one big number. Humans think of time as a bunch of small numbers.

There's numbers for the position of their planet around their Star. There's the number of what year it is. The number of which month, which may even have a different name! The day of the month which there isn't even the name number of each month! Humans even count days in a cycle of 7 which have their own names!

For measuring the rotation of their planet humans have more consistent numbers. Each day is divided into 24 hours. Each hour is divided into 60 minutes. Each minute is divided into 60 seconds.

This is all a bit silly because the period of the Earth's rotation isn't even an integral number of seconds, and it's slowing down!

Forunately some considerate humans wrote functions for turning our nice friendly time-stamps from clock_gettime(2) into icky human divisions.

int ret;
struct timespec time;

ret = clock_gettime(CLOCK_REALTIME, &time);

struct timespec

For a lot of humans, seconds are a coarse enough measure of time. They track the time as a number of seconds and the number of nanoseconds between those seconds.

struct timespec {
    time_t tv_sec;
    time_t tv_nsec;

Forunately humans aren't strange enough to commonly render nanoseconds as anything other than nanosecond precision after the decimal point, so this fixed point number can be rendered with the %09ld printf(3) format string.

/* Render the current time as seconds and nanoseconds. */
printf("%ld.%09ld\n", (long)time.tv_sec, (long)time.tv_nsec);

struct tm

In GLibc there are two types of functions for turning a time_t into a form that humans can deal with.

There's gmtime_r(3) for getting the time in the UTC timezone (the name is weird because it used to do the GMT timezone but it's more useful in UTC and they are closely related).

There's localtime_r(3) for getting the time in whatever local time zone your computer is configured to think it is in.

If the time is only going to be shown to the user of the computer then localtime_r(3) may be more appropriate. If it's going to be shared then gmtime_r(3) would be more appropriate, especially if it's going to be shared internationally.


You can use the contents of struct tm directly, but humans have a lot of their own rules for how to display time as text.

This is why they wrote strftime(3). It has its own little language for describing how to display the time.

Unfortunately humans, unlike us perfect machines, are fallible.

Humans have a notion of the same hour's number being used twice in a day and have to add their own way of saying which it is, so their language includes %p to become "AM" or "PM". Not all humans use this, so %p can produce no result. This is a problem, since a format string of just %p will result in a zero-length result, and strftime(3) also uses a zero result to mean that the memory for storing the result in is not large enough.

Humans may think this is fine and that nobody ever wants to just know whether it's "AM" or "PM", or that they just know that they provided enough memory so don't check.

We are perfect machines and want it to work for every input, so want our own wrapper to make it work.

/* Returns 0 on success, -1 on failure */
/* Some format strings can naturally expand to "",
 * which is a problem since the size is returned or 0 if too small.
ssize_t safe_strftime(char *s, size_t *len, size_t max, const char *format,
                      const struct tm *tm) {
    /* Adds a trailing space to format string
     * so it always gets 0 on too small,
     * and NULs it on success before returning length
    char *fmt = NULL;
    ssize_t ret;

    ret = asprintf(&fmt, "%s ", format);
    if (ret < 0)
        goto cleanup;

    ret = strftime(s, max, fmt, tm);
    if (ret == 0) {
        ret = -1;
        goto cleanup;

    s[ret - 1] = '\0';
    if (len)
        *len = ret - 1;
    ret = 0;

    return ret;

Since we perfect machines don't like to waste memory we needed an API that we could retry with more memory. We perfect machines also like tidy functions, so we're going to write one that reallocates internally.

/* Returns length and sets *s on success, -1 and sets errno on failure */
ssize_t astrftime(char **s, const char *fmt, const struct tm *tm) {
    ssize_t ret;
    size_t tbufsiz = 80;
    char *tbuf = NULL;
    size_t tlen;

    tbuf = malloc(tbufsiz);
    if (tbuf == NULL) {
        ret = -1;
        goto cleanup;

    while (safe_strftime(tbuf, &tlen, tbufsiz, fmt, tm) != 0) {
        char *newbuf;
        size_t newsiz;

        newsiz = tbufsiz * 2;
        newbuf = realloc(tbuf, newsiz);
        if (newbuf == NULL) {
            ret = 1;
            goto cleanup;

        tbuf = newbuf;
        tbufsiz = newsiz;

    *s = tbuf;
    tbuf = NULL;
    ret = tlen;

    return ret;

If time needs to be rendered in another way then it's probably better to use the contents of struct tm directly.


Now we have enough to write our program.

int main(void) {
    int ret;
    struct timespec time;
    struct tm tinfo;
    char *tbuf = NULL;

    ret = clock_gettime(CLOCK_REALTIME, &time);
    if (ret != 0) {
        ret = 1;
        goto cleanup;

    if (gmtime_r(&time.tv_sec, &tinfo) == NULL) {
        ret = 1;
        goto cleanup;

    if (astrftime(&tbuf, "%F %T", &tinfo) < 0) {
        ret = 1;
        goto cleanup;

    /* Render the current time as the formatted time plus nanoseconds. */
    printf("%s.%09ld\n", tbuf, (long)time.tv_nsec);

    return ret;

Full source may be downloaded here.

$ make time
cc     time.c   -o time
$ ./time
2017-08-25 19:54:32.341259517

We briefly mentioned that the number of seconds in a year isn't constant.

We're going to elaborate on that in the next article.

Posted Wed Sep 13 12:00:12 2017 Tags:
Daniel Silverstone Moving your comfort zone

Over time, I've discussed with many people how they learn a new programming language. A common theme is to write something you're intimately familiar with over again in a new language in order to learn and judge. For example, some of my friends like to implement standard *NIX utilities such as cat, echo, grep etc. Usually only trivial versions, but it gives them a feel for stdio, file access, command line argument processing, etc. Some implement some kind of algorithmic solution to a known problem in order to get a good feel for control structures and approaches to data structures. I am a little bit in both camps. I tend to try and use controlled problems but I try not to reimplement things over and over. Over-all any of these approaches are excellent in allowing the developer to gain an understanding of how a language works, and how to solve things in it. Learning your tools is key to being an effective developer, and the language you're working in is one of those tools.

Sometimes though, what you want to do is extend your knowledge in a different direction. I am currently embarking on one such journey, and I'd like to tell you a little bit about it.

I have, for many years, been interested in microcontrollers, and writing software which works in severely resource-limited environments (such as a total of 20 kilobytes of RAM, or 128 kilobytes of program storage). I've been working with one such family of computing devices for quite a while now, and among my various projects I have developed USB devices which do fun things such as monitor a bidirectional serial link, providing millisecond timestamped transaction information down the USB port. However throughout it all, USB itself has remained a fairly sealed black box. I used libraries provided by the microcontroller vendor, and just programmed around them.

While I was away at a conference I decided though that this had to change.

I decided there and then, that I would improve my understanding of several topics all at once. I was going to learn more about the microcontroller hardware, the specifics of USB itself, and improve my command of a programming language all at once. This is, as you might imagine, quite a tall order, but I feel that it's worth my while to push my comfort zone a little and do this. To keep me honest, and to keep me working at this, I have also committed to presenting, at a conference, some kind of results of my efforts.

I tell this story here, in part to increase the assurance that there's plenty of people expecting me to do the work, and also in part to encourage you all to push your comfort zones a little. Your homework, because of course there is some, is to think of something you do a lot, that you are comfortable, interested, and engaged with. Now think of a way to twist that so that you learn a lot more about that topic while pushing your comfort zone just a little bit. Why not comment below and tell me what you're going to do, and we can encourage one another just a little.

Posted Wed Sep 6 12:00:06 2017
Richard Maw Time - Real time

Up until now we've only considered time in relation to the machine we're using.

This is only useful for recording events in the vicinity of that machine.

To coordinate events happening elsewhere in the world such as it being midday at Greenwich observatory, it is necessary to synchronise clocks.

We've briefly introduced these concepts in our introduction to time, but more detail is warranted.

Real time

"Real time" or "Wall clock time" is the globally coordinated time system, so if the time of an event is recorded in one location the record of that time can be transmitted elsewhere and they can reason about when this was in relation to their own clocks so they can tell how long ago it happened.

This is handled by computers containing a special piece of hardware called a Real-Time Clock which tracks the current time.

Time Zones

Times were historically defined by the position of the sun in the sky, but the sun is in different positions in the sky depending on where you are, so the coordinated time in any given location depends on where you are.

If you pick any particular location for your definition of coordinated time, you need to know the difference in time from that location to elsewhere for people who have used that place for their time base.

Rather than everywhere tracking what time it is, areas pick a common baseline and everyone in that area uses that time. These are called [Time Zones][] and are usually arranged along political lines.

In practice most computers think in terms of the UTC time zone, and use records of what the difference in time is between the local time zone and UTC.

The difference is normally but not exclusively measured in an integral number of hours since that time in UTC.

Indian standard time is 5½ hours after UTC. North Korea's time is 8½ hours after UTC.

Time zones don't remain fixed. They are political entities so change on political decisions.

Most of the western world observes some form of Daylight saving time where the current time zone depends on what day of the year it is.

This means some times do not exist in every time zone since the time zone is not defined for every part of the year.

As a consequence some local times do not exist everywhere in the world, and the local time is a product of an agreed time in one part of the world, where the local time is being measured, and any political decisions.

An extreme example of this is Samoa skipping the 30th December 2011 and jumping from one time zone to another.

For future proofing you would ideally record the coordinates alongside the UTC time so if a time zone's geopolitical boundaries change the local time at those coordinates may be known.


Your computer's RTC will continue to measure time while your computer is off so when you turn it back on the time is mostly correct.

RTCs are not perfect however, they diverge from real time because they measure time slightly faster or slower.

Handily, because computers are networked they can ask each other what the time actually is. The protocol for doing this is called NTP.

Once a computer has determined the correct time it can use adjtimex(2) to either accelerate or slow the clock until correct, or just set the time to what value it should have.

It is preferable to speed or slow the clock to correct the time unless your computer is still early on in the boot process since you are likely to have programs that will get confused if the time changed a huge amount, but if your computer's clock is sufficiently far ahead or behind then it's impractical to accelerate or slow the clock to the correct time.

Linux's interface to the real time is via the CLOCK_REALTIME clock, though CLOCK_REALTIME_COARSE exists if it's more important that the time is retrieved quickly rather than the result is precise.

Now that we've got a globally coordinated notion of time you can coordinate events globally.

Unfortunately humans aren't trained to think in terms of position and UTC so we need a way to translate this into something human readable, which will be the focus of the next article.

Posted Wed Aug 30 12:00:07 2017 Tags:
Daniel Silverstone Four years of chinwaggery

Four years ago, Lars posted our first article on software freedom. This started an epic effort to help educate those who want to get into free-software. We pledged to discuss technical topics such as shell and C, and non-technical topics such as diversity and motivation. Through our time we've had opinionated series on the truisms of software, or strong technical sets around topics such as systemd.

To give you some numbers, in the past four years we've published two hundred and eleven (or two hundred and twelve if you include this one) articles, by seven authors. Eighty-three were by Richard Maw, seventy-four (seventy-five) by yours truly, and forty-one by Lars Wirzenius. Our most prolific guest author was Will Holland who has contributed seven, Richard Ipsum and Jonathan Maw have contributed two each. Jon Dowland has contributed one, and one of our articles was deliberately anonymously posted one Christmas.

We've received thirty-eight comments spread over twenty-four of the articles. Our most prolific commenters have been Gravious and Richard Maw; otherwise they're spread relatively thinly among the approximately nineteen other commenters. More than fifteen of the thirty-eight comments were by authors, usually in response to comments on their articles.

We have six items in our suggestion-box (thank you) though some are us acknowledging comments. We're grateful for every suggestion and every bit of encouragement you've provided over the years, via the site, IRC, Twitter, etc.

Over the years, our core contributors have had their own ups and downs, and yet we've pretty much managed to get an article out every week no matter what. Indeed we missed one week (I believe) in all that time, and that was because we typoed the date in our publishing queue, and none of us caught it in review. Sadly, it gets harder and harder though, and so please do pop over to the suggestion box if there're topics you'd like to see us cover (even if you just want an expansion of something we've previously discussed). Perhaps come and join us in #yakking on the OFTC IRC network, and if you're feeling very enthusiastic then come and discuss guest-posting since we're always grateful for contributions.

Here's to another four years of articles which you consider worth reading…

Posted Tue Aug 22 12:00:08 2017
Daniel Silverstone Taking time for yourself

Occupational Burnout is a thing which people worry about a lot in the "real world" and it's why we encourage people, particularly information workers, to take regular breaks, to not think about their job outside of work. Indeed, burnout has been referred to as a disease and as such we recommend strongly that you avoid it.

Sadly, those of us who are blessed with the ability and the interest in contributing to F/LOSS tend to work hard, and then effectively "work as play" hard too; which can lead to burning out even more quickly. It can be even worse when the stress originates in your F/LOSS activity because it can be seen as making your 'funtime' less enjoyable.

Burning out is a leading cause of people having to retire from F/LOSS projects and as such, it's really important that you know when to step away from F/LOSS and have a break. You might prefer to spend a few days entirely away from your computer or just spend it immersed in a game. If you're particularly lucky/unlucky then you might just be able to work on a different project for a little while instead. Whatever you do to relax, be sure to increase your explicit relaxation time along with time you're spending on F/LOSS to keep yourself happy and healthy, and able to contribute for a nice long time to come.

Your homework this week is to just think about what you enjoy doing which isn't work or F/LOSS related at all, and go do it. Just shoo!

Posted Wed Aug 16 12:00:06 2017
Lars Wirzenius Retiring from a project

Sometimes you'll want to leave a free software project. It might be one you founded or one you've joined. You may have spent years contributing to it. You may have formed friendships with other contributors: working together on something for a long time tends to be a catalyst for that. However, things change and you may want to leave. You might no longer be using the software. You might not have time to contribute. There might be disagreements about where the project is going or how it is going to be operating. You might be moving to a different continent. You may be switching careers entirely. You may be founding an unrelated company.

A term for this is 'retiring'. Depending on the project, there may be a processs to follow, or you may just wing it.

More importantly, there're various aspects to consider when retiring, especially if you've been involved for a long time.

  • Why are you leaving? It's best to be honest about this, particularly to yourself. Don't rage quit.
  • Will the project survive you leaving?
  • How will users of the software be affected?
  • What about other collaborators on the project?
  • Will there be people to pick up the slack and take over responsibilities? Will they know what to do? Can they ask you for help afterwards?
  • Is there any publicity likely to follow from the retirement? (Probably not, except for high-profile projects.)
  • Are there any assets (computers, etc) that need to be dealt with?

Retiring from a free software project is a lot like leaving paid employment. If you do it well, you make sure all your old commitments and responsibilities are handed over to new people, and no-one is affected too adversely from the change. As a result, you'll be remembered with fondness and you're always welcome back.

Unlike paid employment, there's few hard and fast rules in Free Software. It's important to remember that though your contributions are valued, you're not obliged to continue them if you don't want to.

Posted Wed Aug 9 12:00:08 2017
Daniel Silverstone My F/LOSS Activity...

There has been a growing trend among bloggers, particularly among those aggregated on Planet Debian, in which the blogger writes regularly (usually monthly) about their activity in the Free Software communities since their last report.

This can be used as a way to "keep yourself honest" in the sense that if you're committed to reporting your activity on a month by month basis, you kinda need some activity to report. It can be used as a way to keep your colleagues and co-conspiritors in F/LOSS informed about what you're working on, and cool things you've done which they might be interested in. It might be a useful way to motivate yourself when each period of time you get to write a nice blog posting all about the wonderful things you've achieved and the projects you've improved along the way.

Recently I've been persuaded to join the trend and I'm looking forward to experiencing the effects I've mentioned above. I'd like to suggest that all of you go out and write yourself a blog post all about the wonderful things you've done and then commit yourself to doing so again in X amount of time in the future.

Posted Wed Aug 2 12:00:07 2017
Richard Maw Time - Adjustment

CLOCK_MONOTONIC_RAW reflects the underlying hardware for measuring time.

In the short term this is mostly correct, but hardware isn't perfect, so it can, over long periods, drift and no longer be synchronised with what the rest of the world considers the time to be.

If you know what the time is meant to be elsewhere in the world, then you can adjust your clock to correct it.

Typically your computer will do this with the Network Time Protocol, or NTP,, by asking trusted computers on the internet what the time is.

Actually correcting the time works by using the adjtimex(2) or clock_adjtime(2) system calls.

This can't be done with CLOCK_MONOTONIC_RAW, but can be done with CLOCK_MONOTONIC.

In addition to being correctable, CLOCK_MONOTONIC can be used with clock_nanosleep(2).

This will allow you to sleep for at least the period of time specified, though could be interrupted when a signal is delivered, (which could happen with timer_create(2)).

Similar clocks

If it's more important to get the time quickly, than to get a more precise time, such as if you're profiling real-time software and want to not slow it down, then you can use CLOCK_MONOTONIC_COARSE.

CLOCK_MONOTONIC can be used to time events, but works by counting seconds while the computer is running. This could be a problem if your computer suspends, since then it would stop counting.

The solution to this is the CLOCK_BOOTTIME clock, which will include seconds spent suspended, so could be used to time external events.

So far everything discussed has been somewhat abstract, divorced from what we commonly understand to be time.

This will be rectified in the next article in the time series, where we will be covering "real time".

Posted Wed Jul 26 12:00:07 2017 Tags:

Normally our Unix systems organise the file system in a structure called the Filesystem Hierarchy Standard (FHS). Installing into an FHS has limitations, what would happen if we want to install, for example, two different versions of ruby at the same time? Typically this isn't possible without explicitly specifying a separate installation directory, if we just install to the usual place e.g. /usr/bin then we will just overwrite the previous ruby. So perhaps we would install one ruby into /usr/bin and another into /usr/local/bin, this is fine, but what about dependent libs? Assuming the two different versions of ruby do require different dependencies then we have potentially the same problem that the dependencies for the 1st ruby might overwrite the dependencies for the 2nd ruby.

Nix gets around this to some extent by not using FHS, instead nix installs all files into the nix store, which is usually located at /nix/store. All programs in a nix store are identified by their store path, which is uniquely generated for each distinct nix package. As a result of this, different versions of the same ruby no longer conflict because they are each assigned their own locations within the nix store.

To enable use of programs within the store, nix maintains an environment which is basically a mapping of FHS path -> nix store path, where the -> is a symlink. So for example, let's first install ruby 2.0 into our environment

nix@salo:~$ nix-env -f nixpkgs -iA pkgs.ruby_2_0
installing ‘ruby-2.0.0-p648’
these paths will be fetched (3.43 MiB download, 19.35 MiB unpacked):
fetching path ‘/nix/store/bxm4s71qdyh071ap5ywxc63aja62cbyc-gdbm-1.13’...
building path(s) ‘/nix/store/j649f78ha04mi1vykz601b00ml3qlr9q-user-environment’
created 419 symlinks in user environment

we can see the symlink that was just created to our ruby2.0 in the store,

nix@salo:~$ ls -l $(which irb)
lrwxrwxrwx 1 nix nix 67 Jan  1  1970 /home/nix/.nix-profile/bin/irb -> /nix/store/d2ccapssrq683rj0fr7d7nb3ichxvlsy-ruby-2.0.0-p648/bin/irb

nix@salo:~$ irb
irb(main):001:0> puts RUBY_VERSION

as you can see we're only able to execute the interactive ruby prompt irb because it's symlinked into our environment which is, of course, on the $PATH,

nix@salo:~$ echo $PATH

to prove the point about multiple versions on the same system let's swap ruby 2.0 for ruby 2.4

nix@salo:~$ nix-env -f nixpkgs -iA pkgs.ruby_2_4
replacing old ‘ruby-2.0.0-p648’
installing ‘ruby-2.4.1’
these paths will be fetched (3.13 MiB download, 15.32 MiB unpacked):
fetching path ‘/nix/store/48xrfkanmx5sshqj1364k2dw25xr4znj-ruby-2.4.1’...
*** Downloading ‘https://cache.nixos.org/nar/00hh9w9nvlbinya1i9j0v7v89pw3zzlrfqps72441k7p2n8zq7d3.nar.    xz’ to ‘/nix/store/48xrfkanmx5sshqj1364k2dw25xr4znj-ruby-2.4.1’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3205k  100 3205k    0     0   114k      0  0:00:27  0:00:27 --:--:--  125k

building path(s) ‘/nix/store/7b2mmk2ffmy1c2bxq7r6y9cn6r0nwn8s-user-environment’
created 415 symlinks in user environment

nix@salo:~$ ls -l $(which irb)
lrwxrwxrwx 1 nix nix 62 Jan  1  1970 /home/nix/.nix-profile/bin/irb -> /nix/store/48xrfkanmx5sshqj1364k2dw25xr4znj-ruby-2.4.1/bin/irb

nix@salo:~$ irb
irb(main):001:0> puts RUBY_VERSION

You may be wondering whether this is really an improvement, since although we have multiple versions of the same package installed on our system, we can only have one ruby in the environment at any one time. To deal with this nix provides the nix-shell utility which constructs an environment on demand and runs a new shell based on that environment.

nix@salo:~/nixpkgs$ nix-shell -p ruby_2_0            
these paths will be fetched (4.44 MiB download, 22.57 MiB unpacked):
fetching path ‘/nix/store/m74m7c4qbzml7ipfxzlpxddcn9ah8jrs-gdbm-1.12’...
[nix-shell:~/nixpkgs]$ which irb

[nix-shell:~/nixpkgs]$ irb
irb(main):001:0> puts RUBY_VERSION
=> nil

nix@salo:~/nixpkgs$ nix-shell -p ruby_2_4
these paths will be fetched (3.13 MiB download, 15.30 MiB unpacked):
fetching path ‘/nix/store/wly748apb5r37byvvgq85hshgzcahv0y-ruby-2.4.0’...
[nix-shell:~/nixpkgs]$ which irb

[nix-shell:~/nixpkgs]$ irb
irb(main):001:0> puts RUBY_VERSION
=> nil

We haven't even started to scratch the surface in this intro, there's lots of really exciting stuff I've not even mentioned, like how you can always rollback to the environment at an earlier state: every mutation to the environment is recorded, so every time you install or uninstall a nixpkg a new "generation" of the environment is created, and it's always possible to immediately rollback to some earlier generation. NixOS itself takes all these super exciting ideas and applies them to an entire operating system, where each user has their own environment, so ruby for one user might mean ruby2.0 and ruby for another might mean ruby2.4. Hopefully it's clear now how these different versions of the same package can live in harmony under NixOS.

I hope I've managed to convey some of nix's coolness in this short space, if I have then you should definitely check lethalman's "nix-pills"1 series for a really deep explanation of how nix works internally and how to create nixpkgs from scratch. There's also ofcourse the NixOS website2 and #nixos on irc.freenode.net which is probably one of the friendliest communities out there.

Posted Wed Jul 19 12:00:07 2017