Today I'd like to talk to you about isolation. Not, as some of you might expect given the topics of my more recent ramblings, about personal isolation - but about software isolation.

Software isolation means different things to different people; from something as common as process isolation which happens on almost every computing platform in the modern world, to something less common such virtual machines provided by hypervisors. For today, let's focus on stuff closer to the latter than the former.

Isolating software has, as in all things, both pros and cons. Briefly, some of the reasons you might want to not consider software isolation may include:

1. It's generally harder work to write software to be isolated easily.
2. Each isolation point is "yet more" software which either has to be written or at least has to be managed by your operations team.
3. If you take a complex system, and distribute it among isolated systems, then you have a more complex system than before.

But, since I'm not going to be a Danny Downer today, let's think more about the benefits. Here are three benefits to counter the detractions I listed above.

1. An isolated software system must have well defined inputs and outputs which can be more effectively documented, mocked, and tested. Leading to:
   1. More easily tested and verified
   2. Easier to debug
   3. Easier to properly document
2. An isolated software system cannot interfere indirectly with any other software system sharing the same hardware or other infrastructure.
3. A system comprised of multiple isolated intercommunicating software systems can be more effectively distributed and thus horizontally scaled.

There are any number of different ways to isolate your software systems from one another. These range in "weight" and "completeness of isolation":

• At the "lightest weight" end of things, we're looking at software written in languages which support virtual environments. This might be Python's venv, Ruby's rvm, or something like luaenv for Lua. These give you only as much isolation as the language's virtual environment support extends to (typically just for the dependencies provided in the language itself).

• Next we might consider chroots which give you filesystem isolation but very little else. This lets you keep the non-language-specific dependencies isolated too, since effectively a chroot is able to be anything from "just enough to run your app" up to a full install of the "user space" of your operating system.

• Building from the chroot context, we can reach for a little more isolation in the form of containers. On Linux, these can be managed by such tools as Docker, LXC, systemd-nspawn, or rkt. This approach uses the Linux kernel namespacing facilities to provide further isolation between containers than the above options can do.

• If containers don't provide enough isolation for you, then you might proceed to hypervisors such as kvm via libvirt, or if you're in a desktop situation then perhaps virtualbox. There're plenty more hypervisors out there though. Hypervisors isolate at the "hardware" level. They effectively provide independently operating virtual hardware elements in the form of emulated hard drives, video cards, etc. Such that the kernel and underlying operating system run just as though they were on real hardware.

• The final isolation option if not even hypervisors cut it for you, is, of course, simply to have more than one computer and run each piece of your software system on a different machine.

There are any number of software packages out there to help you manage your isolated software environments. From schroot for chroots, through the container providers linked above, and on to hypervisor options. But what's even more exciting, as you start to think about scaling your software horizontally, is that there's also any number of providers out there offering services where they look after the hardware and the management layer, and you get to deploy isolated software elements onto their platform (usually for a fee). These might be hypervisor based virtualisation on Amazon AWS, or any number of providers using Openstack. Where it gets taken to the next level though is that those providers are starting to offer a range of isolation techniques, from containers upward, so whatever your needs, there will be software and providers out there to help you with software isolation.

Your homework… (What? You surely didn't think that just because it's a new year you wouldn't get any homework on the first day?) …is to think about any software systems you're responsible for, and whether they might benefit from having their elements isolated from one another more effectively, so that you might glean some of the benefits of isolation.

A few months ago we published an article on how to retire. It was followed by my decision to retire Obnam. Today, we will discuss how you know you should retire from a free software project.

Here are some possible reasons:

• It's no longer fun.
• You no longer have time for it.
• It's no longer a useful project.
• It requires hardware you no longer have.
• You don't like the technical decisions made in the project and changing them now would be too much work.
• The project has no users anymore.
• You have a falling out with the other contributors.
• You just don't want to work on the project anymore.
• You win the lottery and can't be bothered to do anything anymore.
• You become enlightened and realise that computers are preventing you from becoming happy, so you give away everything you have, and move into a cave to meditate.
• The project is finished and requires no more work: it has all the features its users need, and all bugs have been fixed.

You may also like the talk Consensually doing things together? by Enrico Zini from the 2017 Debconf.

ssize = recv(fd, NULL, 0, MSG_PEEK|MSG_TRUNC) lets you see how big the next message in a UDP or UNIX socket's buffer is.

This can be important if your application-level communications can support variable message sizes, since you need to be able to provide a buffer large enough for the data, but preferrably not too much larger that it wastes memory.

Unfortunately, a lot of programming languages' bindings don't make this easy.

Python's approach is to allocate a buffer of the provided size, and then reallocate to the returned size afterwards, and return the buffer.

This behaviour is intended to permit a larger allocation to begin with, but as a side-effect it also permits a smaller one to not break horribly.

Well, mostly. In Python 2 it breaks horribly.

$ python -c 'import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM, 0)
s.bind("/tmp/testsock")
s.recv(0, socket.MSG_PEEK|socket.MSG_TRUNC)'
Traceback (most recent call last):
  File "<string>", line 4, in <module>
SystemError: ../Objects/stringobject.c:3909: bad argument to internal function

Python 3 instead returns an empty buffer immediately before reading the socket.

$ python3 -c 'import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM, 0)
s.bind("/tmp/testsock")
m = s.recv(0, socket.MSG_PEEK|socket.MSG_TRUNC)
print(len(m), m)'
0 b''

You can work around this by receiving a minimum length of 1.

$ python -c 'import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM, 0)
s.bind("/tmp/testsock")
m = s.recv(1, socket.MSG_PEEK|socket.MSG_TRUNC)
print(len(m), m)'
(4, 'a\x00n\x00')

The returned buffer's length is that of the message, though most of the buffer's contents is junk.

The reason these interfaces aren't great is that they return an object rather than using a provided one, and it would be unpleasant for it to return a different type based on its flags.

Python has an alternative interface in the form of socket.recv_into, which should fare better, since it can return the size separately, it should be able to translate a None buffer into a NULL pointer.

$ python -c 'import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM, 0)
s.bind("/tmp/testsock")
m = s.recv_into(None, 0, socket.MSG_PEEK|socket.MSG_TRUNC)
print(m)'
Traceback (most recent call last):
  File "<string>", line 4, in <module>
TypeError: recv_into() argument 1 must be read-write buffer, not None

Unfortunately, this proves not to be the case.

In Python 2 we can re-use a "null byte array" for this purpose.

$ python -c 'import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM, 0)
s.bind("/tmp/testsock")
nullbytearray = bytearray()
m = s.recv_into(nullbytearray, 0, socket.MSG_PEEK|socket.MSG_TRUNC)
print(m)'
4

Unfortunately, Python 3 decided to be clever.

$ python3 -c 'import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM, 0)
s.bind("/tmp/testsock")
nullbytearray = bytearray()
m = s.recv_into(nullbytearray, 0, socket.MSG_PEEK|socket.MSG_TRUNC)
print(m)'
0

Like with plain recv it returns early without waiting for the message.

I had hoped to provide a counter-example of a programming language that provided a way to expose this as part of its standard library.

Distressingly, the best behaved standard libraries were the ones that exposed the system calls directly with all the lack of safety guarantees that implies.

In conclusion, MSG_TRUNC is a thing you can do on Linux.

If you want to use it from a higher-level language than C you won't find a good interface for it in the standard library.

If you find yourself in a position to design a language, please bear in mind people may want to do this on Linux, so at least provide access to the un-mangled system call interface.

There are any number of metrics, heuristics, entrail-reading techniques, etc. which can be used to determine whether or not a project is "alive" for a variety of meanings of that word. Depending on what you need to use a project for, you might need to make such an assessment and so this week I'd like to give you a few ways to assess a software project's "liveness". In addition, you might be a project developer who needs to assess the "liveness" of something you're working on yourself. (For example, to check it's well organised for your expected user-base).

If you're trying to use a project in a production situation, then you likely want to look to whether the codebase is relatively stable, whether it makes sensibly structured releases with some kind of semantic versioning, what its tracker looks like (relatively few open BUGS, short times between bug reports and fixes), and what its documentation looks like. You may also look for evidence that others rely on the project, but that's perhaps less important. It's fine for the codebase to have a lot of churn providing there seems to be some commitment to a stable/reliable interface (be that a CLI or an API).

If, on the other hand, you're looking for a project to use in a fairly experimental situation, you may be looking for a project whose developers are very active and responding well to feature requests, bug reports, etc., however, here the number of open bugs is much less relevant than the way in which the developers react to new uses of their codebase.

If you're looking for a project to contribute to, then assuming you've already determined your own interest in the project to be high, I'd suggest that you look to the developers/community: how active (or otherwise) they are, how well they respond to comments/queries/issues/bugs etc.. A project which is beautifully written but whose developers take months over responding to simple queries might not be suitable for you to join. On the other hand, a nicely written project whose developers are essentially absent is very much ripe for an offer of adoption and/or simple forking.

(As a corollary, if you're a project developer and someone turns up looking at your project and you know it's not right for them, consider pointing them somewhere else.)

So, in summary, useful things to look at include:

• The churn in the project codebase
• Whether the project has adopted semantic versioning
• Whether the project makes lots of bugfix releases or seems stable
• How well the project team responds to new issues/bugs
• How well the project accepts contributions.

This week your homework is to assess a project you're part of, with the eyes of someone who might be coming to it to use/contribute-to. See if you can find ways you could improve your own projects, and give it a go.