Sometimes our brains fail us and we just don't feel very creative. Fatigue can strike even the most enthusiastic of F/LOSS contributors and there are a number of ways in which it can be combatted. However the one way I find works for me time after time is simply to 'code for the lulz'. The Internet is full of doing people who do stuff for short-term amusement so why not bend that approach to productive output.

When I can think of nothing better to do, but I've promised myself that I will lay off Minecraft for the week and I've reached my social-media saturation point, I often find myself sat at an empty editor window pondering what I should write.

This is the perfect time to learn a new programming language, or have a go at something entirely un-related to projects you normally work on. For example, once I was so fatigued by my project to write a web application framework that I instead learned how regular expressions work and wrote a thorough implementation of regular expressions (and through that, DFAs and NFAs) which I have subsequently never used. But I had great fun doing it and learned an awful lot about how to write regular expressions which will be evaluated efficiently.

I'm not entirely sure I'll ever use the understanding of finite automata that I gained, but I sure know I had a crazy amount of fun writing that awful code. Sometimes all you really need to do is take a step back, pick something odd, and code for the lulz.

What are network namespaces

Network Namespaces are a Linux feature, that allows different processes to have different views of the network.

Aspects of networking that can be isolated between processes include:

• Interfaces

  Different processes can connect to addresses on different interfaces.

• Routes

  Since processes can see different addresses from different namespaces, they also need different routes to connect to networks on those interfaces.

• Firewall rules

  Firewalls are mostly out of scope for this article, but since firewall rules are dependant on the source or target interfaces, you need different firewall rules in different network namespaces.

How do you manage network namespaces

• When a network namespace is created with the unshare(2) or clone(2) system calls, it is bound to the life of the current process, so if your process exits, the network namespace is removed.

  This is ideal for sandboxing processes, so that they have restricted access to the network, as the network namespaces are automatically cleaned up when the process they were isolating the network for no longer exists.

• A privileged process can make their network namespace persistent, which is useful for if the network namespace needs to exist when there are no processes in it.

  This is what the ip(8) command does when you run ip netns add NAME.

• The ip netns delete NAME command undoes this, and allows the namespace to be removed when the last process using it leaves the namespace.

• ip netns list shows current namespaces.

  $ sudo ip netns add testns
  $ ip netns list
  testns
  $ sudo ip netns delete testns
  $ ip netns list
  

• A network namespace is of no use if it has no interfaces, so we can move an existing interface into it with the ip link set dev DEVICE netns NAME command.

  $ ip link show dev usb0
  5: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
      link/ether 02:5e:37:61:60:66 brd ff:ff:ff:ff:ff:ff
  $ sudo ip link set dev usb0 netns testns
  $ ip link show dev usb0
  Device "usb0" does not exist.
  

• Network namespaces would be of limited use if we can't run a command in them.

  We can run a command in a namespace with the ip netns exec NAME COMMAND command.

  $ sudo ip netns exec testns ip link show dev usb0
  5: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
      link/ether 02:5e:37:61:60:66 brd ff:ff:ff:ff:ff:ff5: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
      link/ether 02:5e:37:61:60:66 brd ff:ff:ff:ff:ff:ff
  

• There is no standard daemon to start persistent network namespaces on-boot, however, the PrivateNetwork= option to systemd service files can be used to start a process in a network namespace on boot.

Uses of network namespaces

Network namespaces can be used to sandbox processes, so that they can only connect through the interfaces provided.

You could have a machine with two network interfaces, one which connects out to the internet, the other connected to your internal network. You could set up network namespaces such that you can run the ssh server on your internal network, and run a web server in a network namespace that has the internet-facing interface, so if your web server is compromised, it can't connect to your internal network.

docker and systemd-nspawn combine network namespaces with other namespaces to provide containers. docker is primarily interested in application containers, while systemd-nspawn is more interested in system containers, though both technologies can be used for either.

A more novel use of network namespaces is virtual routers, as implemented in OpenStack Neutron.

Network namespaced routing

We previously mentioned that routes were a property of network namespaces, that virtual ethernet devices have a pair of ends, and that we can move a network interface into another network namespace.

By combining these features, we can construct a series of virtual computers and experiment with routing.

Two node networking

To start with we're going to add another virtual computer, start a simple echo server on it, and configure the network such that we can connect to it from our host computer.

As we've done in previous articles, we're going to make the left and right virtual ethernet pair.

$ sudo ip link add left type veth peer name right

But to demonstrate routing to another machine we need to create one, so we're making another network namespace, as we only care about its networking.

$ sudo ip netns add testns

We're going to move one half of the virtual ethernet device pair into another namespace, to simulate a physical link between the two virtual computers.

$ sudo ip link set dev right netns testns

We're going to say that we connect to interfaces in the testns namespace by connecting to addresses in the 10.248.179.1/24 subnet.

$ sudo ip address add 10.248.179.1/24 dev left
$ sudo ip netns exec testns ip address add 10.248.179.2/24 dev right

We're also going to say that there's another network in there on the 10.248.180.1/24 subnet.

Rather than having any more complicated network interfaces, we assigning it to the loopback interface.

$ sudo ip netns exec testns ip address add 10.248.180.1/24 dev lo

Now that we've assigned addresses, we need to bring the interfaces up.

$ sudo ip link set dev left up
$ sudo ip netns exec testns ip link set dev right up
$ sudo ip netns exec testns ip link set dev lo up

This has created a chain of interfaces linking left → right → lo. Where left is in your namespace, and right is in testns with its own private lo.

Now we can demonstrate sending messages through the network namespaces, by creating a server on the lo interface inside testns.

Start an echo server in the test namespace by running this command in a new terminal.

It is important to run this in another terminal, as if you background the process, netcat may decide it doesn't need to actively output data as it recieves it.

$ sudo ip netns exec testns nc -l 10.248.180.1 12345

If we were to try to connect to it, we would not be able to send messages.

$ nc -v 10.248.180.1 12345
nc: connect to 10.248.180.1 port 12345 (tcp) failed: Network is unreachable

This is because your host namespace does not understand how to reach that address, since it is not an address in your network namespace, nor is it an address on any of the subnets in the namespace, and there are no defined rules for how to reach it.

So let's add a rule!

$ sudo ip route add 10.248.180.0/24 dev left via 10.248.179.2

This says that you can find the 10.248.180.0/24 subnet by sending packets to the 10.248.179.2 address through the left interface.

You should now be able to type messages into the following netcat command, and see the result in your other terminal.

$ nc -v 10.248.180.1 12345
Connection to 10.248.180.1 12345 port [tcp/*] succeeded!

3 namespaces

This works fine, but in the wider internet you need to connect through long chains of computers before you reach your final destination, so we're going to use 3 network namespaces to represent a longer chain.

We are going to have two extra network namespaces, called near and far.

$ sudo ip netns add near
$ sudo ip netns add far

We are going to create a chain of network interfaces called one, two, three and four.

$ sudo ip link add one type veth peer name two
$ sudo ip link add three type veth peer name four

one will be in our network namespace, connecting to two in the near namespace, and three will connect to four in the far namespace.

$ sudo ip link set dev two netns near
$ sudo ip link set dev three netns near
$ sudo ip link set dev four netns far

This produces a chain one → two → three → four.

To speak to these interfaces, we need to assign address ranges, so for our host to near link we will use 10.248.1.0/24, for our near to far link we will use 10.248.2.0/24, and for our destination address we will use 10.248.3.1/24.

$ sudo ip address add 10.248.1.1/24 dev one
$ sudo ip netns exec near ip address add 10.248.1.2/24 dev two
$ sudo ip netns exec near ip address add 10.248.2.1/24 dev three
$ sudo ip netns exec far ip address add 10.248.2.2/24 dev four
$ sudo ip netns exec far ip address add 10.248.3.1/24 dev lo

We need to bring these interfaces up.

$ sudo ip link set dev one up
$ sudo ip netns exec near ip link set dev two up
$ sudo ip netns exec near ip link set dev three up
$ sudo ip netns exec far ip link set dev four up
$ sudo ip netns exec far ip link set dev lo up

Now let's this time start our server in the far namespace, so we can be sure that we can't connect to it directly from our namespace.

As before, start this echo server in a new terminal.

$ sudo ip netns exec far nc -l 10.248.3.1 12345

Now let's try to connect to it from our namespace.

$ nc -v 10.248.3.1 12345
nc: connect to 10.248.3.1 port 12345 (tcp) failed: Network is unreachable

If you try to send a message, it won't arrive. If you inspect the routing table with ip route, you will see that there is no match.

The relevant routing rules are:

$ ip route
default via 192.168.1.1 dev wlan0  proto static 
10.248.1.0/24 dev one  proto kernel  scope link  src 10.248.1.1 

This says that if you're broadcasting from 10.248.1.1 and going to anything in 10.248.1.0/24 then it goes to the one interface. However this doesn't match because we want to go to the four interface which has address 10.248.3.1.

We can make this route to our near interface by adding a new rule.

$ sudo ip route add 10.248.2.0/24 dev one via 10.248.1.2

We can prove this hop works by starting a server in the near namespace (again, in a separate terminal).

$ sudo ip netns exec near nc -l 10.248.2.1 12345

We can try to talk to it:

$ nc -v 10.248.2.2 12345

This doesn't yet work, because it's a bidirectional protocol, so the return route needs to work too.

$ sudo ip netns exec near ip route change 10.248.1.0/24 dev two via 10.248.1.1

This still doesn't work, since while there's now routing rules, they aren't for any addresses which have desinations in the near network namespace, so by default they get dropped.

To change this we can turn on ip forwarding with:

sudo dd of=/proc/sys/net/ipv4/ip_forward <<<1

Now we can talk to the near namespace, but we can't yet talk to the far namespace. We need to add another routing rule for that.

$ sudo ip route add 10.248.3.0/24 dev one via 10.248.1.2

As before, this means you can route to the near namespace. However, it doesn't know how to reach the far namespace from there, so we need to add another routing rule.

$ sudo ip netns exec near ip route add 10.248.3.0/24 dev three via 10.248.2.2

Now you can reach the far namespace, but it's still not working. This is because tcp is a bidirectional communication protocol, and it doesn't know how to send the response to a message from 10.248.1.1, so we need to add another rule.

$ sudo ip netns exec far ip route add 10.248.1.0/24 dev four via 10.248.2.1

Conclusion

I think you will agree that it's far too much faff to set up this routing just to talk to a machine on another network.

You are likely thinking that there must be a better way, and you are correct, there's a couple of ways, but we will cover those in later articles.

Once upon a time a great King wrote a glorious script called foo1.2.

However, after the fifth or sixth person, the King grew tired of explaining to people how to use his script. To remedy this he got the royal scribe to write a little README containing everything his subjects needed to know to use the script.

By the release of foo2.2 it had grown into a host of scripts and the README became unmanageable. At this point the king decided to commision the writing of man pages.

Many peaceful years later the documentation had grown into a complex web of cross-referenced manpages and dozens of peasants each week were complaining that they could not find the information they needed.

So the king commanded his clerks to simplify the manpages and to create a wonderful wiki to hold all the other information which lived on through the ages, and was marvelled at for thousands of years.

---

Okay maybe this story is completely made up but, wikis are useful for big or small projects. Most people these days are familiar with the format of a wiki; people can easily navigate to the information they need and more importantly, easily edit the infomation there.

There are several wiki engines readily availiable. By far the most popular of these is mediawiki, which was developed for Wikipedia. It is easy to set up and runs almost anywhere (I once ran it on a Rasberry Pi). If you have an aversion to touching PHP to write your own, there are plenty of plugins and themes floating around on the internet.

With the rising popularity of git several git-backed wikis have become popular. GitHub's solution Gollum is very popular, in part because of the wide variety of markup it supports and in part because of the hosting offered by GitHub.

If you are a grumpy old fart like me then you will love git-backed wikis because they allow you to edit from the command-line rather than in the browser. I have known people turn web-editing off completely, but this may not be advisable since it will put an awful lot of people off and is against the spirit of wikis, being that they should be quick to edit. If a passing user does not know how to use git then it will not be quick to edit at all.

The option which Yakking has opted for is Ikiwiki. Ikiwiki is also git-backed and uses markdown. Ikiwiki is painless to get running.

In conclusion wikis are great for large and small projects because they are easy to run, easy to use and quick to edit, for better or worse.

Any software project with users other than the developers should make releases.

• A release is the developers saying that the software is in a good shape and ready to be used. This is important for the users, since otherwise they would have to try various version control revisions to find something that looks like it might work. Linux distributions and others who package software want releases for the same reason.

• Sysadmins want releases for an additional reason: they want to be able to reproduce an installation, so that they can have many identical machines.

• Releases are important for the developers themselves as well. Without releases, it becomes harder to support users, since the first support step is to determine the version the user is running, and that becomes harder if it might be any commit from any branch in the history of the project.

Given that releases are useful, what's a good way to make them? A popular opinion today is that a release is made by tagging a commit in a version control repository. This is the minimum for what constitutes a release, and it's fine for many people. Your author is more picky.

For a proper release, the following is a reasonable checklist:

• A tag in version control (in git, it should be OpenPGP signed).
• A release tar archive, compressed using a suitable compression tool.
  • The archive may contain more files than are in the tagged version control commit. For example, autoconf-generated files for configuring a build.
• A detached OpenPGP signature of the compressed tar archive.

An actual tar archive is necessary because it may not be possible to reproduce the release tar archive at a later date. For example, the way git produce a tar archive may change, and the compression tool may also change.

This even has security implications: it is plausible that an attack could happen by exploiting a flaw in the decompression software, if the attacker can use any compression program. If a release is made, and the release artifacts (tar archive, detached signature) are verified and tested, it gets much harder to construct an attack.

---

A short checklist for making a release:

• Ensure the software is in a fit state to be released. It should work correctly; documentation, translations, and release notes should be up to date; and so on.
• In particular, ensure the desired version number is correct and consistently applied across the software.
• Sign the release tag.
• Produce the release tar archive (compressed), including any generated files it should contain.
• Produce the detached signature for the tar archive.
• Produce any other release artifacts.
• Test the release artifacts in some suitable way.
• Publish the release artifacts and push the signed tag.
• Announce the release in a suitable way: on the project website, blog, mailing list, or using a message in a bottle, depending on the project.
• Figure out a way to automate as much as possible of this so it's easier to do the next time.

For most projects, making a release should happen often enough that it pays off to automate most of the process.

Orchestration is, simply, the automated management of software and hardware systems. It is one of the central tools in the toolbelt of a competent sysadmin who has been given the time to do their job properly rather than suffering under a constant deluge of …could you just…s and firefighting the mess they inherited. Orchestration is also sometimes referred to as configuration management.

Over the past few years the number of frameworks has increased substantially, each trying to solve problems perceived in their predecessors. Problems which might vary from "I can't understand the configuration syntax" through "I do not like the language this is written in" to "I don't want any special agent on the target machine."

Here are a few of the currently available and in-use orchestration frameworks. You are likely to have heard of at least one or two of them:

• CFEngine is written in C and is a well establised configuration management engine.
• Chef is written in Ruby and Erlang. It is very popular and a lot of projects offer Chef cookbooks to help people to orchestrate their software.
• Salt is written in Python and is quite popular among adminstrators of large environments which change regularly, for which rapid configuration deployment is needed.
• Puppet is written in Ruby and is also often supported by software projects directly.
• Ansible is written in Python and focusses on use-cases where the systems under management may not be able to have an agent installed on them. Ansible works directly with SSH connections or similar access methods and then runs up python code on the target system.
• Propellor is written in Haskell and unlike the other options in this list, expects you to write your configuration in Haskell. Propellor uses the power of Haskell's type system to ensure that configurations are as consistent as they can be before they are deployed.

There are, of course, many more and you can read all about them on Wikipedia.

If you're responsible for administering any systems (and lets face it, unless you're really unlucky, you at least administer your own personal system) then you should really look into whether you could effectively use an orchestration framework to manage your systems, allowing you to keep configuration under revision control in a controlled and reproducible way.