Networking is one of the most important things a computer can do, since it's no so useful on its own.
systemd-networkd
Networking under Linux is powerful, but managed by a set of wildly different tools, with different capabilities and with very different interfaces.
To handle this, systemd introduced networkd(8) to provide a common interface to all the network configuration, and is becoming a standard way of doing this across the different distributions.
At the time of writing, networkd(8) is intended to be a replacement for static network config in the form of the Debian-derived interfaces(5) file, or the Red Hat-derived ifcfg(5), rather than the dynamic network config managed by NetworkManager or connman.
However, you can do some form of dynamic network configuration by
putting files in /etc/systemd/network and restarting networkd(8)
by running systemctl restart systemd-networkd.service.
You can configure networks by adding a systemd.network(5) file, or create virtual network devices (such as bridges) with a systemd.netdev(5) file.
This is normally the domain of the system administrator, but systemd ships a few default rules to make networking with containers launched with [systemd-networkd(1)][] work better.
systemd ships a rule for containers to use DHCP on network
interfaces named hostN, to get their IP address. And ships a rule
such that the other end of these host-only interfaces provide a DHCP
server, and use NAT to allow containers to speak to the internet.
systemd-resolved
DHCP isn't the whole story though, as you need names to usefully access the internet, so systemd has some support for DNS.
resolved(8) provides per-interface DNS hostname resolution. This is an improtant improvement over the current state of Linux networking, because network interfaces are a lot more dynamic than they used to be.
It is not unusual to be connected to multiple networks, especially if you need to use a VPN to talk to machines that are not allowed to connect to the internet.
If name resolution is not handled per-interface, then you need to flush the DNS cache to avoid having entries that are only relevant to networks you are no longer connected to.
resolved(8) is only indirectly used, via the standard name lookup procedure specified in the "hosts" line in the nsswitch.conf(5) file, by adding "resolve".
resolved(8) doesn't normally need further configuration, as name resolution is typically per-network configuration, and this network configuration is usually shared by a DHCP server.
However, if this is not reliable, then the name servers can be configured per-interface in a systemd.network(5) file, or system-wide in the [resolved.conf(5)][] configuration file.
We previously wrote a brief introduction to networking, however networking is a larger topic than that, so we've decided to produce a larger series to try and do the subject justice.
Binding services to sockets
We're going to illustrate binding this service from Python since shell is not appropriate, and C is too verbose.
However we're going to use the low-level Python bindings rather than the socketserver class, to show how it works beneath the layers of abstraction.
As with some previous articles, we are going to describe how it works in the form of a script with commentary. You can download the full script here.
#!/usr/bin/python
#chuckle.py
Python's bindings are in the socket module, we're doing a qualified import to resemble the API from C.
from socket import socket, AF_INET, SOCK_STREAM
There's lots of setup stages involved in making a connection available, following the general builder pattern, rather than having a single syscall that takes a huge number of options, as it is more extensible.
The socket() syscall takes the parameters family, socket type and
protocol. This describes exactly what kind of socket to create. This will be
explained in greater detail later, but here we create a socket with an IP
address which acts as a stream.
s = socket(AF_INET, SOCK_STREAM, 0)
One side of the connection needs to bind to a port, so that the other can connect to it. The address specifies which address your machine has, that the service should be served on. '0.0.0.0' means "all my addresses", but we're going to use '127.0.0.1' which means "only my internal address" so that we can't have anything on the network connect to it:
addr = '127.0.0.1'
A port number also needs to be chosen to provide the service. The address denotes how to find the service, while the port represents which of the potential services available should be connected to.
port = 12345
s.bind((addr, port))
The call to listen() is further configuration, of how many connections to buffer before refusing new connections.
s.listen(0)
while True:
The accept() call blocks until there are available connections to service and returns some information about where the connection was from.
conn, hostaddr = s.accept()
The python connection object wraps a file descriptor that can be used as any normal file, so we do some conversion so we can use it as one.
f = conn.makefile('rw')
while True:
f.write('> ')
f.flush()
line = f.readline()
if not line:
break
if line == 'to me\n':
print('to you')
elif line == 'oh dear\n':
print('oh dear oh dear')
break
conn.close()
We can demonstrate that this works with the netcat command.
$ python chuckle.py &
$ nc 127.0.0.1 12345
> to me
to you
> oh dear
oh dear oh dear
^C
$
We can make this service start on boot by creating and enabling a systemd unit like this:
$ sudo install -D -m755 chuckle.py /usr/local/libexec/chuckle.py
$ sudo tee /etc/systemd/system/chuckle.service >/dev/null <<'EOF'
[Unit]
Description=Chuckle server
[Service]
ExecStart=/usr/local/libexec/chuckle.py
[Install]
WantedBy=multi-user.target
EOF
$ sudo systemctl daemon-reload
$ sudo systemctl enable chuckle.service
Connecting to services
We glossed over how connecting works and just used netcat, so let's see what's involved by writing our own program. You can download this script here.
#!/usr/bin/python
#connect.py
import sys
As before we need the same socket definitions to make a socket.
from socket import socket, AF_INET, SOCK_STREAM
s = socket(AF_INET, SOCK_STREAM, 0)
Now instead of binding and listening we call connect().
s.connect(('127.0.0.1', 12345))
The rest is just copying input from your terminal to the socket. Note that this is unidirectional and line based, while netcat is bidirectional and asynchronous.
f = s.makefile('w')
while True:
line = sys.stdin.readline()
if not line:
break
f.write(line)
f.flush()
It is simpler to create connections than bind services, but the address conventions are the same.
Our existing service should still be running, so we can connect to it just like we did before with netcat.
$ python connect.py
> to me
to you
> oh dear
oh dear oh dear
^C
$
I decided that I would write this article one lunchtime, but by the time evening had come around, I had completely forgotten the topic of the article I had intended to write. Fortunately Richard had been at lunch with me and he could remind me that I was to write about Bus Factor -- I can't think of a single better example of why this is important to do.
Bus factor is, quite simply, the number of people who need to become "indisposed" before a project will falter. As the linked Wikipedia article says, it is a measure of the concentration of critical information in a project.
We encounter the issues of bus factor almost every day when we learn something unique in our day-to-day work. As information workers, the generation of unique knowledge is our bread-and-butter and we need ways to ensure that the knowledge we create is not lost if we are. We do this by way of writing things down, telling them to others, putting them in knowledge-management systems and any other thing we can do to ensure that the knowledge is safe.
As software engineers we do this a lot of the time by committing our new knowledge as source code to a revision control system and pushing it elsewhere. Of course, we then sometimes don't improve the bus-factor but instead create a single-point-of-failure for the hardware infrastructure we depend upon.
What's more, as knowledge workers, we have two conflicting desires at the root of our pursuit of knowledge. We like to know things that others do not know since that gives us a measure of smugness or superiority. We enjoy learning new things but above all that we enjoy inventing new things and holding those precious nuggets of joy to ourselves for a short while. Conflicting with this desire is the desire to be seen to be clever/wonderful/inventing new things. Fortunately that second desire is the one which reduces the bus factor for the new knowledge and it is to this second desire that we must adhere.
When software projects begin, typically they begin with one person having a clever idea. That project intrinsically begins with a bus factor of one. Hopefully, if the project gains users, the project will gain further developers and in doing so will reduce its bus factor providing the developers share knowledge and understanding rather than segmenting the project and never encroaching on one another's domain. Fortunately F/LOSS projects tend to be run in a way which encourages the sharing. So if you start a project, please make sure you write stuff down and do your best to get at least one other person involved, ideally before you have users depending on you.
After all, it could be you who goes under a bus tomorrow.
Back in the day, a programmer was expected to be able to turn more or less concrete requirements into code that mostly worked. That was mostly it. It was a long time ago. Life was simple. Computers were simple. Programs were made of spaghetti.
Things have changed. Especially in the world of free software, much more is expected from those who develop the software, either from each person or from the development team as a whole. There's no official list, but here's an opinionated, aspirational one.
A programmer should be good at…
…communication. It can't be emphasised enough how important communication is.
…using version control, and using it well. Most programmers in the world still don't use any version control, so using anything is already being ahead of the majority. The free software developing and using community requires more than that. If your free software isn't in a public repository, you're making it harder for others to collaborate with you.
…automated testing of various kinds.
- Unit testing.
- Integration testing.
- System testing.
- UI testing.
- Benchmarking.
- Static code analysis and other checking tools.
- Basically, given how widespread use free software can easily get, and the impact of it not working properly, it's in the developer's best self-interest to raise quality as high as possible.
…communication. You know what I mean.
…the use of continuous integration and delivery. In general, automating as much mundane, repetitive work as possible makes developers much more productive.
…producing (at least rudimentary) documentation, including web site development. It doesn't have to be pretty or complete, but it needs to be clear.
…using, and providing, build systems. Free software is primarily delivered as source code, and if it's difficult to build it, it's hard to install or support it. This includes support by, say, Linux distribution security teams, who will need to build the software to release a security update.
…being friendly to packaging for various operating systems, etc. This includes being at least somewhat stable as far as interfaces, file formats, etc, so that upgrades from one version to another are plausible. Also, making and supporting releases. Also, managing one's dependencies.
…communication. This should not be forgotten.
…managing their own systems, including development systems, in a reasonable manner. Automating that in some suitable way is preferable.
…being helpful and supportive when users need help. This includes being able to adopt a customer service attitude, but additionally requires a debugging mentality for situations where you can't look at the misbehaving program yourself, and need to extract the information you need from the user, who is already annoyed by things not working.
…understanding that culture is not universal, and either translating and adapting software and documentation to various languages and cultures or, at least, making the software so that this is possible.
…communication. You don't have to be a world-class speaker. If you're prompt, frequent, open, and friendly, people will forgive most things, including the sin being too repetitive.
…working in a team. It's an entirely different way of doing things than working alone. A good team can be a big productivity booster, but it requires everyone to work together well.
…appreciating the security process enough to avoid the worst and most obvious problems. Depending on what one does, this may require quite deep understanding, but all developers now need to understand at least the basics.
…understanding and appreciating any privacy implications of what one is doing.
…communication. I mention it last, but it is by far not the least of the skills.
This is, of course, at the same time an incomplete list and a list that in itself is already nearly impossible for one person to meet. Take from it what you will, but what do you think is missing from the list?
We previously touched on some addresses and ports. For many circumstances you don't need to know any more than this, as your operating system will have sufficient tooling that everything just works out of the box.
However this stops being true when your computer is connected to multiple networks, as it then needs to manage multiple interfaces.
Inspecting Interfaces
You can run the ip link command to see which interfaces your computer has.
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
link/ether b8:88:e3:e1:3a:00 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
link/ether 60:36:dd:13:98:22 brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether e6:a6:ad:95:9f:27 brd ff:ff:ff:ff:ff:ff
The first column is the numeric ID of the interface, the second its name, and the rest is various configuration.
lo is the local only virtual interface,
this is always available so that networking is always possible,
even on machines that do not have external network ports,
as you may want to use networking for IPC.
Names starting en or eth are generated by Linux for physical ethernet links.
There is usually one of these per ethernet port on your computer.
Names starting wl are wireless interfaces.
Bridge interfaces have many names, usually with br in it.
These allow physical devices to be virtually connected,
which is useful to share a connection on one link to machines connected on another,
or virtual connections to a physical one,
which is what my virbr0 interface does.
veth interfaces are a virtual link pair.
They can take any name, but the ones systemd manages are named ve-*, vb-* and host*.
This is useful for network testing, and network isolation,
so some processes only have access through the virtual link,
and the other end of the link can decide to conditionally forward it on
to its destination or block it.
macvlan or ipvlan virtual interfaces share a physical interface.
This is useful for containers
so you can host another network connected machine
and let it manage its own address while sharing your interfaces.
*.[0-9] - vlan, layered on top of any of the others such that packets are
tagged to only be visible to interfaces with the same vlan number.
A more complete list can be seen by running ip link help.
Configuring Interfaces
Virtual interfaces can be removed with the ip link del $NAME command,
or added with the ip link add $NAME type $TYPE command.
We can safely make a virtual ethernet link pair without confusing your operating system's network configuration too much.
$ sudo ip link add left type veth peer name right
This left and right pair will be used in further examples rather than using one of the interfaces managed by your operating system.
There's no point making these interfaces if we can't prove they work, so let's demonstrate.
First we need to get the ip addresses of those interfaces. All interfaces have an ipv6 link-local address we can use.
$ getip(){ ip -o addr | grep "$1" | tr -s ' ' '\n' | sed -ne '/inet6/{n;p}'; }
$ leftip="$(getip left)"
$ leftip="${leftip%/*}"
$ rightip="$(getip right)"
$ rightip="${rightip%/*}"
Netcat has a few implementations with differing behaviour. IPv6 support is not guaranteed, so we're going to use socat, which is not shipped as standard in many Linux distributions, but has a consistent interface.
We can then start a simple echo service on the left.
$ socat "TCP6-LISTEN:12345,bind=${leftip}%left" - &
The number after the colon specifies to listen on port 12345,
the bind= option specifies to bind the socket
to the link local address by passing that address,
and only on the left interface with the %left suffix.
The - just says to feed the socket input from your terminal
and write output from the socket to your terminal.
We can then connect to this service from the right.
$ socat - "TCP6:[${leftip}%right]:12345,sourceport=${rightip}%right"
This says to connect the input from your terminal,
to a tcp socket,
connecting to the address of the left end of the veth link,
by connecting to the right end,
coming from the left end with the sourceport option.
Now by entering text in one terminal, it will also appear on the other.
Permanent network interface configuration with networkd
If you find this left-right veth link useful,
you can make its configuration permanent,
by running the following command,
to add systemd-networkd netdev configuration.
$ sudo tee /etc/systemd/network/veth-pair.netdev >/dev/null <<EOF
[NetDev]
Name=left
Kind=veth
[Peer]
Name=right
EOF
Further reading
I could not find many resources on this topic, hence why I felt it would be useful to write about it. I found these and would happily take suggestions to extend this list.