pages tagged routeryakkinghttp://yakking.branchable.com/tags/router/yakkingikiwiki2015-09-09T11:00:12ZNetworking - Namespaceshttp://yakking.branchable.com/posts/networking-4-namespaces-and-multi-host-routing/Richard Maw2015-09-09T11:00:12Z2015-09-09T11:00:06Z
<h1>What are network namespaces</h1>
<p>Network Namespaces are a Linux feature,
that allows different processes to have different views of the network.</p>
<p>Aspects of networking that can be isolated between processes include:</p>
<ul>
<li><p>Interfaces</p>
<p>Different processes can connect to addresses on different interfaces.</p></li>
<li><p>Routes</p>
<p>Since processes can see different addresses from different namespaces,
they also need different routes to connect to networks on those interfaces.</p></li>
<li><p>Firewall rules</p>
<p>Firewalls are mostly out of scope for this article,
but since firewall rules are dependant on the source or target interfaces,
you need different firewall rules in different network namespaces.</p></li>
</ul>
<h1>How do you manage network namespaces</h1>
<ul>
<li><p>When a network namespace is created with the <a href="http://man7.org/linux/man-pages/man2/unshare.2.html">unshare(2)</a> or <a href="http://man7.org/linux/man-pages/man2/clone.2.html">clone(2)</a> system calls,
it is bound to the life of the current process,
so if your process exits,
the network namespace is removed.</p>
<p>This is ideal for sandboxing processes,
so that they have restricted access to the network,
as the network namespaces are automatically cleaned up
when the process they were isolating the network for
no longer exists.</p></li>
<li><p>A privileged process can make their network namespace persistent,
which is useful for if the network namespace needs to exist when there are no processes in it.</p>
<p>This is what the <a href="http://man7.org/linux/man-pages/man8/ip.8.html">ip(8)</a> command does when you run <code>ip netns add NAME</code>.</p></li>
<li><p>The <code>ip netns delete NAME</code> command undoes this,
and allows the namespace to be removed
when the last process using it leaves the namespace.</p></li>
<li><p><code>ip netns list</code> shows current namespaces.</p>
<pre><code>$ sudo ip netns add testns
$ ip netns list
testns
$ sudo ip netns delete testns
$ ip netns list
</code></pre></li>
<li><p>A network namespace is of no use if it has no interfaces,
so we can move an existing interface into it with the
<code>ip link set dev DEVICE netns NAME</code> command.</p>
<pre><code>$ ip link show dev usb0
5: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 02:5e:37:61:60:66 brd ff:ff:ff:ff:ff:ff
$ sudo ip link set dev usb0 netns testns
$ ip link show dev usb0
Device "usb0" does not exist.
</code></pre></li>
<li><p>Network namespaces would be of limited use if we can't run a command in them.</p>
<p>We can run a command in a namespace with the <code>ip netns exec NAME COMMAND</code> command.</p>
<pre><code>$ sudo ip netns exec testns ip link show dev usb0
5: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 02:5e:37:61:60:66 brd ff:ff:ff:ff:ff:ff5: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 02:5e:37:61:60:66 brd ff:ff:ff:ff:ff:ff
</code></pre></li>
<li><p>There is no standard daemon to start persistent network namespaces on-boot,
however, the <a href="http://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateNetwork=">PrivateNetwork=</a> option to systemd service files can be used
to start a process in a network namespace on boot.</p></li>
</ul>
<h1>Uses of network namespaces</h1>
<p>Network namespaces can be used to sandbox processes,
so that they can only connect through the interfaces provided.</p>
<p>You could have a machine with two network interfaces,
one which connects out to the internet,
the other connected to your internal network.
You could set up network namespaces
such that you can run the ssh server on your internal network,
and run a web server in a network namespace that has the internet-facing interface,
so if your web server is compromised,
it can't connect to your internal network.</p>
<p><a href="https://www.docker.com/">docker</a> and <a href="http://www.freedesktop.org/software/systemd/man/systemd-nspawn.html">systemd-nspawn</a> combine network namespaces
with other namespaces to provide containers.
<a href="https://www.docker.com/">docker</a> is primarily interested in application containers,
while <a href="http://www.freedesktop.org/software/systemd/man/systemd-nspawn.html">systemd-nspawn</a> is more interested in system containers,
though both technologies can be used for either.</p>
<p>A more novel use of network namespaces is virtual routers,
as implemented in <a href="https://wiki.openstack.org/wiki/Neutron">OpenStack Neutron</a>.</p>
<h2>Network namespaced routing</h2>
<p>We previously mentioned that routes were a property of network namespaces,
that virtual ethernet devices have a pair of ends,
and that we can move a network interface into another network namespace.</p>
<p>By combining these features,
we can construct a series of virtual computers and experiment with routing.</p>
<h2>Two node networking</h2>
<p>To start with we're going to add another virtual computer,
start a simple echo server on it,
and configure the network such that we can connect to it from our host computer.</p>
<p>As we've done in previous articles,
we're going to make the <code>left</code> and <code>right</code> virtual ethernet pair.</p>
<pre><code>$ sudo ip link add left type veth peer name right
</code></pre>
<p>But to demonstrate routing to another machine we need to create one,
so we're making another network namespace,
as we only care about its networking.</p>
<pre><code>$ sudo ip netns add testns
</code></pre>
<p>We're going to move one half of the virtual ethernet device pair
into another namespace,
to simulate a physical link between the two virtual computers.</p>
<pre><code>$ sudo ip link set dev right netns testns
</code></pre>
<p>We're going to say that we connect to interfaces in the <code>testns</code> namespace
by connecting to addresses in the <code>10.248.179.1/24</code> subnet.</p>
<pre><code>$ sudo ip address add 10.248.179.1/24 dev left
$ sudo ip netns exec testns ip address add 10.248.179.2/24 dev right
</code></pre>
<p>We're also going to say that there's another network in there
on the <code>10.248.180.1/24</code> subnet.</p>
<p>Rather than having any more complicated network interfaces,
we assigning it to the loopback interface.</p>
<pre><code>$ sudo ip netns exec testns ip address add 10.248.180.1/24 dev lo
</code></pre>
<p>Now that we've assigned addresses,
we need to bring the interfaces up.</p>
<pre><code>$ sudo ip link set dev left up
$ sudo ip netns exec testns ip link set dev right up
$ sudo ip netns exec testns ip link set dev lo up
</code></pre>
<p>This has created a chain of interfaces linking <code>left</code> → <code>right</code> → <code>lo</code>.
Where <code>left</code> is in your namespace,
and <code>right</code> is in <code>testns</code> with its own private <code>lo</code>.</p>
<p>Now we can demonstrate sending messages through the network namespaces,
by creating a server on the <code>lo</code> interface inside <code>testns</code>.</p>
<p>Start an echo server in the test namespace by running this command in a new
terminal.</p>
<p>It is <em>important</em> to run this in another terminal,
as if you background the process,
netcat may decide it doesn't need to actively output data as it recieves it.</p>
<pre><code>$ sudo ip netns exec testns nc -l 10.248.180.1 12345
</code></pre>
<p>If we were to try to connect to it,
we would not be able to send messages.</p>
<pre><code>$ nc -v 10.248.180.1 12345
nc: connect to 10.248.180.1 port 12345 (tcp) failed: Network is unreachable
</code></pre>
<p>This is because your host namespace does not understand how to reach that address,
since it is not an address in your network namespace,
nor is it an address on any of the subnets in the namespace,
and there are no defined rules for how to reach it.</p>
<p>So let's add a rule!</p>
<pre><code>$ sudo ip route add 10.248.180.0/24 dev left via 10.248.179.2
</code></pre>
<p>This says that you can find the <code>10.248.180.0/24</code> subnet
by sending packets to the <code>10.248.179.2</code> address
through the <code>left</code> interface.</p>
<p>You should now be able to type messages into the following netcat command,
and see the result in your other terminal.</p>
<pre><code>$ nc -v 10.248.180.1 12345
Connection to 10.248.180.1 12345 port [tcp/*] succeeded!
</code></pre>
<h2>3 namespaces</h2>
<p>This works fine,
but in the wider internet you need to connect through long chains of computers
before you reach your final destination,
so we're going to use 3 network namespaces to represent a longer chain.</p>
<p>We are going to have two extra network namespaces, called <code>near</code> and <code>far</code>.</p>
<pre><code>$ sudo ip netns add near
$ sudo ip netns add far
</code></pre>
<p>We are going to create a chain of network interfaces called <code>one</code>, <code>two</code>, <code>three</code> and <code>four</code>.</p>
<pre><code>$ sudo ip link add one type veth peer name two
$ sudo ip link add three type veth peer name four
</code></pre>
<p><code>one</code> will be in our network namespace, connecting to <code>two</code> in the <code>near</code>
namespace, and <code>three</code> will connect to <code>four</code> in the <code>far</code> namespace.</p>
<pre><code>$ sudo ip link set dev two netns near
$ sudo ip link set dev three netns near
$ sudo ip link set dev four netns far
</code></pre>
<p>This produces a chain <code>one</code> → <code>two</code> → <code>three</code> → <code>four</code>.</p>
<p>To speak to these interfaces, we need to assign address ranges, so for our
host to <code>near</code> link we will use <code>10.248.1.0/24</code>,
for our <code>near</code> to <code>far</code> link we will use <code>10.248.2.0/24</code>,
and for our destination address we will use <code>10.248.3.1/24</code>.</p>
<pre><code>$ sudo ip address add 10.248.1.1/24 dev one
$ sudo ip netns exec near ip address add 10.248.1.2/24 dev two
$ sudo ip netns exec near ip address add 10.248.2.1/24 dev three
$ sudo ip netns exec far ip address add 10.248.2.2/24 dev four
$ sudo ip netns exec far ip address add 10.248.3.1/24 dev lo
</code></pre>
<p>We need to bring these interfaces up.</p>
<pre><code>$ sudo ip link set dev one up
$ sudo ip netns exec near ip link set dev two up
$ sudo ip netns exec near ip link set dev three up
$ sudo ip netns exec far ip link set dev four up
$ sudo ip netns exec far ip link set dev lo up
</code></pre>
<p>Now let's this time start our server in the <code>far</code> namespace,
so we can be sure that we can't connect to it directly from our namespace.</p>
<p>As before, start this echo server in a new terminal.</p>
<pre><code>$ sudo ip netns exec far nc -l 10.248.3.1 12345
</code></pre>
<p>Now let's try to connect to it from our namespace.</p>
<pre><code>$ nc -v 10.248.3.1 12345
nc: connect to 10.248.3.1 port 12345 (tcp) failed: Network is unreachable
</code></pre>
<p>If you try to send a message, it won't arrive.
If you inspect the routing table with <code>ip route</code>,
you will see that there is no match.</p>
<p>The relevant routing rules are:</p>
<pre><code>$ ip route
default via 192.168.1.1 dev wlan0 proto static
10.248.1.0/24 dev one proto kernel scope link src 10.248.1.1
</code></pre>
<p>This says that if you're broadcasting from <code>10.248.1.1</code> and going to
anything in <code>10.248.1.0/24</code> then it goes to the <code>one</code> interface. However
this doesn't match because we want to go to the <code>four</code> interface which
has address <code>10.248.3.1</code>.</p>
<p>We can make this route to our <code>near</code> interface by adding a new rule.</p>
<pre><code>$ sudo ip route add 10.248.2.0/24 dev one via 10.248.1.2
</code></pre>
<p>We can prove this hop works by starting a server in the <code>near</code> namespace
(again, in a separate terminal).</p>
<pre><code>$ sudo ip netns exec near nc -l 10.248.2.1 12345
</code></pre>
<p>We can try to talk to it:</p>
<pre><code>$ nc -v 10.248.2.2 12345
</code></pre>
<p>This doesn't yet work, because it's a bidirectional protocol,
so the return route needs to work too.</p>
<pre><code>$ sudo ip netns exec near ip route change 10.248.1.0/24 dev two via 10.248.1.1
</code></pre>
<p>This still doesn't work,
since while there's now routing rules,
they aren't for any addresses which have desinations in the near network namespace,
so by default they get dropped.</p>
<p>To change this we can turn on ip forwarding with:</p>
<pre><code>sudo dd of=/proc/sys/net/ipv4/ip_forward <<<1
</code></pre>
<p>Now we can talk to the <code>near</code> namespace, but we can't yet talk to the <code>far</code> namespace.
We need to add another routing rule for that.</p>
<pre><code>$ sudo ip route add 10.248.3.0/24 dev one via 10.248.1.2
</code></pre>
<p>As before, this means you can route to the <code>near</code> namespace.
However, it doesn't know how to reach the <code>far</code> namespace from there,
so we need to add another routing rule.</p>
<pre><code>$ sudo ip netns exec near ip route add 10.248.3.0/24 dev three via 10.248.2.2
</code></pre>
<p>Now you can reach the <code>far</code> namespace, but it's still not working.
This is because tcp is a bidirectional communication protocol,
and it doesn't know how to send the response to a message from <code>10.248.1.1</code>,
so we need to add another rule.</p>
<pre><code>$ sudo ip netns exec far ip route add 10.248.1.0/24 dev four via 10.248.2.1
</code></pre>
<h2>Conclusion</h2>
<p>I think you will agree that it's far too much faff
to set up this routing just to talk to a machine on another network.</p>
<p>You are likely thinking that there must be a better way,
and you are correct, there's a couple of ways,
but we will cover those in later articles.</p>