Latest revision as of 16:25, 3 January 2019

[edit] Split shared networks with util-vserver:

What are we trying to manage here: vservers are pretty good and pretty secure wrt networking. But there comes a time when your iron is to big and can server vservers on security wise multiple networks. Multi-homes is always a pain especially if you have an external firewall. You will make vlans on your big iron, but how will you firewall between one vlan and another on that same iron? Will you duplicate functionality, and have extra firewalling on your big iron, while you have another big iron doing that dedicated already? That's where network namespaces come into the picture. Network namespaces give the ability to have a seperate ip-stack per namespace. That's right: that's including iptables, multiple routing tables and whatever. If we can have multiple ip-stacks, we can give each vserver a seperate ip-stack, right? Well, yeah, but why? An ip-stack usually means another l2 or l3 interfacing, especially if those vservers are supposed to be on the same network. So we group all vservers in that belong security wise into same network also into the same network namespace. This gives the benefit that we don't have to fool around with special virtual bridges, virtual ethernet devices or whatever.

So this page will describe how to setup multiples of (a group of vservers which share the same ipstack on a single vlan) on a single system with only using vlans as network virtualization.

[edit] Namespace ramblings

Playing with the latest and greatest ip route I discovered that ip supports the creation and naming of network namespaces, albeit with a trick. The current ip route utility manages network namespaces by creating a new namespace, and then bind-mounting /proc/self/ns/net (a file!) to /var/run/netns/<a file named after the namespace>.

Network namespaces always had a big problem of not being able to reach them, unless you know a pid in that namespace. There is another technique, and that is to open a file which is somehow located in that namespace. Enough of that.

[edit] Vserver to ip

How can we use that? First we must make sure that we are compatible: we can make vserver create the network namespace and let the ip utility know about that:

mkdir -p /var/run/netns
touch /var/run/netns/$VSERVER
vspace -e $VSERVER --net mount -o bind /proc/self/ns/net /var/run/netns/$VSERVER

From then on you can use the following incantations:

ip netns exec $VSERVER cmds
vspace -e $VSERVER --net cmds

[edit] Interface to vserver

To put a network device into that namespace we do something like this:

ip link set dev $IFACE netns $VSERVER

We can check that with:

ip netns exec $VSERVER ip a ls
vspace -e $VSERVER --net ip a ls

[edit] Useful interface creationism

What's a network namespace without an interface? Here are some create statements:

# Create an interface on a a "real" device
# A macvlan in bridge mode is as if we have a second network card
# in the same network. It already knows all mac-addresses, so the "bridging" code is lightweight
ip link add link $BASEDEVICE name $VSERVERDEVICE type macvlan mode bridge

# Create a virtual network tunnel
# This is basically a virtual cross cable type of interface
# You can use a veth like any ordinary device like
# plugging it into a bridge device
ip link add name $DEVICE_A type veth peer name $DEVICE_B

# Create a normal vlan interface on a device.
# Great when giving vservers a seperate vlan
ip link add link $BASEDEVICE name $VSERVERDEVICE type vlan id $VLANID

Remember though: all interfaces need valid mac-addresses, especially when going to the outside. I use this complex script to generate a pair of static mac-addresses for a veth:

stringasmac() {
        echo "$1"|sed 's/\(..\)\(..\)\(..\)\(..\)\(..\)\(..\)/\1:\2:\3:\4:\5:\6/'
}
read CONTEXTID < /etc/vservers/${VSERVER}/context
MACSTRING=$(printf "%04x" $VID)$(printf "%04x" $CONTEXTID)
MACADDRC=$(stringasmac 02${MACSTRING}00)
MACADDRH=$(stringasmac 02${MACSTRING}01)

Valid addresses are addresses that have bit0 unset and bit1 set in the first byte. It means a normal (bit0: 1 == multicast) and locally administrated (bit1: 1==Not official) address. You set them like:

ip link set address ${MACADDRC} dev ${CLIENTDEV}

[edit] ip to vserver

Turned around, could we create a namespace and use it in vserver? We can then prepare interfaces in the namespace before starting the vserver. If it works, it should go like this:

ip netns add $VSERVER
ip netns exec $VSERVER vserver $VSERVER start

[edit] interface management

Any vserver with a seperate network namespace should have it's lo brought up (in *that* namespace). Any link configuration should be done within that namespace. Util-vserver needs some patches for that.

[edit] /proc/net/

Initially the contents of /proc/net/ is not visible in the vserver. vprocunhide unhides only /proc/net from the host network namespace. So to be able to see /proc/net/* inside the vserver you will have to run vprocunhide in the created network namespace again. Something like that could be useful:

ip netns exec $VSERVER /usr/lib/util-vserver/vprocunhide

[edit] Applied namespace ramblings

[edit] The Script

This script should be symlinked from /etc/vservers/<name>/scripts/(initialiaze,prepre-start...).d .

function createMacVidId() {
        (
                printf "02"
                printf "%04x" $1 # I usually use VID
                printf "%04x" $S_CONTEXT
                printf "%02x" $2 # IFACE number f.i.
        )|
        sed 's/\(..\)\(..\)\(..\)\(..\)\(..\)\(..\)/\1:\2:\3:\4:\5:\6/'
}
function usesipnetns() {
        local netspace
        if [ -s "$VSERVER_DIR/spaces/net" ]
        then
                read netspace < "$VSERVER_DIR/spaces/net"
                [ "$netspace" = "$VSERVER_NAME" ]
                return $?
        fi
        return 1
}
case "${1}" in
        initialize)
                if usesipnetns
                then
                        _HIP="$_IP"
                        _IP="$_HIP netns exec ${2} $_HIP"
                        $_HIP netns add ${2}
                        $_IP link set dev lo up
                fi
                ;;
        prepre-start)
                if usesipnetns
                then
                        $_HIP link add link eth0 name fw-vlan2 type vlan id 2
                        $_HIP link set dev fw-vlan2 address $(createMacVidId 2 0)
                        $_HIP link set dev fw-vlan2 netns ${2}
                        $_HIP netns exec ${2} $_VPROCUNHIDE
                fi
                ;;
        # By now all options are generated, time to override them
        pre-start)
                # NICE_CMD=(/sbin/ip netns exec ${2} "${NICE_CMD[@]}")
                if usesipnetns
                then
                        local index
                        local NEWVSPACE_CMDS
                        declare -a NEWVSPACE_CMDS
                        index=0;

                        while [ $index -lt ${#VSPACE_SHARED_CMD[@]} ]
                        do
                                if
                                        [ "${VSPACE_SHARED_CMD[$((${index}+0))]}" = "$_VSPACE" ] &&
                                        [ "${VSPACE_SHARED_CMD[$((${index}+1))]}" = "--enter" ] &&
                                        [ "${VSPACE_SHARED_CMD[$((${index}+2))]}" = "${2}" ] &&
                                        [ "${VSPACE_SHARED_CMD[$((${index}+3))]}" = "--net" ]
                                then
                                        NEWSPACE_CMDS=("${NEWSPACE_CMDS[@]}" $_HIP netns exec "$2")
                                        index=$(($index+5))
                                else
                                        NEWSPACE_CMDS=("${NEWSPACE_CMDS[@]}" "${VSPACE_SHARED_CMD[$index]}")
                                        index=$(($index+1))
                                fi
                        done
                        VSPACE_SHARED_CMD=("${NEWSPACE_CMDS[@]}")
                        unset NEWVSPACE_CMDS
                        unset index
                fi
                ;;
        post-start)
                ;;
        pre-stop)
                if usesipnetns
                then
                        _HIP=_IP
                        _IP="$_HIP netns exec ${2} $_HIP"
                        $_HIP netns add ${2}
                        $_IP link set dev lo up
                fi
                ;;
        post-stop)
                $_IP link del fw-vlan2
                $_HIP netns delete ${2}
                ;;
esac

[edit] Explanation

The script uses a configuration fault (e.g: <vserver>/spaces/net contains it's own name) to indicate that the script should create the network namespace. Actually util-vserver will think that it is using a shared network namespace, and we replace that incantation of vspace with our own. You are expected to create and give the interfaces in the prepre-start section. Standard util-vserver functionality can configure those interfaces by prefixing $_IP with the vspace or ip netns prefix. This way, the interfaces are UP *and* RUNNING before the vserver itself starts, hence making it ideal for f.i. DHCP servers on dedicated vlans.

Configuration issues: for the script to work you now have to have the right /run/netns/vserver-name in the /etc/vserver-name/namespace-cleanup-skip file.

[edit] future

I will add functionality to allow shared network namespaces. It's not that hard.