util-vserver:SplitSharedNetworks
From Linux-VServer
Contents |
What are we trying to manage here: vservers are pretty good and pretty secure wrt networking. But there comes a time when your iron is to big and can server vservers on security wise multiple networks. Multi-homes is always a pain especially if you have an external firewall. You will make vlans on your big iron, but how will you firewall between one vlan and another on that same iron? Will you duplicate functionality, and have extra firewalling on your big iron, while you have another big iron doing that dedicated already? That's where network namespaces come into the picture. Network namespaces give the ability to have a seperate ip-stack per namespace. That's right: that's including iptables, multiple routing tables and whatever. If we can have multiple ip-stacks, we can give each vserver a seperate ip-stack, right? Well, yeah, but why? An ip-stack usually means another l2 or l3 interfacing, especially if those vservers are supposed to be on the same network. So we group all vservers in that belong security wise into same network also into the same network namespace. This gives the benefit that we don't have to fool around with special virtual bridges, virtual ethernet devices or whatever.
So this page will describe how to setup multiples of (a group of vservers which share the same ipstack on a single vlan) on a single system with only using vlans as network virtualization.
Namespace ramblings
Playing with the latest and greatest ip route I discovered that ip supports the creation and naming of network namespaces, all be it with a trick. The current ip route utility manages network namespaces by creating a new namespace, and then bind-mounting /proc/self/ns/net (a file!) to /var/run/netns/<a file named after the namespace>.
Network namespaces always had a big problem of not being able to reach them, unless you know a pid in that namespace. There is another technique, and that is to open a file which is somehow located in that namespace. Enough of that.
Vserver to ip
How can we use that? First we must make sure that we are compatible: we can make vserver create the network namespace and let the ip utility know about that:
mkdir -p /var/run/netns touch /var/run/netns/$VSERVER vspace -e $VSERVER --net mount -o bind /proc/self/ns/net /var/run/netns/$VSERVER
From then on you can use the following incantations:
ip netns exec $VSERVER cmds vspace -e $VSERVER --net cmds
Interface to vserver
To put a network device into that namespace we do something like this:
ip link set dev $IFACE netns $VSERVER
We can check that with:
ip netns exec $VSERVER ip a ls vspace -e $VSERVER --net ip a ls
Useful interface creationism
What's a network namespace without an interface? Here are some create statements:
# Create an interface on a a "real" device # A macvlan in bridge mode is as if we have a second network card # in the same network. It already knows all mac-addresses, so the "bridging" code is lightweight ip link add link $BASEDEVICE name $VSERVERDEVICE type macvlan mode bridge # Create a virtual network tunnel # This is basically a virtual cross cable type of interface # You can use a veth like any ordinary device like # plugging it into a bridge device ip link add name $DEVICE_A type veth peer name $DEVICE_B # Create a normal vlan interface on a device. # Great when giving vservers a seperate vlan ip link add link $BASEDEVICE name $VSERVERDEVICE type vlan id $VLANID
Remember though: all interfaces need valid mac-addresses, especially when going to the outside. I use this complex script to generate a pair of static mac-addresses for a veth:
stringasmac() {
echo "$1"|sed 's/\(..\)\(..\)\(..\)\(..\)\(..\)\(..\)/\1:\2:\3:\4:\5:\6/'
}
read CONTEXTID < /etc/vservers/${VSERVER}/context
MACSTRING=$(printf "%04x" $VID)$(printf "%04x" $CONTEXTID)
MACADDRC=$(stringasmac 02${MACSTRING}00)
MACADDRH=$(stringasmac 02${MACSTRING}01)
Valid addresses are addresses that have bit0 unset and bit1 set in the first byte. It means a normal (bit0: 1 == multicast) and locally administrated (bit1: 1==Not official) address. You set them like:
ip link set address ${MACADDRC} dev ${CLIENTDEV}
ip to vserver
Turned around, could we create a namespace and use it in vserver? We can then prepare interfaces in the namespace before starting the vserver. If it works, it should go like this:
ip netns add $VSERVER ip netns exec $VSERVER vserver $VSERVER start
interface management
Any vserver with a seperate network namespace should have it's lo brought up (in *that* namespace). Any link configuration should be done within that namespace. Util-vserver needs some patches for that.
/proc/net/
Initially the contents of /proc/net/ is not visible in the vserver. vprocunhide unhides only /proc/net from the host network namespace. So to be able to see /proc/net/* inside the vserver you will have to run vprocunhide in the created network namespace again. Something like that could be useful:
ip netns exec $VSERVER /usr/lib/util-vserver/vprocunhide
Applied namespace ramblings
The Script
This script should be symlinked from /etc/vservers/<name>/scripts/(initialiaze,prepre-start...).d .
function createMacVidId() {
(
printf "02"
printf "%04x" $1 # I usually use VID
printf "%04x" $S_CONTEXT
printf "%02x" $2 # IFACE number f.i.
)|
sed 's/\(..\)\(..\)\(..\)\(..\)\(..\)\(..\)/\1:\2:\3:\4:\5:\6/'
}
function usesipnetns() {
local netspace
if [ -s "$VSERVER_DIR/spaces/net" ]
then
read netspace < "$VSERVER_DIR/spaces/net"
[ "$netspace" = "$VSERVER_NAME" ]
return $?
fi
return 1
}
case "${1}" in
initialize)
if usesipnetns
then
_HIP="$_IP"
_IP="$_HIP netns exec ${2} $_HIP"
$_HIP netns add ${2}
$_IP link set dev lo up
fi
;;
prepre-start)
if usesipnetns
then
$_HIP link add link eth0 name fw-vlan2 type vlan id 2
$_HIP link set dev fw-vlan2 address $(createMacVidId 2 0)
$_HIP link set dev fw-vlan2 netns ${2}
$_HIP netns exec ${2} $_VPROCUNHIDE
fi
;;
# By now all options are generated, time to override them
pre-start)
# NICE_CMD=(/sbin/ip netns exec ${2} "${NICE_CMD[@]}")
if usesipnetns
then
local index
local NEWVSPACE_CMDS
declare -a NEWVSPACE_CMDS
index=0;
while [ $index -lt ${#VSPACE_SHARED_CMD[@]} ]
do
if
[ "${VSPACE_SHARED_CMD[$((${index}+0))]}" = "$_VSPACE" ] &&
[ "${VSPACE_SHARED_CMD[$((${index}+1))]}" = "--enter" ] &&
[ "${VSPACE_SHARED_CMD[$((${index}+2))]}" = "${2}" ] &&
[ "${VSPACE_SHARED_CMD[$((${index}+3))]}" = "--net" ]
then
NEWSPACE_CMDS=("${NEWSPACE_CMDS[@]}" $_HIP netns exec "$2")
index=$(($index+5))
else
NEWSPACE_CMDS=("${NEWSPACE_CMDS[@]}" "${VSPACE_SHARED_CMD[$index]}")
index=$(($index+1))
fi
done
VSPACE_SHARED_CMD=("${NEWSPACE_CMDS[@]}")
unset NEWVSPACE_CMDS
unset index
fi
;;
post-start)
;;
pre-stop)
if usesipnetns
then
_HIP=_IP
_IP="$_HIP netns exec ${2} $_HIP"
$_HIP netns add ${2}
$_IP link set dev lo up
fi
;;
post-stop)
$_IP link del fw-vlan2
$_HIP netns delete ${2}
;;
esac
Explanation
The script uses a configuration fault (e.g: <vserver>/spaces/net contains it's own name) to indicate that the script should create the network namespace. Actually util-vserver will think that it is using a shared network namespace, and we replace that incantation of vspace with our own. You are expected to create and give the interfaces in the prepre-start section. Standard util-vserver functionality can configure those interfaces by prefixing $_IP with the vspace or ip netns prefix. This way, the interfaces are UP *and* RUNNING before the vserver itself starts, hence making it ideal for f.i. DHCP servers on dedicated vlans.
Configuration issues: for the script to work you now have to have the right /run/netns/vserver-name in the /etc/vserver-name/namespace-cleanup-skip file.
future
I will add functionality to allow shared network namespaces. It's not that hard.
