Post by Niklaas Baudet von GersdorffThen, I could use devd to assign the public failover IP (that I actually
wanted to share with CARP on vtnet0) to the public interface vtnet0.
CARP(4) provides an example on how to use carp status change events for
--------------------8<--------------------
Processing of carp status change events can be set up by using the fol-
notify 0 {
match "system" "CARP";
match "type" "(MASTER|BACKUP)";
action "/root/carpcontrol.sh $subsystem $type";
};
-------------------->8--------------------
Depending von $type, carpcontrol.sh could either
ifconfig vtnet0 <public-ip> alias
or
ifconfig vtnet0 <public-ip> -alias
I am not sure whether that works but I'll post my solution if I find
one.
Obviously, the following set-up is very limited to a prober
configuration with CARP on a public network. However, it seems as if
supporting CARP in virtualised environments isn't possible because the
multicast packages would collide with those of other customers in the
same network. So the only solution is to use CARP within a private
network for some pseudo-CARP IP. Then, one can use the devd events that
are triggered if connection via the private network is lost, to assign
and remove the floating public IP that is used for failover.
So, let's see how this can be done. I have three hosts A, B, and C.
I could not manage to get CARP advertising over a virtual private
network of my provider that connects these hosts (CARP used the wrong
MAC addresses so the packets weren't translated correctly). But what
I managed to get going is CARP advertising over my tinc VPN tap0.
So if any host comes online and connects to the other hosts on the VPN,
tinc will assign a CARP vhid on tap0. This is
/usr/local/etc/tinc/host-up on host A:
--------------------8<--------------------
ifconfig $INTERFACE vhid 1 advbase 01 advskew 000 pass <some-pasword> 10.99.99.99 alias
-------------------->8--------------------
On host B and C the file looks exactly the same except that advbase is
five seconds longer each (06 and 11 respectively). (While testing the
set-up I realised that latency is too high, so that advskew wouldn't
suffice.)
I use exactly the same devd.conf as quoted above. I put it under
/usr/local/etc/devd.
Further, I did the following:
sysctl net.inet.carp.preempt=1
which should be added to /etc/sysctl.conf.
/root/carpcontrol.sh is on every host and looks like the following:
--------------------8<--------------------
1 #!/bin/sh
2
3 subsystem=$1
4 type=$2
5 vhid="1"
6 c_if="tap0" # CARP interface
7
8 e_if="vtnet0" # external interface
9 IPv4="<floating-IPv4>"
10 IPv4_net="/24"
11 IPv6="<floating-IPv6>"
12 IPv6_net="/64"
13
14 log_tag="carpcontrol.sh"
15
16 remove_floating_ip() {
17 if ifconfig $e_if | grep $IPv4 >/dev/null
18 then
19 ifconfig $e_if $IPv4$IPv4_net -alias && logger -t $log_tag "Removed $IPv4 on $e_if."
20 fi
21 if ifconfig $e_if | grep $IPv6 >/dev/null
22 then
23 ifconfig $e_if inet6 $IPv6$IPv6_net -alias && logger -t $log_tag "Removed $IPv6 on $e_if."
24 fi
25 }
26
27 add_floating_ip() {
28 ifconfig $e_if $IPv4$IPv4_net alias && logger -t $log_tag "Assigned $IPv4 on $e_if."
29 ifconfig $e_if inet6 $IPv6$IPv6_net alias && logger -t $log_tag "Assigned $IPv6 on $e_if."
30 }
31
32 if [ "$subsystem" = "$vhid@$c_if" ]
33 then
34 case $type in
35 "MASTER")
36 add_floating_ip
37 ;;
38 "BACKUP")
39 remove_floating_ip
40 ;;
41 esac
42 elif [ "$subsystem" = "tinc-down" ]
43 then
44 remove_floating_ip
45 fi
-------------------->8--------------------
Probably that's not the best way to do it and probably it could be
extended but my sh-skills are quite bad. This works though.
The script should be self-explanatory; the only ting that might be
unclear are lines 42-45. This is triggered by /usr/local/etc/tinc-down:
--------------------8<--------------------
/root/carpcontrol.sh tinc-down
-------------------->8--------------------
So, in case I stop tincd on purpose, the floating IP will be removed and
becomes free for the other machines. The second host will take over the
IP after some seconds.
Obviously, there are quite some drawbacks to this but it serves my
purpose. Since the machines are virtualised I don't expect them to down
by some hardware failure. But I do need failover in case I want to
upgrade one machines and need to reboot it.
In case I need to do some maintenance, I cut the machine in question
from the private network. For some seconds services won't be reachable
but after a short while host B will take over. I can upgrade host A and
restart it eventually. When host A runs again, it will connect to the
tinc VPN and while advertise CARP on that network. So, host A will
becomes MASTER again, host B will switch to BACKUP, and C will remain
BACKUP.
Nonetheless, this is not a perfect solution. "High-availability" (these
quotes are on purpose) as achieved this way, depends on the connectivity
between the tinc nodes. E.g., if tinc malfunctions and doesn't shutdown
properly, the floating IP will stay assigned to the public interface,
the other client will loose connectivity to the node, and an additional
MASTER will rise. In addition, I had problems using a faster advbase
because of latency, I guess, thus there will be some downtime if any
MASTER goes down. In this case, the problem was that two hosts could not
decide who should be MASTER because there were some connectivity
problems between them on the VPN.
The crux is that CARP, as implemented in FreeBSD, cannot advertise on
unicast. If it could, I would have been able to assign it to the private
virtual interface that connects the virtual machines. However, CARP, as
implemented in OpenBSD, can do so using a feature called "carppeer".
There have been requests whether/when that will be brought to FreeBSD
[1-3] but the questions remained without responses.
1: https://lists.freebsd.org/pipermail/freebsd-net/2013-November/037106.html
2: https://lists.freebsd.org/pipermail/freebsd-net/2011-February/027999.html
3: https://lists.freebsd.org/pipermail/freebsd-pf/2009-December/005486.html
So, where can I post feature requests? :-) Without a way for unicast
CARP packets, FreeBSD is of much less value for virtualised environments
where multicast packets are an issue because they are blocked by
providers. So, setting up a failover set-up becomes a mess (such as the
one above).
Post by Niklaas Baudet von GersdorffFor that to work I must bind processes to non-local IP addresses. How do
I do that?
I found this
https://lists.freebsd.org/pipermail/freebsd-hackers/2011-January/034033.html
with some recommendations to do so with ipfw. Can I do something similar
with pf? Or is there even another solution for binding to non-local
addresses?
No longer necessary. I use a pf rule that redirect any traffic arriving
on a specific port (this could also be any traffic arriving on the
pseudo-CARP IP) to the jail where the load-balancer is listening.
Anyway, I'm happy for any comments and further suggestions.
Niklaas