High Availability on Linux: The service IP address, part 2

Víctor Román Archidona
8 min readDec 28, 2021
Photo by Erik Mclean on Unsplash

In this second part of the article the service IP address is configured and the use of virtual network interface “dummy0” is explained.

Configuring the service IP address

As already mentioned in the first part, using “keepalived” -which implements the VRRP protocol-, we will assign an IP address to a node as starting point; and if it fails, that IP address will be automatically assigned to the slave node.

The configuration file for “keepalived” is “/etc/keepalived/keepalived.conf”. Each server will have its own file, very similar to its pair.

For the first node we execute the following:

superuser@server-1 # cat <<EOF > /etc/keepalived/keepalived.conf
# Some global definitions, like this node identifier
global_defs {
router_id server-1
}
vrrp_instance service-100 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
# User/password to link
authentication {
auth_type PASS
auth_pass 4h[-/BhJbj$vU6>=^]hS.cYt)>8+WaA+
}
# Service IP address(es) we will have
virtual_ipaddress {
192.0.2.100/24 brd 192.0.2.255 dev dummy0
}
# VRRP unicast traffic instead multicast over the internal NIC.
unicast_peer {
192.0.2.6
}
# Checks eth0/dummy0 interface status. If one of them isn't available
# it moves the service IP address(es).
track_interface {
eth0
dummy0
}
}
EOF

For the second node, we execute:

superuser@server-2 # cat <<EOF > /etc/keepalived/keepalived.conf
# Some global definitions, like this node identifier
global_defs {
router_id server2-2
}
vrrp_instance service-100 {
state BACKUP
interface eth0
virtual_router_id 100
priority 50
advert_int 1
# User/password to link
authentication {
auth_type PASS
auth_pass 4h[-/BhJbj$vU6>=^]hS.cYt)>8+WaA+
}
# Service IP address(es) we will have
virtual_ipaddress {
192.0.2.100/24 brd 192.0.2.255 dev dummy0
}
# VRRP unicast traffic instead multicast over the internal NIC
unicast_peer {
192.0.2.5
}
# Checks eth0/dummy0 interface status. If one of them isn't available
# it moves the service IP address(es)
track_interface {
eth0
dummy0
}
}
EOF

The highlighted lines are those that differ depending on the node.

Consider this file generically:

global_defs {
router_id server-X
}
vrrp_instance service-X {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 4h[-/BhJbj$vU6>=^]hS.cYt)>8+WaA+
}
virtual_ipaddress {
192.0.2.100/24 brd 192.0.2.255 dev dummy0
}
unicast_peer {
192.0.2.5
}
track_interface {
eth0
dummy0
}
}

Lines 1 to 3 are a global definitions block. In this cases we have only used “router_id” to identify the node when it exchanges VRRP information. Other functionality not shown here is notify by mail when a status change occurs.

Lines 5 to 29 defines an “VRRP instance”. An instance is a node group which communicates together exchanging status information following rules we specify.

Inside the VRRP instance we define the initial status (line 6) each server has: MASTER, which will have the service IP address assigned by default; or BACKUP, which will have it if the MASTER node doesn’t answer or notifies that some rule is not being followed.

When a node in BACKUP status can’t communicate with the MASTER node, or the MASTER node informs the rules of the VRRP instance are not followed, the BACKUP node will take the MASTER status and the MASTER node will take the BACKUP states,

The information will be exchanged with a symmetric cipher over “eth0” network interface (line 7) to avoid that a malicious server alters the exchange of status information (lines 12 to 15).

VRRP is a protocol who exchanges status information using multicast network packets. As we only have two nodes, we will setup a direct communication between them -unicast- (lines 21 to 23) and we don’t use the network multicast traffic. Another reason to do this is the possibility that switch where nodes are connected has some multicast traffic filter.

In lines 17 to 19 we indicate that this VRRP instance must assign to “dummy0” network interface the service IP address “192.0.2.100/24”. This is where the IP (or IPs) the node will have assigned if it’s in MASTER status are defined.

In lines 25 to 28 we define a VRRP rule to this instance: Check the “eth0” and “dummy0” network interfaces. If any of these interfaces aren’t available for whatever reason, then a notification is emitted and the BACKUP node assigns the service IP addresses to himself.

Finally we’ll restart the “keepalived” service on each node:

superuser@server-X # systemctl restart keepalived

We may verify the correct “keepalived” start with the configuration shown above using the “journalctl” command:

For the first node we’ll see something like this:

root@server-1:~# journalctl -u keepalived
[...]
ago 30 13:09:04 server-1 VRRP_Instance(service-100) Transition to MASTER STATE
ago 30 13:09:04 server-1 Keepalived_vrrp[562]: VRRP_Instance(service-100) Received lower prio advert, forcing new election
ago 30 13:09:05 server-1 Keepalived_vrrp[562]: VRRP_Instance(service-100) Entering MASTER STATE
[...]

We can see in the last two lines how a new master election is forced and how this node, “server-1”, is chosen as that (MASTER STATE).

We can also see that network interface “dummy0” has the IP address “192.0.2.100” assigned as it’s the MASTER:

root@server-1:~# ip -4 addr show dev dummy0
3: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 6a:0c:c1:40:8f:26 brd ff:ff:ff:ff:ff:ff
inet 192.0.2.100/24 brd 192.0.2.255 scope global dummy0
valid_lft forever preferred_lft forever

On the second node we see:

root@server-2:~# journalctl -u keepalived
[...]
ago 30 13:09:05 server-2 Keepalived_vrrp[563]: VRRP_Instance(service-100) Transition to MASTER STATE
ago 30 13:09:05 server-2 Keepalived_vrrp[563]: VRRP_Instance(service-100) Received higher prio advert
ago 30 13:09:05 server-2 Keepalived_vrrp[563]: VRRP_Instance(service-100) Entering BACKUP STATE
[...]

In the last two lines we see this node gets an “higher priority” advice (from node “server-1”) and then it takes the BACKUP state.

On the second node there won’t be any IP address assigned to the “dummy0” network interface as being that node as backup:

root@server-2:~# ip -4 addr show dev dummy0

Verifying functionality

With “keepalived” running and nodes defined as master and slave, its time to run the necessary tests to verify that the behavior is the expected.

We’ll use the “ping” command from some system on the same 192.0.2.100/24 subnet to check the service IP address availability. It can be done from the backup node, but is preferable to use an external system other than the nodes that are being configured:

root@client-1:~# ping -c 5 192.0.2.100
PING 192.0.2.100 (192.0.2.100) 56(84) bytes of data.
64 bytes from 192.0.2.100: icmp_seq=1 ttl=64 time=1.79 ms
64 bytes from 192.0.2.100: icmp_seq=2 ttl=64 time=0.401 ms
64 bytes from 192.0.2.100: icmp_seq=3 ttl=64 time=0.377 ms
64 bytes from 192.0.2.100: icmp_seq=4 ttl=64 time=0.359 ms
64 bytes from 192.0.2.100: icmp_seq=5 ttl=64 time=0.451 ms
--- 192.0.2.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 0.359/0.676/1.795/0.560 ms

The first real test we will do is verify what happens when the master node becomes unavailable due a reboot or power failure; and the second one consists on network failure simulation.

In case of active node reboot

Connected to both nodes, on the master we’ll reboot the server and on the second we check what happens:

root@server-1:~# systemctl rebootroot@server-2:~# journalctl -fu keepalived
[...]
ago 30 13:40:34 server-2 Keepalived_vrrp[570]: VRRP_Instance(service-100) Transition to MASTER STATE
ago 30 13:40:35 server-2 Keepalived_vrrp[570]: VRRP_Instance(service-100) Entering MASTER STATE

The backup node becomes the master and the service IP address is assigned to it:

root@server-2:~# ip -4 addr show dev dummy0
3: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
inet 192.0.2.100/24 brd 192.0.2.255 scope global dummy0
valid_lft forever preferred_lft forever

The service IP address continues accesible from other systems on the network. The following output shows what perception a client had during the process since the first node failed until the secondary took the control and started giving service:

root@client-1:~# ping -D 192.0.2.100
PING 192.0.2.100 (192.0.2.100) 56(84) bytes of data.
[1472557717.182408] 64 bytes from 192.0.2.100: icmp_seq=1 ttl=64 time=0.341 ms
[1472557718.181466] 64 bytes from 192.0.2.100: icmp_seq=2 ttl=64 time=0.419 ms
### REBOOT LAUNCHED AT THIS POINT ###[1472557743.375291] 64 bytes from 192.0.2.100: icmp_seq=27 ttl=64 time=0.779 ms
[1472557744.373882] 64 bytes from 192.0.2.100: icmp_seq=28 ttl=64 time=0.366 ms
[1472557745.372843] 64 bytes from 192.0.2.100: icmp_seq=29 ttl=64 time=0.354 ms
[1472557746.372258] 64 bytes from 192.0.2.100: icmp_seq=30 ttl=64 time=0.411 ms
[1472557747.373111] 64 bytes from 192.0.2.100: icmp_seq=31 ttl=64 time=0.398 ms
[1472557748.372796] 64 bytes from 192.0.2.100: icmp_seq=32 ttl=64 time=0.398 ms
[1472557749.373202] 64 bytes from 192.0.2.100: icmp_seq=33 ttl=64 time=0.371 ms
[1472557750.373515] 64 bytes from 192.0.2.100: icmp_seq=34 ttl=64 time=0.357 ms
[1472557751.373162] 64 bytes from 192.0.2.100: icmp_seq=35 ttl=64 time=0.372 ms
^C
--- 192.0.2.100 ping statistics ---
35 packets transmitted, 11 received, 68% packet loss, time 34190ms
rtt min/avg/max/mdev = 0.341/0.415/0.779/0.117 ms

As it can be observed, the virtual IP address (the service IP address) was not available over a 25 seconds period.

In case of physical network failure

Is possible to simulate the loss of a network interface with the following command:

root@server-1:~# ip link set eth0 down

We can check the slave node detects such failure on the master and assigns the service IP address to its “dummy0” interface:

root@server-2:~# journalctl -fu keepalived
[...]
sep 12 10:17:12 server-2 Keepalived_vrrp[578]: VRRP_Instance(service-100) Transition to MASTER STATE
sep 12 10:17:13 server-2 Keepalived_vrrp[578]: VRRP_Instance(service-100) Entering MASTER STATE

As in the previous case, the service IP address is reassigned:

root@server-2:~# ip -4 addr show dev dummy0
3: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
inet 192.0.2.100/24 brd 192.0.2.255 scope global dummy0
valid_lft forever preferred_lft forever

The current service status is like the following image:

If disabling the network interface on the second node by:

root@server-2:~# ip link set eth0 down

Having no enabled interface (either the first node or the second), the service would no longer be accessible.

The “dummy0” virtual interface usage

Instead using the physical network adapter “eth0” En lugar de utilizar la interfaz de red física “eth0”, we decided to use “dummy0”, a virtual interface.

An advantage presented by the use of virtual interface is the ability to disable it without losing connection to the computer at the time of administration.

We can also disable the network interface “dummy0” as appropriate for administrative tasks, such as updating server software. Once disabled, traffic will flow automatically to the slave node while these tasks are completed without affecting users.

Final remarks

For simplicity along these two articles related to the service IP address, we used a single physical network interface for both server management and providing the service.

It is advisable to use multiple physical network interfaces, each for a task. Ideally the physical network interface “eth0” will be used to provide the service, while another network interface “eth1” will be used to administer the server and where “keepalived” exchanges the status of both nodes.

Similarly, for simplicity, we used only one connection for each task. Environments where high availability is essential must have double network links using “bonding” or “teaming” with separate network cards; in the future I will post about it.

We have used a virtual network interface “dummy0” for flexibility, so we can disable it and continue operating on the server for maintenance. It also allows us to force the slave node take control disabling that interface in the master.

--

--

Víctor Román Archidona

CEO, Co-Founder and SRE at LoadFront. Husband. Father. Remote work advocate. And sometimes I write about how to do things with computers.