Clustering and Distributed Architectures

The basic idea behind redundant systems and services is to provide at least a second resource that can either take over the duty in a hot-standby fashion or, even better, is a member of a server/service farm that constantly contributes via load-balancing schemes. One step further would distribute such architecture geographically, and that would pretty much define the boundaries (not limits) of such an architecture.

One big advantage of a loosely coupled load-sharing cluster approach is that it is independent of the failure of one component. This approach affects overall cluster performance only to a certain extent, depending on the capacity planning of the cluster architect. One major reason for such deployments is the possibility to schedule maintenance for a cluster constituent without affecting normal operation in high-availability networks.

Historically, clustering is a domain of the (Open)VMS world. OpenVMS is a product of HP after HP bought Compaq and Compaq bought Digital. If you are interested in OpenVMS, look at http://www.openvms.org and Hewlett Packard's web page.

The apex of clustering is a tight integration (virtual supercomputer) of an almost arbitrary number of multiprocessor systems to form a modern cluster under the control of redundant cluster controllers with distributed storage, memory, sockets, and I/O. Examples of such an approach are high-performance clusters (HPC). In contrast, GRID architectures (computing grids) for scientific number crunching such as the well-known SET@Home project (http://setiathome.ssl.berkeley.edu) have evolved, where every home or office workstation contributes to a calculation when idle. A thorough discussion of cluster concepts, such as MOSIX and Beowulf, goes beyond the scope of this book. Nevertheless, the following sections cover three prominent examples of Linux HA approaches.

Linux Virtual Server Project (LVSP)
The LVSP is a scalable and transparent load-balancing architecture based on the Linux operating system. It requires a kernel patch and a user-space administration tool, ipvsadm. The constituent servers of a load-balancing group can be dispersed geographically and still can be Layer 4 controlled by the LVS architecture.

Detection of node or daemon failures and the appropriate reconfiguring of the system lead to high availability. IP-level load balancing offers performance and transparency advantages over application-level solutions such as caches and proxies. The virtual server uses three different load-balancing techniques and a repertoire of scheduling algorithms. The load-balancing techniques are as follows:

Virtual server via NAT— Based on Port Address Translation and port forwarding.

Virtual server via IP tunneling— The virtual server sends requests to physical servers via IP-IP tunnels.

Virtual server via direct routing— The return packets to client requests are routed directly from the real servers without involving the load balancer.

The scheduling algorithms are as follows

Least connection

Weighted least-connection

Round-robin (allocates connections evenly to all real servers)

Weighted round-robin

Locality-based least connection

Locality-based least connection with replication

Destination hashing

Source hashing

For further configuration details and ancillary tools, consult http://www.linuxvirtualserver.org/.

Connection Integrity Issues
Modern director approaches such as LVS support connection-integrity maintenance. Connection integrity refers to the capability of the director to maintain a connection in a way that it never changes the real server after the first scheduling decision has been made based on the initial packets. This is necessary for certain protocols such as Secure Sockets Layer (SSL) and File Transfer Protocol (FTP). This is sometimes referred to as stateful load balancing. Optionally, the connection behavior of a service can be marked persistent. Persistency can be accomplished via manual configuration, whereas connection-integrity tracking works on a per-connection basis with finer granularity.

LVS—Virtual Services
You can configure Linux Director virtual service definition in these ways:

IP address— The virtual IP address of a global service.

TCP/UDP port— The port of a service.

Protocol— For example, HTTPS of FTP.

Linux netfilter/iptables firewall marks— A firewall mark can be matched to virtual services. This adds another level of flexibility and granularity (for example, for persistency setups).

Linux Ultra Monkey
Ultra Monkey is a project to create load-balanced and highly available services on a LAN using open-source components on the Linux operating system. Figure 12-3 describes the entire architecture
Ultra Monkey includes the following key features/components:

Layer 4 switching using the Linux Virtual Server (LVS)

High availability provided by Heartbeat protocol

Service-level monitoring using ldirectord

Supports highly available or load-balanced topologies

Integrates with Super Sparrow

Supports monitoring of HTTP, HTTPS, FTP, IMAP, POP, SMTP, LDAP, and NNTP

IP Address Takeover with Heartbeat
The takeover approach chosen largely depends on topology and effective switchover timer requirements. As a rule of thumb, MAC address takeover works fastest, IP address takeover is a little bit slower, and dynamic DNS reconfiguration takes its time. Heartbeat is an integral part of the Ultra Monkey architecture, based on ARP spoofing/gratuitous ARP for takeover with the help of an additional physical interface or an IP alias.

All Heartbeat protocols are based on the assumption that keepalive messages (hence the name heartbeat) are exchanged between systems. If a message is not received in due time, a failure is assumed and takeover or master node re-election activities are triggered. The useful thing about the heartbeat protocol is that it works over serial links, PPP links, and Ethernet and incorporates aspects of VRRP such as virtual addresses. It is much more cost-effective to use RS232 interfaces instead of committing dedicated, although almost idle, Ethernet NIC resources.


The Service Routing Redundancy Daemon (SRRD)
You have already learned a fair bit about SRRD in Chapter 9, "Dynamic Routing Protocols—Interior Gateway Protocols." For purposes of this discussion, just be aware that SRRD can do without an external heartbeat mechanism and instead just requires Quagga Open Shortest Path First (OSPF) for signaling based on opaque link-state advertisements (LSAs). Undeservedly, this elegant concept is underrepresented in HA architectures. According to http://www.srrd.org, "SRRD is fully configurable over the web and supports SSL and PKI client and server authentication. It implements cluster server features like Service Groups, Service Dependencies, as well as Critical Services."[2]

A Few Words About Content Caches and Proxies

Content-caching architectures and engines such as in the Cisco product palette deal with the challenge to deliver content reliably, efficiently, and effectively to the network edge and access layer where customers subscribe to certain content. Vice versa, they are necessary to provide sufficiently clustered server farms to feed these requests.

NOTE

Historically, caching was the initial purpose of proxies and proxy chaining. Protection of expensive and rare WAN bandwidth was their prime directive. Today, with cheap bandwidth in abundance, the focus has shifted toward intelligent security, content screening, and load-balancing content and cache-engine architectures. However, these are Layer 4 through 7 issues and not the focus of this chapter. For a background on caching strategies, look at the Internet's most popular open-source proxy, squid, at http://www.squid-cache.org/, and the proxy capabilities of the Apache web server, at http://www.apache.org.



Modern proxies fall in different categories:

Transparent caching proxies

Security (intercepting) proxies

Load-balancing proxies

Mangling proxies (packet rewrites)

Reverse proxies

VRRP
Gateway redundancy protocols solve the problem of eliminating single points of failure in a LAN environment for clients that require a reliable HA default path and next hop. Provisioning of a default route/multiple routes can, in general, be achieved in various ways:

Dynamic Host Configuration Protocol (DHCP) with default gateway provisioning

Manual default route entry/entries plus load balancing (optional Linux feature)

Dynamic routing protocols on the client workstation

Internet Router Discovery Protocol (IRDP) clients

Virtual Router Redundancy Protocol/Hot Standby Router/Protocol/Gateway Load Balancing Protocol (VRRP/HSRP/GLBP)

The concept of a secondary default route (floating static route) does not work on multi-access networks. The affected workstation has no standardized and reliable way of telling that a particular gateway just went down, and an incomplete ARP entry does not trigger a route removal from the forwarding table. This is implementation-specific and different on point-to-point links with line protocols.

VRRP (RFC 2338) is essentially an election protocol that elects a master from a pool of candidate routers and takes care of associated Layer 3 and Layer 2 virtual and real addresses for resilient LAN forwarding responsibility. The virtual router that is associated with each alternate path under VRRP uses the same IP address and MAC address as the routers for other paths. As a result, the host's gateway information does not change, no matter which path is used. This applies to both the IP and MAC address used for the actual frame delivery.

VRRP is related to the Cisco proprietary and highly successful HSRP and supplemented by GLBP. Look at the Cisco.com document "Virtual Router Redundancy Protocol" for more information. The Cisco implementation requires Cisco IOS Release 12.2(T)or 12.0ST.

CAUTION

HSRP is not compatible with VRRP.



The following sections discuss two implementations of VRRP for UNIX.

VRRPd
VRRPd is a Linux implementation of the Virtual Router Redundancy Protocol originally developed by ImageStream and released to the public as a GNU project.

VRRP's primary design goal is eliminating a single point of failure represented by a next-hop gateway for static LAN routes, often routes of last resort. Figure 12-5 presents the lab layout used for the following configurations. The specification of VRRP was influenced considerably by the Cisco HSRP protocol. In addition to this primary function (next-hop redundancy), RFC 2338 also states that the protocol should

Minimize the duration of black holes.

Minimize the steady-state bandwidth overhead and processing complexity.

Function over a variety of multiaccess LAN technologies that support IP traffic.

Provide for election of multiple virtual routers on a network for load balancing.

Support multiple logical IP subnets on a single LAN segment

OpenBSD CARP
Because of patent issues with VRRP and the Internet Engineering Task Force's (IETF) point of view about patented technology in standards (that is, RAND [reasonable and nondiscriminatory]), the OpenBSD community developed their own redundancy protocol, Common Address Redundancy Protocol (CARP), which was introduced in the OpenBSD 3.5 release. One positive aspect of that architecture is the tight integration with the pf firewall and pf's clustering capabilities via pfsync.

For more information about OpenBSD CARP, look at the CARP man page at http://www.openbsd.org/cgi-bin/man.cgi?query=carp&sektion=4. Notable features are ARP-based Layer 2 load-balancing capabilities, support for both IPv4 and IPv6, and hash protection for enhanced protocol security. Example setups are provided at http://www.countersiege.com/docpfsync-carp/.
Freevrrpd
Freevrrpd is part of the HUT Project. A feature overview according to the HUT Project includes the following:

Provides an RFC 2338-compliant daemon

Implements virtual addresses

Supports multiple virtual router IDs (VRIDs)

Enables master state announcement (by sending multicast packets via BPF)

Changes routes and IP in 3 seconds

Performs gratuitous ARP requests to clean the cache of all hosts

Elects from among different slave servers

Allows the same host to be slave and master simultaneously

Automatically downgrades to slave if a master is up again

Provides anti-address-conflict system

Provides multithreaded vrrp daemon

Supports plain-text passwords

Supports netmask for virtual IP address(es)

Simple version of "monitored circuits"

Supports executing master/backup scripts during transition state

Comparison of the VRRP Implementations
Example 12-9 provides a comparison of different VRRP implementations.

Example 12-9. VRRP Lab (Linux VRRPd, CISCO IOS Architecture, and FreeBSD freevrrpd)

scar# show running-config

...

interface ethernet 1/0

ip address 192.168.14.254 255.255.255.0

vrrp 1 priority 50

vrrp 1 ip 192.168.14.10

...



[root@castor:~#] cat /usr/local/etc/freevrrpd.conf



# Each VRID section must begin with [VRID] keyword

[VRID]

# serverid is needed to specify the number of the VRID, here VRID = 1

serverid = 1



# you must set the interface with a real interface name of your system

interface = ed0



# priority = 255 is a MASTER of the VRID

# priority < 255 is a BACKUP with a priority 0 to 254

# 254 is a higher BACKUP priority

priority = 100



# addr option is needed to specify ip address(es) associated with the VRID

# you can specify multiple addresses separated by ','

# netmask is specified with CIDR notation, so numbers after '/' represent the

# number of bits set to 1 for the netmask.

# eg: /24 is 11111111 11111111 11111111 00000000 = 255.255.255.0

addr = 192.168.14.10/24



[root@castor:~#] freevrrpd –f /usr/local/etc/freevrrpd.conf



[root@callisto~#] vrrpd i eth0 –p 255 –v 1 192.168.14.10

IRDP

The Internet Router Discovery Protocol (IRDP, RFC 1256), sometimes also referred to as the ICMP Router Discovery Protocol, is the ancestor of redundancy concepts such as VRRP or HSRP. It is a timelessly elegant concept that recently has attracted renewed attention because of its essential role in mobile IP deployments (RFC 3344). Extensions to the original protocol were necessary to provide for the requirements of mobile node-to-agent communication.

The IRDP is a protocol based on multicast route discovery ICMP messages. It eliminates the need for manual configuration of router addresses and is independent of any specific routing protocol. Therefore, it supplements a statically configured default router.

The ICMP router discovery messages are called router advertisements and router solicitations. By default, neither router discovery advertisements nor solicitations are sent over point-to-point links (for example, PPP). As the advertisement address, the router defaults to 224.0.0.1 if the router supports IP multicast on the interface; otherwise, 255.255.255.255 is used. Router solicitations are sent to 224.0.0.2. For further information, consult the RFC and the BSD manual page routed(8).

Example 12-10 demonstrates the IRDP featuring advertisements and client solicitations. In the gated example, IRDP server (advertiser) is running on ganymed, the Cisco IOS IRDP server on scar, callisto is listening via rdisc client, and castor is running routed -q "quiet mode" to demonstrate routed's IRDP client behavior; no dynamic routing protocols are running.

The preference statements (as highlighted in Example 12-10) allow weighting of the two available gateways. (This also works with two gateways on the same network.) The server side of the ICMP router discovery protocol is supported by Cisco IOS architecture, routed, and gated and can be tuned with preference statements. The command ip irdp multicast turns off the Cisco default broadcast behavior (also highlighted in Example 12-10).

Note that MRTd and Zebra do not support IRDP currently. Experimental code was added recently to Quagga's Zebra daemon, though. GateD and routed can run in either server or client mode. Linux provides a client implementation with the rdisc(8) tool. As you can see from the configurations and output, ganymed and scar are acting as IRDP advertisers (servers), whereas castor and callisto act as IRDP clients sending IRDP solicitations to trigger responses from candidate IRDP routers on directly connected networks.

Example 12-10. IRDP Server and Client Operation

[root@ganymed:~#] cat /etc/gated.cfg

### IRDP section ###

routerdiscovery server yes{

traceoptions state;

address 192.168.1.254 preference 100 multicast;

};

#routerdiscovery client yes{

# traceoptions state;

# interface ne3 multicast solicit;

#};

icmp{

traceoptions routerdiscovery;

};



[root@castor:~#]routed –q –T /var/log/routed.log



scar# show running-config

...

interface Ethernet1

ip address 192.168.14.254 255.255.255.0

no ip proxy-arp

ip irdp

ip irdp preference 10

ip irdp multicast

no ip route-cache

no ip mroute-cache

media-type 10BaseT

!

interface Ethernet0

ip address 192.168.7.254 255.255.255.0

no ip proxy-arp

ip irdp

ip irdp multicast

ip irdp preference 20

no ip route-cache

no ip mroute-cache

media-type 10BaseT

...



scar(config-if)# ip irdp ?



address addresses to proxy-advertise

holdtime how long a receiver should believe the information

maxadvertinterval maximum time between advertisements

minadvertinterval minimum time between advertisements

multicast advertisements are sent with multicasts

preference preference level for this interface



scar# show ip irdp ethernet 1

Ethernet1 has router discovery enabled

Advertisements will occur between every 450 and 600 seconds.

Advertisements are sent with multicasts.

Advertisements are valid for 1800 seconds.

Default preference will be 10.



[root@callisto:~#] rdisc -vst

Sending solicitation to ALL-ROUTERS.MCAST.NET (224.0.0.2)

ICMP Router Advertise from 192.168.14.254, lifetime 1800

address 192.168.14.254, preference 0xa

ICMP Router Advertise from ganymed (192.168.1.254), lifetime 1800

address ganymed (192.168.1.254), preference 0x64

address 192.168.45.254, preference 0x0

ICMP Router Advertise from 192.168.45.254, lifetime 1800

address ganymed (192.168.1.254), preference 0x64

address 192.168.45.254, preference 0x0



[root@castor:~#]cat /var/log/routed.log

...

turn on Router Discovery client using 192.168.7.254 via ed0

Add 0.0.0.0 -->192.168.7.254 metric=15 ed0

...



[root@callisto:~#] tethereal -i eth0

Capturing on eth0

0.000000 192.168.14.1 -> 224.0.0.2 ICMP Router solicitation

0.001527 192.168.14.254 -> 192.168.14.1 ICMP Router advertisement

...

167.026360 192.168.14.254 -> 224.0.0.1 ICMP Router advertisement



[root@callisto:~#] tethereal –i eth1

0.343388 192.168.1.1 -> 224.0.0.2 ICMP Router solicitation

0.344884 192.168.45.253 -> 224.0.0.2 ICMP Router solicitation

2.360836 192.168.1.254 -> 192.168.1.1 ICMP Router advertisement

2.361085 192.168.45.254 -> 192.168.45.253 ICMP Router advertisement



[root@callisto:~#] netstat -rne

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ipsec0

192.168.14.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

192.168.45.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo

0.0.0.0 192.168.1.254 0.0.0.0 UG 0 0 0 eth1