The network in a cluster consists of several independent parts:
- The cluster interconnect acts as the communication backbone between the nodes
and needs to be a high speed network with low latency and high
bandwidth. 100 Mbps Ethernet is minimum, 1000 Mbps typical and special high speed networks are nice to have.
The interconnect is primarily used for inter node communication as there is:
- Cluster management
- node boot, cluster membership (heartbeat, I'm alive), load balancing, cluster administration.
- Remote file access
- open, read and write of files (and devices) that are managed on remote nodes, record and/or file locking, synchronization of I/O.
- packet routing between nodes on different subnets or nodes with dissimilar protocol stacks.
- Distributed applications
- any form of communication among parallel instances of an application or middleware (distributed database, distributed lock manager, applications implemented via PVM or MPI).
- The storage area network is an optional part used to access disk and tape devices. Network hubs and switches give shared access to multiple nodes enabling both scalable and highly available services. Fiber channel is the dominant network media today but IP based protocols like iSCSI and HyperSCSI are gaining momentum.
- While the former two were private the public network lets clients access services on the cluster. Synchronous protocols like SDLC and X.25 played a major role in the past. They scaled by cloning (replicated controller and links) and a minimum of two links is required to achieve high availability. In todays computing centers the ubiquitous IP protocol rules. Virtual private networks (VPN) give you scalable, uninterrupted access to almost any place on the globe.
Within the cluster IP scales by cloning. Vertical scalability increases the capacity of links but protocol stacks scale only if parallel or SMP capable implementations are available. Failure protection is at best limited to controller and link failures. Horizontal scalability does it all: adding nodes with additional IP interfaces expands the link and protocol bandwidth (without contention) and guarantees high availability through a variety of techniques.
In order to appear as a single entity clusters need a single (or multiple) IP address that clients connect to. But at the same time clusters require scalable and fault tolerant networking which is achieved by interface cloning. You solve these conflicting requirements by using virtual IP addresses (VIP). A VIP is advertised on the network but is not bound to a physical interface before an ARP request comes in. Binding the VIP at ARP time yields both scalability and high availability and works best with a large population of clients and short sessions.
VIPs are implemented either
- within the cluster running on a dedicated (director) node that answers ARP requests and passes the request to other nodes that service the request and reply either directly to the client or via the director. Making this setup highly available is not a trivial task.
- outside of the cluster running on an external load balancer. A variety of devices exist that differ in network throughput, load balancing algorithms and other features. Again care has to be taken to avoid SPOF.