Elements of a globally distributed HA cluster

by Jason Rexilius
This presentation is on the technical elements of the web transaction model that impact the design and architecture of a geographically distributed HA web cluster.

Background

There are a number of approaches to building scalable, highly-available web applications. As with most things there are trade-offs between cost and goals and the nature of the application involved can have a significant impact on these choices.

Web applications today can be grouped into a few major categories:

• static content delivery - typically thought of as only brochure-ware, however there are a number of other profiles such as download sites (music, software, content, etc.), semi-static data (mapping, census, geospatial data, etc.), corporate compliance (SEC regs for publicly traded companies).
• lightly cutomized or processed semi-static content sites - news, maps, information, pre-processed reports, code repositories, etc. These typically may have user login, profile or customization, and light access controls all wrapped around data that is mostly read-only from the users perspective.
• interactive non-realtime applications - typically ecommerce sites, online banking, common business applications. These involve significant interaction with heavy writes to session and moderate writes to data persistence layers.
• fat-client replacement real-time web applications - the most demanding type of application that requires performance in all aspects of its data IO and heavy analytics (hits network, CPU, mem, and hard disk). These are typically trading applications, real-time business processing applications, and the like.

Each of these, obviously, has differing demands and constraints. The bottlenecks and costs will change with each category as well as with the nature of the data and usage patterns. External constraints such as data intergity tolerance, performance tolerance, and integration can also alter the model quite a bit.

Typically one or more of the following approaches are used to achieve scaling, performance, and/or availability beyond a single node:

• DNS Round Robin
• Content caching proxies
• HTTP/TC/IP load balancing proxy method
• HTTP load balancing redirect method
• HA heartbeat, IP assumption
• App Server Clustering (shared memory space)
• NFS or netowrk shared file storage
• RDBMS replication
• RDBMS clustering

There are a few other cutting-edge or non-mainstream approaches such as DB connection clustering (RAIDb), high-performance caching, distributed file systems, and the like.

The Elements of a Web Transaction

To clarify, the term transaction will be used loosely to describe a logical Input/Processing/Output sequence regardless of method, protocol, or application. So transaction can mean an HTTP Request/App Server Processing/HTTP Reply sequence as much as a SQL update in an RDBMS.

The sequence of events and elements involved in an atomic web transaction are as follows:

• User Agent, typically, resolves the Host Name portion of a URL into an IP address. note 1
• The User Agent establishes a TCP (often SSL wrapped) connection with a webserver.
• The User Agent submits an HTTP Request, often with a payload (POST body) and a session identifier (typically a cookie).
• The web server process the request and hands it off to application logic if needed.
• The application logic will interact with both the session data and, usually, some form of data persistence layer (RDBMS), execute logic, persist state and data as needed, and provide an HTTP Reply.
• The User Agent will receive the HTTP Reply and deliver or it to the client.

So the elements in this sequence are:

• DNS Resolution
• SSL/TCP/IP connection establishement
• HTTP Request
• Session Identification
• Request execution
• Session data store I/O
• Data Persistence I/O
• HTTP Reply

Additionally (although they are truly just another piece of application logic), there are concepts of (Access Control):

• User Authentication - (who are you)
• Process Authorization - (are you allowed to do that)
• Contextual Authorization - (are there limits or constraints on doing)

Considerations To Architectural Approach

Given an application profile, requirements and constraints you can start to design a system that mitigates the choke points and break points in the elements of the web transaction model. A few other real-world considerations can come into play that should be accounted for in the design. A partial list follows:

• Maintainability - does your organization have the ability to maintain the approach
• Developer/Admin Skill - do your developers and admins have the ability to work with certain levels of complexity
• Third party requirements - the evil nemesis of architecture, vendor lock-in or other third-party demands

Typical Transaction Flow

common problem spots are highlighted in yellow..
dotted lines are implicit flow communications, solid lines are explicit..

Current Approaches

Here are some really great presentations on existing approaches:

• Clustering - http://talks.php.net/show/clustering-ezcamp2005
• Massive Scaling - http://talks.php.net/show/lt2004-lamp
• Performance Tuning - http://talks.php.net/show/perf_tunning

Essential Elements of Extending Current Approaches

This is usefull background information to extending the above approaches:

• PHP Internals - http://talks.php.net/show/php-ticks-ffm2005
• Writing Extensions - http://talks.php.net/show/vancouver-ext
• Hacking PHP - http://talks.php.net/show/hacking-fall-2003
• Encryption in PHP - http://talks.php.net/show/playingsafe-tropics
• Session Security - http://talks.php.net/show/phpworks2004-php-session-security
• Web Services - http://talks.php.net/show/sdphp_webservices
• SRM - http://talks.php.net/show/srm-ez-internal

The Next Step

The current state of art provides the ability to scale horizontally by adding nodes to a local cluster. By leveraging LVS and a few other platform technologies High Availability can be achieved as well as Scaling. And by following tuning and design best practices one can achieve reliable Performance.

But what about the constraints of the single data center. Bandwidth saturation, physical distance latency, facility outages, DoS, local disaster, fire, theft, ISP failures..

lions and tigers and bears, oh my..

To achieve extreme high availability, massive scaling, optimal performance and mitigate vendor issues it is desirable to go beyond a single data center or service provider.

This is where geographically distributed clusters of clusters comes into play.

It shares technical challenges and approaches with grid computing but it is largely assumed that grid computing means shared resources, which, is more of an organizational distinction.

And now for The Approach For A Globally Distributed Web Cluster