Geographically Distributed Web Cluster Approach

by Jason Rexilius

This is the approach my company is taking to building an extreme high-availability, scalable platform for delivering software as a service.

We currently have a basic model in place with two data center locations and are building out the rest of the platform this year.

It may be helpful to read the background article on the Web Transaction Model if you are not familiar with the links in the chain of web transactions.

1) Location, Location, Location!

it matters even on the 'Net

• Where Are Your Users?

• Providers Gone Bad

Level 3 and Cogent - http://www.isp-planet.com/business/2005/cogent_level_3.html
FBI raids hoster - http://www.carrierhotels.com/news/2004/Feb/19/fbi_shutters_web_host.shtml
Akamai Outage - http://www.internetnews.com/infra/article.php/3369371
Equinix Outage - http://blogs.feedburner.com/feedburner/archives/001280.html

• Mother Nature's Mood Swings

• Pissed Off People With Planes

2) High Level Overview

the same thing we do everyday Pinky..

• 5 locations to cover globe

• 99.999% availability for 10 million users

http://en.wikipedia.org/wiki/The_Myth_of_the_Nines

• Lies, Damn Lies, and Statistics

http://www.spec.org/web99/
single properly tuned node should handle:
- 2,000 concurrent SSL connections
- 1 transaction per second, per connection
- 10meg memory per connection/transaction

3) DNS

first link in the chain

• RFC spec allows for "about" 5 Authoritative Name Servers in record.
• everage DNS lookups for HA, load-balancing and performance
• resolve request - response algorithm points users to closest, available and least-loaded node
• 1 auth NS per DC, if whole DC/region down, requests go elsewhere
• MS Windows and some ISPs ignore TTL, 1 hour Max is safest
• Assuming 5 AuthNS, even distribution and RFC compliant TTL expires, loss of an entire datacenter effects 20% of users for 1 hour or less. gives 99.99% uptime

notes:

- Win -> Start > Run > Cmd > ipconfig /flushdns
- OSX -> lookupd -flushcache
- Lin -> /etc/init.d/nscd restart
- more Windows hacks
	http://support.microsoft.com/default.aspx?scid=kb;en-us;318803
- Block Negative Entries
	HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters
	DWORD: MaxNegativeCacheTtl
	Value: 0
- Cache TTL
	HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters
	DWORD: MaxCacheTtl
	Value: 60

4) Node Clusters

current approach within single data center

• LVS, IP assumption w/ keepalived
• Assuming properly redundency within DC, loss of single node, with ARP cache and route updates will effect n% of users for less than 1 minute. gives 99.999% uptime

5) Self Describing Distributed Sessions

dealing with bouncing users

• what happens when a user shows up at a node in Boston when they were last at a node in Chicago?
• load-balancing, redirects, fail-overs and the like should be transparent to both user and developer
• Hack The Session Manager (modify PHP internals)
• use symetric encrypted cookies for data persistence, user carries session with them in one line of code:
• store session in two memcached locations, store pointer to locations in encrypted cookie http://www.danga.com/memcached/

// AT TRANSACTION (PAGE) BEGINNING: if(isset($_COOKIE['SESDATA'])){$_SESSION=unserialize(gzuncompress(base64_decode(myDecrypt(urldecode($_COOKIE['SESDATA'])))));}
// AT TRANSACTION (PAGE) END:
setcookie('SESDATA',urlencode(myEncrypt(base64_encode(gzcompress(serialize($_SESSION))))),0,'/','',1);

• 4k of data is limit in most browsers cookie jar. So real method is

if ( $Compressed_Encrypted_Session_Data < 4000 bytes ) {

   Store_Session_In_Cookie( $Compressed_Encrypted_Session_Data )
 }else{
   Store_Session_In_Two_Nodes( $Node1, $Node2, $Compressed_Encrypted_Session_Data )
   Store_Location_Of_Session_In_Cookie ( $Node1, $Node2 )
 }

• storing session in cookie adds bandwidth cost
• store in remote locations adds latency at page begin and end
• data cleanup and garbage collection needs to be managed (unless hack PHP session code)

6) Data Persistence

those damn DB's, too big to cover in depth

• SRM or other remoting abstraction layer http://www.vl-srm.net/index.php
• RAIDb or DB connection cluster http://c-jdbc.objectweb.org/
• Partioned DBs, mirrored across nodes, with Hash Map data lookups

7) Tidbits

• dedicated fiber between locations, BGP, class C from ARIN
• 30ms latency between locations, except hopping ocean (100-500ms)