More and more websites these days are relying on external resources for data and functionality. Each one of these services introduces a stability hole in your application. You're now relying on the uptime of an external resource. Yahoo, Google, etc.. ALL have downtime at some point. No one has a 100% SLA. This means that more than ever you need to code defensively to prevent external resources from tarnishing the user experience of your site. This is where the Circuit Breaker Stability Pattern comes into play (From the book "Release It").
The Circuit Breaker Pattern is a simple concept. If too many connections fail while accessing a resource the circuit is "opened" or stopped at that point. After a certain period of time we let one connection go through, if that one works the circuit is re-closed and operations resume as normal. Let's break this down further.
Your site connects to Google to get a list of widgets. The Google service goes down and now your customer is stuck in a timeout waiting for the connection to fail. You have no idea this is happening. You think everything is fine, until people get tired of waiting, your connections pool up and you run out of resources on your machines while 10,000 people are waiting 30 seconds to timeout. This makes your site look horrible to the user and they find somewhere else to query for these widgets. The goal is that if something is constantly failing you stop calling it and notify administrators and users.
The Circuit Breaker pattern defines a max number of failures (threshold) that can occur before the circuit is tripped. So let's say we want to stop connecting to Google after 5 people's connections have failed in a row. Now we can immediately return responses back to our users letting them know the service is down or they're query has been stored for processing at a later time. An informed user is a happy user, even if something does work they like to know about it immediately. After 10 minutes or so we want to open up ONE connection to test the waters to see if Google is still down so we open the circuit "halfway". If that connections fails we reopen the circuit and await the next timeout. Once a connection is successful again we re-close the circuit and operations can resume as normal. When a circuit does open we want to alert our operations group or the site admin so they know there is a problem with this service.
This allows you to automatically shut off and re-instate services if failures occur. It's a very powerful concept in increasing the stability of your system.
I've created a very simple, sample Circuit Breaker class in PHP that can be used for this purpose. It currently assumes a mysql database to store this information but you can replace it with a memcached or other in-memory caching option for increased performance. You can find all the code and samples here:
http://code.google.com/p/plushcode/source/browse/#svn/trunk/stability_patterns/circuit_breaker
The schema is listed out below for the table
CREATE TABLE IF NOT EXISTS `circuit_breaker` (
`id` int(5) NOT NULL auto_increment,
`app_id` varchar(255) NOT NULL,
`failure_count` int(5) NOT NULL,
`threshold` int(5) NOT NULL,
`state` enum('open','closed','half') NOT NULL,
`timeout` int(5) NOT NULL COMMENT 'timeout in seconds',
`last_open_time` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
Let's start going through the fields....
app_id - Each external resource you have (including databases!) should get it's own application id that is unique. Something such as "google_widgets" works fine.
failure_count - Each time a failure occurs this is incremented by 1. Once a successful call has been made it is reset to 0
threshold - The number of times a service can fail before we open the circuit and stop all processing
state - The current state of the service. It's either closed (normal), open(failure occurred) or half(we're trying one more time)
timeout - The number of seconds to wait before attempting a connection again
last_open_time - A timestamp when the circuit was last opened which would indicate a failure
One of the nice aspects to having this in the database is that you can easily monitor all of your external services from one site. All you have to do is query the table and you can see exactly what the health of your system is. Here is a quick and dirty sample report
http://www.litfuel.net/tutorials/cb_report.htm
To use this class you simply call it as seen below:
require_once('CircuitBreaker.php');
$cb = new CircuitBreaker('myapp');
$cb->logging = true;
if($cb->isClosed()) {
$cb->fail();
} else {
$cb->success();
}
} else {
// notify the user, send them to a different service or anything!
echo 'CIRCUIT TRIPPED! NO CALLS MADE';
return false;
}
-
What we're doing is including the CircuitBreaker class, then we're making sure the circuit is closed which means everything is A-OK. We make a web services call over to my test server page. If that connection fails then we call $cb->fail which then tests to see if our failure count is over our threshold and if not, increases it by one. If we are over the threshold it will open the circuit and all future calls will return false when isClosed() is called. If the connection is successful then we call $cb->success() which resets the failures back to 0.
To test this out:
1.create a test database
2.import the schema into the new database
3.enter in one record called "myapp" as the app_id, put in 5 as the threshold and 10 as the timeout value
4.paste the Test.php code into your own file
5.When you run the code you should see a var dump of some xml data which indicates success
6.Now change the url in the file_get_contents line to a url that does not exist
7.Start to refresh your page and you start to see the connections fail and then the circuit trips, after 10 seconds it opens up halfway, does one test then reopens for another 10 seconds
