Partial Outage

In our ongoing effort to report all availability issues, we experienced a partial issue this morning from about 5:30AM until 8:30AM Eastern.  Due to our high level of redundancy, we processed transactions throughout and user would have only had an issue on their initial connection to the site and upon clicking refresh they would have gotten through.  This was probably less than half of users hitting a page for the first time.

Our automated processes to detect and repair the instance were not successful since there was a hardware failure on one of the load balancers.  The script normally handles this type of issue, but there was a unique occurrence we had not seen before.  Needless to say, we have updated the script and can now handle this type of error.

Here is a log of the successful transactions so you can see that most went thru.  The ramp up after 8 is normal since that is when more users are awake and beginning to sign up for races: