A major set of problems at Amazon Web Services East Region started around 10:30 AM on December 7. Fortunately, GiveSignup | RunSignup continued to process transactions and serve our customer’s tens of thousands of websites during the entire time. Here is the post mortem description from AWS. The impact was very widespread: Ticketmaster had to […]
Unfortunately, our 5 year, 9 month and 1 day of 100% uptime with zero time down even for maintenance of the GiveSignup | RunSignup system came to an end at 4:16 Eastern time on June 28, 2021. It lasted for 4 minutes before we were fully operational again. The last time the system was unavailable […]
We usually do our Year in Review in a single blog post. This year has been much too big to put it all in one place, so we are breaking it up a bit into this one, with a recap of this year, and then a blog about our 2021 roadmap. The development team had […]
This is Bob. We will be releasing several year end blogs over the coming weeks. A couple of them will be an overview of the features we released and another for our 2021 product roadmap. But I thought yesterday was worth mentioning. Each morning, I get an email that sends me to a page to […]
Like much of the web, RunSignup and GiveSignup host our infrastructure on Amazon AWS. Unfortunately, they had a pretty major outage on Wednesday before Thanksgiving in one of their data centers. It took down major websites like 1Password, Acorns, Adobe Spark, Anchor, Autodesk, Capital Gazette, Coinbase, DataCamp, Getaround, Glassdoor, Flickr, iRobot, The Philadelphia Inquirer, Pocket, RadioLab, Roku, RSS Podcasting, Tampa Bay Times, Vonage, The Washington Post, and WNYC. Apparently this was caused by their rush […]
Over the past couple of weeks the RunSignup and GiveSignup infrastructure has gone thru a large upgrade. The good news is that there has been no downtime! Also, the site speed has increased a bit as well and just squeaked under 2 seconds last week when there were over 5 Million Pageviews, even though speed […]
Knock on wood, but it was the third year in a row with 100% uptime – zero downtime. We actually just completed a migration to the newest version of PHP that as you can see from the graph below saved about 15-20 milliseconds: We had several highlights from the year on the positive side: Major […]
Our Processor for Advanced Payment accounts (Vantiv, now Worldpay) had an incident impacting transaction settlement and the funding instruction reports for Tuesday 11/27/2018 that resulted in us being unable to execute funding instructions for transactions processed on that date. This caused a skipped payment on 11/28/2018 for daily payment customers. Payments resumed as normal on […]
As we’ve discussed before, part of being PCI Level 1 Compliant is doing monthly updates to our infrastructure. This involves upgrades and updates and patches to the operating systems and other core pieces like NGINX and Apache servers. This month we also did an upgrade to the latest version of the Aurora Database on Amazon. […]
Soooo, we messed up. A large marathon used our email system to send out pre-race information emails. One of the cool features of our CRM-driven email is that you can put “tags” into an email and they populate from the database automatically. Pretty cool. Except when there is a miscommunication and two of the tags […]
We have been doing a number of infrastructure updates over the past few weeks to improve availability, increase performance and assure security. For example, there was a major Amazon AWS outage on September 5 that took out the SQS queuing service for 1 hour and 40 minutes. Most other registration companies were offline, but our […]
We have released a number of performance improvements that make the RunSignUp site faster, especially for mobile users. The overall average page has dropped from about 2.5 seconds to 2.2 seconds, with sub-second response on modern computers and mobile phones with fast internet access. Every 6 months we take a look at how we can […]
We are in the midst of some infrastructure improvements. This is split among a variety of improvements – performance, security, maintainability, availability, and staying on the most current technology. We did a set of improvements in our caching and server connections that improved the core response time of a race page load from 140 milliseconds […]
One of our payment providers, Vantiv, had a full outage on Saturday morning from 6:48AM until 7:52AM Eastern. This impacted any registrations that were for races that used that back end payment processing. People trying to register during this time saw an error message and hopefully tried again later in the day. We are working […]
Some of our team was out presenting to a prospective customer yesterday. During the demo, the customer requested a feature – forcing participants to scroll thru the entire waiver form before allowing them to confirm. One of our team members sent a message to Bob to see if we could impress them. In 35 minutes, […]
Sorry! We ran a database migration (something we do often to upgrade the database to add tables or data fields) that really hit the servers hard for about 6 minutes from about 2:12PM until 2:18PM Eastern today. We apologize for the slow down – average response time rose to 5-10 seconds. The good news is […]
Knock on wood, but it has been another great year for RunSignUp availability, hitting a perfect 100% for the second year in a row! As we covered in February when a major outage for many sites on the Internet like Slack, Trello and Venmo, RunSignUp’s advanced infrastructure was able to weather the storm. And having […]
You may have seen a new piece of data on your Dashboard Overview page – Transactions! As of Nov. 2, the data still shows $0: While it will be cool to see the real data in another day or so, for geeks, the interesting part is how we are going to show the data! If […]
One of the early disaster recovery mechanisms we built into RunSignUp is saving a full list of your participants in a CSV file once a day starting the week before your race. We send race directors an email link when this is first created each year. In the event RunSignUp is not available (knock on […]
Amazon, the leading Cloud provider had some major issues today. The screen shot on the right shows their status page around 1PM today – and RED is not good. Even USA Today, and CNN reported the 4-5 hour outage this afternoon: “People reported outages and delays on services like Slack, Trello, Sprinklr, Venmo and even […]
We had a DDOS attack on RunSignUp today. It lasted from about 4:00PM until 4:45PM when we were successful in cutting it off. It averaged over 1,000 requests per second. The attack was looking for vulnerabilities like SQL injection. This slowed down the average response time on our website about a half second from about 2.7 […]
UPDATE: We have made two changes to our system to try to improve this situation if it happens again. First, we have adjusted the settings on our New Relic Monitoring tool to try to catch this type of error so we can respond more quickly than hearing about errors from users. Second, we have added […]
We try to improve our infrastructure in Q1 of each year to make sure we do not build up technical debt and stay on top of the most current trends in technology. We have four goals when we look at this each year: Improve Availability – reduce the chance our systems will go down, and […]
As part of our big infrastructure upgrade, we have improved our processes for database backups that are in addition to the AWS Aurora automated backups. The Aurora backups are good from a reliability point of view with features like enables point-in-time recovery. However, the backup is only good for 35 days and the time to recover […]
There are a number of processes that run on a scheduled basis in a big system like RunSignUp – these are called Chronjobs. They are typically run on one server in most environments, and that is how we did things until today. As part of our infrastructure upgrade, we have added the ability for these […]
Yes, we know these infrastructure updates are kind of boring. But we want to document them. This one involves if there is a loss of connection to the read replica database(s). It now auto-failovers to the master database. We had two instances of this happening over the past year and wanted to handle it in […]
We continue to do our annual big infrastructure update. Improvements to date have decreased average page load time across all devices (over 50% are mobile phones) from about 3.2 seconds to 2.85 seconds. While that does not sound like a lot, that is 980,000 seconds (11 days and 8 hours) of wait time last week […]
We have upgraded our web servers to PHP 5.6. to continue to keep our software stacks up to date. We have also upgraded our log handling mechanism. We now save logs for 3 years in AWS S3. We make use of the the Amazon Glacier storage after the first year to save on costs.
As we continue to upgrade our infrastructure over these few weeks, we have improved our error handling and auto-recovery on our NGINX load balancers. What this means is if there are network or server problems, the system we have in place will automatically repair itself with near zero disruption to users. We have had in […]
We have also completed an upgrade of our hardware infrastructure over the past week. This includes upgrades to our front end load balancers, our web servers and our database server (we left the memcache servers the same since they are so over-sized for their load). The hardware upgrades have resulted in an improvement of at […]
You may have noticed a speed up in the Participant Reports on your race dashboard. We did some optimizations of the queries as part of our infrastructure upgrades. We are also benefiting from the upgrade we made to the Aurora Database yesterday. The database upgrade is probably not noticeable to the human eye. Individual database […]
We did a major database upgrade to Amazon Aurora MySQL this morning (with zero downtime!). This upgrade increases performance, improves scalability, improves reliability, availability and positions RunSignUp for continued growth and change. We were one of the Beta test sites for Aurora, and were very impressed with the performance and other gains from this amazing […]
UPDATE: This is now complete. We had Zero transaction loss. We had registrations completing during the process successfully. We think that 3 users had to wait up to a minute for their registration to complete (the running guy at the end of registration). We are doing a major database infrastructure upgrade on Monday morning, Feb. […]
Most people do not understand technology, which is fine. The problem is that many people translate that into not valuing technology or the complexity behind the many simple uses of the web and mobile apps they see every day. This winds up being a risk for race directors when it comes to evaluating registration systems. […]
Perhaps tempting fate writing this 2 weeks before the end of the year… We had 8 minutes of total down time in 2015. That is 99.998% uptime. The good news is that all 8 minutes were planned downtime for required system upgrades and we were able to notify users and not miss transactions: 2 Minutes […]
We did a database upgrade this morning, and some users may have received an error message during the approximately 1 minute the database was down. If they re-submitted their page everything would continue to process as normal. We upgraded from MySQL 5.5 to 5.6 on our main database, our read replica and our shard servers. […]
We had a non-critical cache outage last night that caused some response time to be over 2 seconds from 4:05 AM until 5:15AM. Due to our highly available design, the only thing users would have noticed is the system being a bit slower. There were no lost transactions or availability. We are reporting this as […]
For the first 6 months of 2015 we had 99.998 availability, with NO unplanned outages (knock on wood). We have had 1,590 releases of our site this year thru August 3, with only 2 requiring us to take the site down (2 minutes once and 5 minutes once). The two planned outages in the first […]
At around 2:00 AM on July 31 our Cloud provider, Amazon AWS, had trouble with three of the services that we use (SQS, SNS, and SES) simultaneously. The outage lasted until about 3:40 AM. This impacted about 50 transactions. We have been able to recover all but 3 of those transactions, and have contacted those […]
We have been asked by a customer to document our disaster recovery plan, and we figured that it might be useful to document this for other customers as well as provide an example to other websites who might want to use a modern approach to these topics. As with most modern apps, most of our […]
This was a major upgrade and we had to do a major database upgrade which caused our system to be unavailable for less than 5 minutes this morning. There was a gap of about 6 minutes between the last registration before the upgrade and the first registration after the upgrade was complete. The good news […]
We have written about this before. At RunSignup this is both a cultural component as well as a technical foundation. Culturally, we seek to continuously improve. We try to develop good solutions for customers, but they are never perfect. We learn from the feedback and then make improvements. You can see this by how we […]
Boilermaker opened “Open” registration today on RunSignUp. They had opened a loyalty program last week. Things went well, as you can see from the post on the right! We always like to do an assessment of our systems and how things worked and share that publicly. The first person registered in 54 seconds – completing […]
Boilermaker, the historic 15K road race in Utica, NY, opened up a Loyalty Program registration this morning at noon. They are opening up registration only to finishers of last year’s race for a week – next weekend they will open remaining slots to everyone else. In the first 10 minutes over 1,800 people registered. 30 […]
UPDATE: the migration was successful. It took 2 minutes from 10:45 – 10:47. There were no lost transactions, but some people did have to wait on the checkout page or click refresh. A second migration ran from 3:00PM- 3:04 PM. We are doing database maintenance and updates this morning. This may cause minor delays of […]
In 2014 we had outages totaling 16-21 minutes (depending on how you count an outage) for an availability of 99.996%. This is down from 99.998% in 2013. As usual, we do a year end (well, almost – there are still 7 days to go as this is written) report on our availability for the year. […]
Our site was down from 6:31 PM Eastern for about 8 minutes. This was due to a deployment we did, which ironically enough was to help prevent issues if the Amazon Content Delivery Network was unavailable, we would be able to quickly recover from that. We have since deployed the change successfully with no interruption.
It looks like some transactions this afternoon about 4:19 PM Eastern were having trouble being processed by First Data. These are fixed as of 5:12 PM Eastern. Registrants would have seen an error message stating that the Payment Processor Network is Unavailable and to try again. Most transactions were going thru, but races may want […]
We had a 5 minute outage early this morning for 5 minutes at 12:53AM Eastern time. We were doing a deployment and it triggered errors. That error was corrected as quickly as possible. We apologize for the outage. This deployment we did was part of investigating a tricky bug that is causing 1 error per […]
We have updated our public facing servers with the patch for this new security vulnerability that was found and reported on yesterday. Here are is a link to learn more if you are interested, or to check to make sure your own servers have installed this: http://www.csoonline.com/article/2687265/application-security/remote-exploit-in-bash-cve-2014-6271.html Note there are likely more patches coming from the […]
