Bayshore Opens with Over 500 Registrations Per Minute

The Bayshore Marathon in Traverse City, MI opened this morning at 9:00AM. It is a very popular race, and there were a bunch of people ready to sign up when registration opened. And many of them signed up very, very fast – 68 in under a minute, 508 in less than 2 minutes and another […]

Availability and Infrastructure Report for 2022

One of our Year End Wrap-up Blogs. Others include 2022 Year in Review, RunSignup 2022 Product Recap and RunSignup 2023 Product Plan. Also, here is our 2023 Roadmap Webinar. We are bringing back our old Availability and Infrastructure Yearly Report after being distracted by the pandemic. We continue to post blogs about important issues that […]

Thanksgiving Infrastructure Report

Note from Dec. 15 2022 – We have deployed an upgraded layer of software that controls our session management that is adaptive to high demand and removes the problem with the Apache Webserver cache mechanism that caused some slow performance on Thanksgiving morning. We had over 750,000 people sign up for 730 Turkey Trots who […]

American Express Outage Friday Night

American Express had some outages on Friday night, November 18, 2022. About 10-15% of credit card transactions on our platform use American Express credit cards, and some of those transactions took extra time and some failed. When we try to process a transaction that fails, our processor (Adyen) gives us a status code back. Sometimes […]

Aurora 2 Database Upgrades Tuesday and Wednesday 9/27 and 9/28

As described last week, we have a planned upgrade happening on our databases this week. We have done a number of tests and are hopeful for a quick and non-impactful transition. We also have planned and tested fallbacks as well. We will be doing the Database Shard (less important database tables) on Tuesday morning, and […]

Upgrade to Aurora 2 Database – Potential Downtime

We are planning to do an upgrade to Amazon AWS Aurora MySQL 2 sometime in the next week and a half. We expect less than 5 minutes of potential system impact. Another blog will be posted the day before we do the upgrades. Planned Impact There should be minimal impact with the whole upgrade process […]

December 7 AWS Outage

A major set of problems at Amazon Web Services East Region started around 10:30 AM on December 7. Fortunately, GiveSignup | RunSignup continued to process transactions and serve our customer’s tens of thousands of websites during the entire time. Here is the post mortem description from AWS. The impact was very widespread: Ticketmaster had to […]

23 Releases and Updating 40 servers – All in a Days Work

This is Bob. We will be releasing several year end blogs over the coming weeks. A couple of them will be an overview of the features we released and another for our 2021 product roadmap. But I thought yesterday was worth mentioning. Each morning, I get an email that sends me to a page to […]

Availability Investments Help Customers Again

Like much of the web, RunSignup and GiveSignup host our infrastructure on Amazon AWS. Unfortunately, they had a pretty major outage on Wednesday before Thanksgiving in one of their data centers. It took down major websites like  1Password, Acorns, Adobe Spark, Anchor, Autodesk, Capital Gazette, Coinbase, DataCamp, Getaround, Glassdoor, Flickr, iRobot, The Philadelphia Inquirer, Pocket, RadioLab, Roku, RSS Podcasting, Tampa Bay Times, Vonage, The Washington Post, and WNYC. Apparently this was caused by their rush […]

2019 Availability & Infrastructure Report

Knock on wood, but this is our 4th year in a row with 100% uptime. As we have covered in the past, we view the core of our company is building reliable transaction processing that allows races and nonprofits to collect money. This year we will process $270 Million – allowing participants, attendees and donors […]

Daily Payment Delay

Our Processor for Advanced Payment accounts (Vantiv, now Worldpay) had an incident impacting transaction settlement and the funding instruction reports for Tuesday 11/27/2018 that resulted in us being unable to execute funding instructions for transactions processed on that date. This caused a skipped payment on 11/28/2018 for daily payment customers. Payments resumed as normal on […]

Infrastructure Updates

We have been doing a number of infrastructure updates over the past few weeks to improve availability, increase performance and assure security. For example, there was a major Amazon AWS outage on September 5 that took out the SQS queuing service for 1 hour and 40 minutes. Most other registration companies were offline, but our […]

More Performance Improvements –

We have released a number of performance improvements that make the RunSignUp site faster, especially for mobile users. The overall average page has dropped from about 2.5 seconds to 2.2 seconds, with sub-second response on modern computers and mobile phones with fast internet access. Every 6 months we take a look at how we can […]

Performance Improvements – 40 milliseconds!

We are in the midst of some infrastructure improvements. This is split among a variety of improvements – performance, security, maintainability, availability, and staying on the most current technology. We did a set of improvements in our caching and server connections that improved the core response time of a race page load from 140 milliseconds […]

Vantiv Transaction Problems

One of our payment providers, Vantiv, had a full outage on Saturday morning from 6:48AM until 7:52AM Eastern. This impacted any registrations that were for races that used that back end payment processing. People trying to register during this time saw an error message and hopefully tried again later in the day. We are working […]

The Power of Continuous Improvement and the RunSignUp Community

Some of our team was out presenting to a prospective customer yesterday. During the demo, the customer requested a feature – forcing participants to scroll thru the entire waiver form before allowing them to confirm. One of our team members sent a message to Bob to see if we could impress them. In 35 minutes, […]

SLLLOOOWWW Site for 6 Minutes

Sorry! We ran a database migration (something we do often to upgrade the database to add tables or data fields) that really hit the servers hard for about 6 minutes from about 2:12PM until 2:18PM Eastern today. We apologize for the slow down – average response time rose to 5-10 seconds. The good news is […]

2017 Availability Report

Knock on wood, but it has been another great year for RunSignUp availability, hitting a perfect 100% for the second year in a row! As we covered in February when a major outage for many sites on the Internet like Slack, Trello and Venmo, RunSignUp’s advanced infrastructure was able to weather the storm. And having […]

Geek Corner – Integrating Transaction Data into RunSignUp Analytics

You may have seen a new piece of data on your Dashboard Overview page – Transactions! As of Nov. 2, the data still shows $0: While it will be cool to see the real data in another day or so, for geeks, the interesting part is how we are going to show the data! If […]

DDOS – What a Distributed Denial of Service Attack Looks Like

We had a DDOS attack on RunSignUp today.  It lasted from about 4:00PM until 4:45PM when we were successful in cutting it off. It averaged over 1,000 requests per second. The attack was looking for vulnerabilities like SQL injection. This slowed down the average response time on our website about a half second from about 2.7 […]

Updated Database Backups

As part of our big infrastructure upgrade, we have improved our processes for database backups that are in addition to the AWS Aurora automated backups. The Aurora backups are good from a reliability point of view with features like enables point-in-time recovery. However, the backup is only good for 35 days and the time to recover […]

Failover for Read Replica Database

Yes, we know these infrastructure updates are kind of boring. But we want to document them. This one involves if there is a loss of connection to the read replica database(s). It now auto-failovers to the master database. We had two instances of this happening over the past year and wanted to handle it in […]

More Performance Improvements

We continue to do our annual big infrastructure update. Improvements to date have decreased average page load time across all devices (over 50% are mobile phones) from about 3.2 seconds to 2.85 seconds. While that does not sound like a lot, that is 980,000 seconds (11 days and 8 hours) of wait time last week […]

Upgraded Load Balancer Error Handling

As we continue to upgrade our infrastructure over these few weeks, we have improved our error handling and auto-recovery on our NGINX load balancers. What this means is if there are network or server problems, the system we have in place will automatically repair itself with near zero disruption to users. We have had in […]

Upgraded Infrastructure Hardware

We have also completed an upgrade of our hardware infrastructure over the past week. This includes upgrades to our front end load balancers, our web servers and our database server (we left the memcache servers the same since they are so over-sized for their load). The hardware upgrades have resulted in an improvement of at […]

Participant Report Speed Up

You may have noticed a speed up in the Participant Reports on your race dashboard. We did some optimizations of the queries as part of our infrastructure upgrades. We are also benefiting from the upgrade we made to the Aurora Database yesterday. The database upgrade is probably not noticeable to the human eye. Individual database […]

Amazon Aurora Database Upgrade

We did a major database upgrade to Amazon Aurora MySQL this morning (with zero downtime!). This upgrade increases performance, improves scalability, improves reliability, availability and positions RunSignUp for continued growth and change. We were one of the Beta test sites for Aurora, and were very impressed with the performance and other gains from this amazing […]

Site Upgrade Monday Morning

UPDATE: This is now complete. We had Zero transaction loss. We had registrations completing during the process successfully. We think that 3 users had to wait up to a minute for their registration to complete (the running guy at the end of registration). We are doing a major database infrastructure upgrade on Monday morning, Feb. […]

Technology Really Does Matter

Most people do not understand technology, which is fine. The problem is that many people translate that into not valuing technology or the complexity behind the many simple uses of the web and mobile apps they see every day. This winds up being a risk for race directors when it comes to evaluating registration systems. […]

RunSignUp Availability and Disaster Recovery Implementation

We have been asked by a customer to document our disaster recovery plan, and we figured that it might be useful to document this for other customers as well as provide an example to other websites who might want to use a modern approach to these topics. As with most modern apps, most of our […]

Database Upgrade – New Financial System

This was a major upgrade and we had to do a major database upgrade which caused our system to be unavailable for less than 5 minutes this morning. There was a gap of about 6 minutes between the last registration before the upgrade and the first registration after the upgrade was complete. The good news […]

Boilermaker Opening

Boilermaker opened “Open” registration today on RunSignUp. They had opened a loyalty program last week. Things went well, as you can see from the post on the right! We always like to do an assessment of our systems and how things worked and share that publicly. The first person registered in 54 seconds – completing […]

WordPress