Daily Payment Delay

Our Processor for Advanced Payment accounts (Vantiv, now Worldpay) had an incident impacting transaction settlement and the funding instruction reports for Tuesday 11/27/2018 that resulted in us being unable to execute funding instructions for transactions processed on that date. This caused a skipped payment on 11/28/2018 for daily payment customers. Payments resumed as normal on […]

Infrastructure Updates

We have been doing a number of infrastructure updates over the past few weeks to improve availability, increase performance and assure security. For example, there was a major Amazon AWS outage on September 5 that took out the SQS queuing service for 1 hour and 40 minutes. Most other registration companies were offline, but our […]

More Performance Improvements –

We have released a number of performance improvements that make the RunSignUp site faster, especially for mobile users. The overall average page has dropped from about 2.5 seconds to 2.2 seconds, with sub-second response on modern computers and mobile phones with fast internet access. Every 6 months we take a look at how we can […]

Performance Improvements – 40 milliseconds!

We are in the midst of some infrastructure improvements. This is split among a variety of improvements – performance, security, maintainability, availability, and staying on the most current technology. We did a set of improvements in our caching and server connections that improved the core response time of a race page load from 140 milliseconds […]

Vantiv Transaction Problems

One of our payment providers, Vantiv, had a full outage on Saturday morning from 6:48AM until 7:52AM Eastern. This impacted any registrations that were for races that used that back end payment processing. People trying to register during this time saw an error message and hopefully tried again later in the day. We are working […]

The Power of Continuous Improvement and the RunSignUp Community

Some of our team was out presenting to a prospective customer yesterday. During the demo, the customer requested a feature – forcing participants to scroll thru the entire waiver form before allowing them to confirm. One of our team members sent a message to Bob to see if we could impress them. In 35 minutes, […]

SLLLOOOWWW Site for 6 Minutes

Sorry! We ran a database migration (something we do often to upgrade the database to add tables or data fields) that really hit the servers hard for about 6 minutes from about 2:12PM until 2:18PM Eastern today. We apologize for the slow down – average response time rose to 5-10 seconds. The good news is […]

2017 Availability Report

Knock on wood, but it has been another great year for RunSignUp availability, hitting a perfect 100% for the second year in a row! As we covered in February when a major outage for many sites on the Internet like Slack, Trello and Venmo, RunSignUp’s advanced infrastructure was able to weather the storm. And having […]

Geek Corner – Integrating Transaction Data into RunSignUp Analytics

You may have seen a new piece of data on your Dashboard Overview page – Transactions! As of Nov. 2, the data still shows $0: While it will be cool to see the real data in another day or so, for geeks, the interesting part is how we are going to show the data! If […]

DDOS – What a Distributed Denial of Service Attack Looks Like

We had a DDOS attack on RunSignUp today.  It lasted from about 4:00PM until 4:45PM when we were successful in cutting it off. It averaged over 1,000 requests per second. The attack was looking for vulnerabilities like SQL injection. This slowed down the average response time on our website about a half second from about 2.7 […]

Updated Database Backups

As part of our big infrastructure upgrade, we have improved our processes for database backups that are in addition to the AWS Aurora automated backups. The Aurora backups are good from a reliability point of view with features like enables point-in-time recovery. However, the backup is only good for 35 days and the time to recover […]

Failover for Read Replica Database

Yes, we know these infrastructure updates are kind of boring. But we want to document them. This one involves if there is a loss of connection to the read replica database(s). It now auto-failovers to the master database. We had two instances of this happening over the past year and wanted to handle it in […]

More Performance Improvements

We continue to do our annual big infrastructure update. Improvements to date have decreased average page load time across all devices (over 50% are mobile phones) from about 3.2 seconds to 2.85 seconds. While that does not sound like a lot, that is 980,000 seconds (11 days and 8 hours) of wait time last week […]

Upgraded Load Balancer Error Handling

As we continue to upgrade our infrastructure over these few weeks, we have improved our error handling and auto-recovery on our NGINX load balancers. What this means is if there are network or server problems, the system we have in place will automatically repair itself with near zero disruption to users. We have had in […]

Upgraded Infrastructure Hardware

We have also completed an upgrade of our hardware infrastructure over the past week. This includes upgrades to our front end load balancers, our web servers and our database server (we left the memcache servers the same since they are so over-sized for their load). The hardware upgrades have resulted in an improvement of at […]

Participant Report Speed Up

You may have noticed a speed up in the Participant Reports on your race dashboard. We did some optimizations of the queries as part of our infrastructure upgrades. We are also benefiting from the upgrade we made to the Aurora Database yesterday. The database upgrade is probably not noticeable to the human eye. Individual database […]

Amazon Aurora Database Upgrade

We did a major database upgrade to Amazon Aurora MySQL this morning (with zero downtime!). This upgrade increases performance, improves scalability, improves reliability, availability and positions RunSignUp for continued growth and change. We were one of the Beta test sites for Aurora, and were very impressed with the performance and other gains from this amazing […]

Site Upgrade Monday Morning

UPDATE: This is now complete. We had Zero transaction loss. We had registrations completing during the process successfully. We think that 3 users had to wait up to a minute for their registration to complete (the running guy at the end of registration). We are doing a major database infrastructure upgrade on Monday morning, Feb. […]

Technology Really Does Matter

Most people do not understand technology, which is fine. The problem is that many people translate that into not valuing technology or the complexity behind the many simple uses of the web and mobile apps they see every day. This winds up being a risk for race directors when it comes to evaluating registration systems. […]

RunSignUp Availability and Disaster Recovery Implementation

We have been asked by a customer to document our disaster recovery plan, and we figured that it might be useful to document this for other customers as well as provide an example to other websites who might want to use a modern approach to these topics. As with most modern apps, most of our […]

Database Upgrade – New Financial System

This was a major upgrade and we had to do a major database upgrade which caused our system to be unavailable for less than 5 minutes this morning. There was a gap of about 6 minutes between the last registration before the upgrade and the first registration after the upgrade was complete. The good news […]

Boilermaker Opening

Boilermaker opened “Open” registration today on RunSignUp. They had opened a loyalty program last week. Things went well, as you can see from the post on the right! We always like to do an assessment of our systems and how things worked and share that publicly. The first person registered in 54 seconds – completing […]

Boilermaker Loyalty Program Opening

Boilermaker, the historic 15K road race in Utica, NY, opened up a Loyalty Program registration this morning at noon. They are opening up registration only to finishers of last year’s race for a week – next weekend they will open remaining slots to everyone else. In the first 10 minutes over 1,800 people registered. 30 […]

First Data Partial Processing Failure

It looks like some transactions this afternoon about 4:19 PM Eastern were having trouble being processed by First Data.  These are fixed as of 5:12 PM Eastern.  Registrants would have seen an error message stating that the Payment Processor Network is Unavailable and to try again.  Most transactions were going thru, but races may want […]

Security Patch for vulnerability in bash “Shellshock” Updated

We have updated our public facing servers with the patch for this new security vulnerability that was found and reported on yesterday. Here are is a link to learn more if you are interested, or to check to make sure your own servers have installed this: http://www.csoonline.com/article/2687265/application-security/remote-exploit-in-bash-cve-2014-6271.html Note there are likely more patches coming from the […]

Partial Outage

In our ongoing effort to report all availability issues, we experienced a partial issue this morning from about 5:30AM until 8:30AM Eastern.  Due to our high level of redundancy, we processed transactions throughout and user would have only had an issue on their initial connection to the site and upon clicking refresh they would have […]

Innovation

Innovation: The act or process of introducing new ideas, devices, or methods. As technology people, we love the opportunity to continually innovate and bring new features to the table…but don’t mistake innovation for instability.  We believe innovative technology should still be reliable technology. Have ideas? Challenge us!

WordPress