Yesterday, February 17, 2020, was a big day. We processed over $3.4 Million across over 50,000 registrations. There was the big 30,000 lottery drawing for the Blue Cross Broad Street Run. Also the 1,400 person drawing for the Horribly Hilly Hundreds. Oh, and there were two sellout races with the Race To Robie Creek selling out 2,400 in less than 10 minutes and Guinness Open Brewery Pub Run selling out 1,300 also in minutes. (And double oh, we processed transactions for over 2,900 other races and nonprofits yesterday as well!)
It was cool to see different approaches to handling sell out events, as well as offered us a chance to fine tune our system since we had never done a lottery pick as large as Broad Street. Let’s look at each a bit more, with a deep dive into how we process lottery at scale with Broad Street and what fine tuning we did.
Fast Sell Outs
This is a really fun way to build excitement for some races. Do some Googling and Youtube searches on the Race to Robie Creek to see a beautiful, hugely challenging and fun race. It has attracted a following not only in Boise, but from around the country as one of the toughest half marathons. They’ve been on RunSignup for 5 years. Their peak yesterday was about 450 registrations per minute right after they opened at Noon in Boise. They have a waiting list and will pick some from that waiting list, as well as allow bib transfers on RunSignup for a $10 charge.
The Guinness Open Brewery Pub Run has two stages. Early registration is open to 1,500. Then they opened it up to everyone else yesterday and added another 1,300 in a few minutes starting at 6PM Eastern.
Lottery
The Horribly Hilly Hundred is a cycling event – with challenging distances of 100K, 150K and 200K and elevation change of over 10,000 feet (who knew there were that many hills in Wisconsin!). It has become very popular and they instituted a lottery based system rather than a one time fast sell out. This does give everyone a chance to get in. It also allows race organizers see what the potential demand is and see if they can add capacity to their race. For example, a 200K was added to help expand the field so they could increase from 1,200 to 1,400 cyclists.
We have been helping this race for several years, and the lottery picking has become routine. They pick via a spreadsheet that is uploaded. Our system takes all the selected participants and processing their credit cards that were previously authorized when they signed up for the lottery. When the credit card transaction is successful, we send them a confirmation email. When their credit card does not process (expired card, or lost/stolen, or perhaps has hit a limit on that account), we send an email automatically to ask the participant to re-enter their card.
As of this morning, there are only 14 people of the 1,400 where the credit card has not processed (1%). These 14 are are available in a report, or can be emailed again to ask for their credit cards.
The people who were not selected are also available in a report. These people can be emailed with a customized email from the race to let them know the disappointing news. We recommend this be done within an hour or two of doing the lottery selection.
Broad Street
Broad Street is the nation’s largest 10 mile race held in Philadelphia each spring. They have several guaranteed entry categories – people who signed up for the “Philly Challenge” (both the Philadelphia Marathon and Broad Street), 10+ year runners, and deferrals from last year. These runners have been able to sign up via our loyalty system that help identify who is qualified when they register for the past couple of weeks.
One of the cool things we added to the lottery was the ability to make a donation at the end of signing up for the lottery. This was done by adding a little pop-up encouraging a donation to the charity partner, the American Cancer Society. This results in over $28,000 of donations that have never been collected before.
The lottery has also been open since February 1. Some of the numbers are interesting to see how the credit card processing works and how many bad credit cards happen, and how quickly and how many cards are corrected with the same automated process used as described above for the Horribly Hilly ride.
- 28,391 lottery selections
- 27,333 successful credit card charges
- 1,058 unsuccessful charges (3.65%) – with automated notifications to participants to correct them
- 198 cards corrected within 3 hours
- 394 cards corrected as of 9AM the next day
- 20 hours after selected 2.3% failure rate with correction still needed
Since we were selecting 28,391 lottery entries to be admitted yesterday, we wanted to do some monitoring and testing to optimize lottery selection. We broke the entries into waves of roughly 1,000, 2,000, 3,000, 5,000, 7,000, and 10,000. This gave us enough volume on each run to be able to monitor the queues and the back end transactions as well as system performance.
When a spreadsheet is uploaded to RunSignup, we break it into “payloads” of 100, and put those payloads into the Amazon AWS SQS queuing service. SQS works exactly like you would think – basically keeps each payload in order kind of like a train with cars. We have 4 webservers that process all the website requests as well as these payloads. Each webserver is set up to take X number of payloads off of the SQS Queue at a time. We process these queues every 15 minutes. So if you upload your spreadsheet(s) for selection at say 11:50AM, it will start processing at noon. If you add more spreadsheets at 12:05, they will simply be added to the queue.
Initially, we had each server set up to process 10 payloads at a time. That meant we were processing 40 payloads at a time across the 4 servers. When we ran with this setting, we were processing 1,977 transactions per minute – about 33 per second. This was a combination of the 40 transactions from the lottery mixed in with the regular registrations and donations that were happening on the website in real time.
As can be seen by the above graph, webserver performance stayed very responsive at under 0.18 seconds during this whole process in spite of the settings. We also monitored the queues and the back end transactions to our payment processor, WorldPay.
You can see the spikes at 12:45 were much higher than the ones starting around 1PM. This was because we changed the queue processors on the webservers to handle 5 payloads at a time rather than 10. This configuration allows for a steadier volume, and accommodates the risk of a sell out race happening at the same time. It lowers the “steady state” of lottery pick transactions to closer to 20 per second, and since the systems are designed to handle 60+ transactions per second it allows for spikes in demand if for example Broad Street were picking from their lottery and the Race to Robie Creek opened at the same time.
Our goal was to leave the system tuned so that it would work optimally without any intervention from humans or need for fine tuning. This means that we can process 30,000 lottery selections without any human intervention in 25 minutes while still being able to handle a 40,000 person sell out race happening in 10 minutes on our normal infrastructure.
While this might seem like overkill, it does prepare us for the future. If we look down the road in 7 years, we expect RunSignup to process roughly double what we do today, and GiveSignup to process a similar amount – so over $1 Billion per year. Which means our average day will be over $2.7 Million – not much different than what we processed yesterday.
