UPDATE: We have made two changes to our system to try to improve this situation if it happens again. First, we have adjusted the settings on our New Relic Monitoring tool to try to catch this type of error so we can respond more quickly than hearing about errors from users. Second, we have added a new queue that will save data before sending the transaction for payment. This will allow us to recover transactions more easily in the event something like this happens again.
One of our servers had a hardware failure on Sunday afternoon. Unfortunately, this created a situation where the credit card transactions were sent and completed, but the registration was not completed in the system. There were a total of 363 transactions with this error out of 6.506 total transactions that completed yesterday.
We are processing refunds for all of those transactions this morning, although people may not see them for a day or two depending on their credit card company.
The error people saw when using RunSignUp was either an error when trying to log into their account, or when they clicked on the final page when they entered their credit card number, the confirmation page never showed. If a participant got a confirmation page or email, or you see them in your Race Director Participant Report, then their transaction completed successfully.
We apologize for this error. Our monitoring system, New Relic, was supposed to pick up errors like this, but did not. We are doing further investigation, and will be adding some extra monitoring and logging to try to catch errors like this much earlier and automate backup processes. We will report on the final changes we are making to the system when they are implemented.