48+ days since last incident
Analysis: The API Submissions queue is designed to be fault tolerant, moving submissions with malformed data into the candidate requests listing if there are any problems processing. In this instance, an address validation component successfully validated a particular API Submission, but a legacy database component that translates the ISO 3166-1 alpha-2 country codes into internal identifiers was missing the country code for Turkmenistan. This caused processing to fail in an unexpected way, and the queue was stuck attempting to save this fulfillment request.
No submissions were lost, just their acceptance into the system was delayed during this time period. All fulfillment requests will ship out on time today.
To mitigate this problem going forward, we’ve reassessed the mapping between these two components and confirmed that there are no mismatched values. While these kinds of incidents have been extremely rare over the past nine years, improved poison message handling is already underway in a planned upgrade to this codebase, expected to be released in Q2 2018.
We apologize for any confusion and inconvenience this incdient may have caused.
updated 20 Feb 2018 12:22 UTC
Resolved: The queue is now processing requests normally and should be caught up within 15 minutes.
updated 20 Feb 2018 11:47 UTC
There is a delay in processing the API Submissions queue.
Degraded Performance effecting Logistics
posted 20 Feb 18 05:05 UTC