48+ days since last incident
Complete: We’ve run through our daily routines, and performance is looking good! We’ll continue to keep a close on eye on things as we progress through next week.
updated 19 May 2018 16:36 UTC
Monitoring: Shipvine Logistics is back online after our database maintenance. We’ll be monitoring performance of the new installation.
updated 19 May 2018 14:07 UTC
Maintenance Continues: We’re cleaning up a few things and verifying data integrity before turning things back on.
updated 19 May 2018 13:05 UTC
Maintenance Continues: The database migration is about halfway there.
updated 19 May 2018 12:45 UTC
Maintenance Started: We’ve started the database maintenance. The Shipvine Logistics website, API, and queue processing will be offline for several hours.
updated 19 May 2018 11:46 UTC
On Saturday, May 19th, the Shipvine Logistics website, API, and queue processing will be taken offline for up to 3 hours as we perform maintenance on the Logistics database. While Shipvine is able to perform most IT maintenance without user impact, the nature of this upgrade requires us to temporarily take the Logistics database offline as we migrate it to a new, upgraded platform at a new data center. The purpose of this maintenance is to improve resiliency and disaster recovery scenarios on a critical IT asset. Other Shipvine systems, such as Labels and Sync, were migrated to this upgraded platform in 2017, and the migration of the Logistics database will complete this work. Thank you for your patience.
Maintenance effecting Logistics
posted 16 May 18 11:57 UTC
Analysis: The API Submissions queue is designed to be fault tolerant, moving submissions with malformed data into the candidate requests listing if there are any problems processing. In this instance, an address validation component successfully validated a particular API Submission, but a legacy database component that translates the ISO 3166-1 alpha-2 country codes into internal identifiers was missing the country code for Turkmenistan. This caused processing to fail in an unexpected way, and the queue was stuck attempting to save this fulfillment request.
No submissions were lost, just their acceptance into the system was delayed during this time period. All fulfillment requests will ship out on time today.
To mitigate this problem going forward, we’ve reassessed the mapping between these two components and confirmed that there are no mismatched values. While these kinds of incidents have been extremely rare over the past nine years, improved poison message handling is already underway in a planned upgrade to this codebase, expected to be released in Q2 2018.
We apologize for any confusion and inconvenience this incdient may have caused.
updated 20 Feb 2018 12:22 UTC
Resolved: The queue is now processing requests normally and should be caught up within 15 minutes.
updated 20 Feb 2018 11:47 UTC
There is a delay in processing the API Submissions queue.
Degraded Performance effecting Logistics
posted 20 Feb 18 05:05 UTC