48+ days since last incident
Resolved: Shipvine resolved the immediate issue by restarting the effected processes.
The Sync system executes tasks in the background using a Microsoft Azure service. The Microsoft Azure service experience DNS name resolution errors which caused the runtime to fail in a nonrecoverable way that did not trigger our alerting systems. We’ve updated our alerting to ping from an external source which will notify us should this error occur in the future, and we’ve reached out to Microsoft for more details about this particular circumstance. Shipvine transitioned the Sync system to this architecture over 14 months ago, so the issue is thankfully rare, but we do want to make sure we are always informed about the status of our systems. We apologie for the delays.
updated 02 Oct 2018 13:28 UTC
At 9:45 p.m., the Sync queue stopped processing, causing delays for orders to appear in Shipvine Logistics. The queue is recovering and Shipvine is working to determine the cause of the failure.
Degraded Performance effecting Sync
posted 02 Oct 18 12:16 UTC
Forecast Update: The forecast for Virginia has improved dramatically since Tuesday, with Hurricane Florence taking a southernly track. While Richmond’s power grid always affords a risk of a power outage during any weather event, it’s looking more likely that it’ll be business as usual at Shipvine as we enter the weekend.
updated 13 Sep 2018 11:20 UTC
Richmond and the rest of the Commonwealth of Virginia are currently under a state of emergency as Hurricane Florence heads toward us. As always, the safety of our employees is our highest priority. Unless the predicted path of the hurricane changes, we expect to have an emergency closing on Friday, September 14th and possibly an early closing on Thursday, September 13th. We will make every attempt to process orders for as long as possible. Shipvine is not in a flood plain; however, we expect widespread power outages and network outages that could impact all shipping operations for an extended period at our facility and shipping carriers. Check back here for the most up-to-date information.
Degraded Performance effecting Logistics
posted 11 Sep 18 11:29 UTC
Complete: We’ve run through our daily routines, and performance is looking good! We’ll continue to keep a close on eye on things as we progress through next week.
updated 19 May 2018 16:36 UTC
Monitoring: Shipvine Logistics is back online after our database maintenance. We’ll be monitoring performance of the new installation.
updated 19 May 2018 14:07 UTC
Maintenance Continues: We’re cleaning up a few things and verifying data integrity before turning things back on.
updated 19 May 2018 13:05 UTC
Maintenance Continues: The database migration is about halfway there.
updated 19 May 2018 12:45 UTC
Maintenance Started: We’ve started the database maintenance. The Shipvine Logistics website, API, and queue processing will be offline for several hours.
updated 19 May 2018 11:46 UTC
On Saturday, May 19th, the Shipvine Logistics website, API, and queue processing will be taken offline for up to 3 hours as we perform maintenance on the Logistics database. While Shipvine is able to perform most IT maintenance without user impact, the nature of this upgrade requires us to temporarily take the Logistics database offline as we migrate it to a new, upgraded platform at a new data center. The purpose of this maintenance is to improve resiliency and disaster recovery scenarios on a critical IT asset. Other Shipvine systems, such as Labels and Sync, were migrated to this upgraded platform in 2017, and the migration of the Logistics database will complete this work. Thank you for your patience.
Maintenance effecting Logistics
posted 16 May 18 11:57 UTC