48+ days since last incident
Resolved: Azure indicated that they resolved the error, due to a misconfiguration on their end, as of 10 a.m. EDT. Shipvine logs show impact between 5:30 a.m. and 9:40 a.m. EDT.
updated 13 May 2019 14:31 UTC
We are seeing sporadic name resolution failures from domains hosted on Azure DNS, including shipvine.net and shipvine.com, which may impact operations and access to our services. Microsoft Azure is investigating the issue.
Degraded Performance effecting Accounts, Bills, Labels, Logistics, Sync, Tickets, Web Print
posted 13 May 19 12:02 UTC
Recovered: Shipvine has verified that critical services are functioning normally again, and is closely monitoring the status at Azure.
updated 02 May 2019 21:39 UTC
At approximately 4:15 p.m. EDT, Microsoft Azure began experiencing global networking issues. As most of Shipvine infrastructure is hosted in the cloud, this has caused our services to be mostly unavailable during the past hour. Network connectivity is coming back online, and we’re working to make sure that services are recovering correctly.
Major Outage effecting Accounts, Bills, Labels, Logistics, Sync, Tickets, Web Print
posted 02 May 19 20:16 UTC
Resolved: Shipvine resolved the immediate issue by restarting the effected processes.
The Sync system executes tasks in the background using a Microsoft Azure service. The Microsoft Azure service experience DNS name resolution errors which caused the runtime to fail in a nonrecoverable way that did not trigger our alerting systems. We’ve updated our alerting to ping from an external source which will notify us should this error occur in the future, and we’ve reached out to Microsoft for more details about this particular circumstance. Shipvine transitioned the Sync system to this architecture over 14 months ago, so the issue is thankfully rare, but we do want to make sure we are always informed about the status of our systems. We apologie for the delays.
updated 02 Oct 2018 13:28 UTC
At 9:45 p.m., the Sync queue stopped processing, causing delays for orders to appear in Shipvine Logistics. The queue is recovering and Shipvine is working to determine the cause of the failure.
Degraded Performance effecting Sync
posted 02 Oct 18 12:16 UTC