The network has remained stable, and Shipvine has resolved the backlog for nearly all merchants serviced from RIC2. The entire backlog will be resolved Friday morning. We will prioritize work to improve our failover capabilities at RIC2.
Posted May 25, 2023 - 16:15 EDT
Monitoring
The network has been restored and stable for the past hour, and Shipvine is working to address the backlog of fulfillment requests at RIC2.
Posted May 25, 2023 - 14:21 EDT
Update
The outage is caused by a failed piece of telco equipment on Richmond Hwy and effects multiple buildings. Shipvine is continuing to manually pick fulfillment requests at RIC2 and transport them to RIC1 for processing. Though these fulfillment requests are "picked" at RIC2, they will not appear as such until they are processed at RIC1. Please note that all of our server infrastructure is hosted at a data center, so the ingestion of orders and transmission of tracking numbers is not effected by the outage.
While RIC1 has redundant ISP connections, RIC2 does not, and we will of course be prioritizing this work with additional failover options next week.
Thank you for your patience as we continue to process fulfillment requests this afternoon.
Posted May 25, 2023 - 12:47 EDT
Identified
An extended network outage is effecting the region surrounding the RIC2 (N Hopkins Rd) warehouse, and we can't fulfill orders from that location. Technicians from the local ISP are actively working on the problem. Shipvine is implementing emergency measures by picking orders at this location using paper tickets and shuttling items back to RIC1 for processing and shipping, which is substantially slower but still allowing us to fulfill orders. We are prioritizing hot and SLA orders first.