Shiptheory Slowness

Everything is running as normal this morning. Over the weekend we restored the final piece of infrastructure, we've removed all throttles this morning. Sorry for any inconvenience caused and thank you for your patience and understanding.

This is Mark one of the founders @ Shiptheory and would like to give a personal update on this issue and how seriously we are taking it.

Current Status We realise the degraded speed of Shiptheory is an issue for many customers at present. Orders are being processed automatically and printing is operating. The Shiptheory UI/webpage is a little slower than normal and processing orders maybe a little slower than expected in automatic mode.

Why has this happened? On Wednesday evening at 9pm BST, we took Shiptheory offline to migrate to new server(s) to allow us to infinitely scale our service in the future. We were not doing this for any other reason than future proofing and increasing our capacity. It was a project we have been working with a third party specialist for over 8 months, with a rollout plan and rollback plan.

At 4am BST following the migration, we noticed slower speeds whilst testing but were assured by our consultants it was all ok. By 9am, we were seeing severely degraded performance and began to look at rolling back to our existing infrastructure following our rollback plan.

This required some prep, discussion and checking of systems on both sides, which we did. By 11.30am BST we began rolling things back (whilst maintaining data). From around 1pm BST we were back on our old infrastructure, minus one of our primary databases(which are big and much harder to move at speed).

By 3pm BST our existing infrastructure had cleared the backlog, however we now have a much longer delay between our web application and our database, (we're talking 10ms or so). However at scale, where we deal with 1000s of requests per second, this compounds dramatically. Thus from about 3pm onwards the delay had compounded to and again reduced the speed of our application on our old infrastructure.

What we're doing about it We have limited some aspects of Shiptheory to maintain a constant performance with the strategy to decrease database usage to minimise the compound delay, this includes:

• New Dashboard functionality
• Reduced speed on some of the channels we poll for orders
• Reduced speed we process orders at

We are going to migrate our database back to our old server over the coming few days to be ready for Monday to be back to full speed and minimal interruption.

Shiptheory is a huge application, we communicate with 100s of different APIs, we deal with huge volumes of spikey traffic and we always endeavor to offer the best customer support and experience to our customers. It makes us genuinely sad to provide a degraded service for any period of time and can only give our sincerest apologies and assurances that we are all working extremely hard to rectify this issue.

Our team are currently investigating the delays currently involved with automatic printing, while we investigate this you can still print labels with no delay from within your Shiptheory account via the "Print" and "Print to" commands. If you are not sure how to do this please get in touch and we can show you how to get your labels out.

To alleviate the stress on the backlog of print jobs we are processing, we have cut any sent from before 4PM UK time. This does not affect any shipments coming into Shiptheory nor any bookings made, it will exclusively target the label print attempts from before that time. Any of these labels that you are waiting on from before 4PM will need to be reprinted from your Shiptheory account.

We've now put a fix in place to resolve the slowness. Users should see orders start to catch up and send through as normal. Users may see some previously booked labels from today as unavailable, which is something we're working to resolve. New orders will still come through but may see a delay in printing whilst the backlog is worked through. We're continuing to monitor the situation of course.

Now that we've identified the issue and resolution regarding the slowness, there will be a period of maintenance to make the needed change beginning at 1:30pm today, which will last for approximately 20-30 minutes. Following on from there, orders that have built up should once again start to feed through.

Some users may be experiencing slowness when using Shiptheory currently. This is something we're aware of and are actively working to resolve as soon as possible.