We have completed the upgrade to our Product Logs database, and product logs are now functioning correctly. Changes from orders placed from the start of the issue now appear on the new logs, but price & qty changes through the inventory page during the window will not be included in the logs.
Posted Aug 11, 2023 - 16:19 PDT
Update
We deployed an update that will allow changes made on the inventory page (e.g. price & qty updates) to save. However, these changes will not show up in the activity logs. This is to restore functionality while we continue to work on the long-term fix.
The long-term fix is an upgrade to our database. We anticipate that will be completed tomorrow (Thursday, Aug 10), at which point activity logs will save correctly. Changes from orders placed from now until the upgrade is complete appear on the new logs, but price & qty changes through the inventory page during this window will not be in the logs.
Posted Aug 10, 2023 - 00:28 PDT
Update
UPDATE: Orders can be placed successfully, both through the POS and frontend site. We are also seeing orders from 3rd party sites come in (eBay, TCGPlayer, etc). They will update inventory properly, but won't show in the activity logs currently.
Activity logs are still down, and manual inventory adjustments still cannot be saved. Our plan is to update the activity logs with the order history once we have the underlying issue fixed.
Posted Aug 09, 2023 - 13:12 PDT
Identified
Inventory changes are still not updating. We are working on a stopgap solution, which will be to temporarily disable the activity logs, as well as a permanent fix.
Posted Aug 09, 2023 - 12:33 PDT
Update
We are diving deeper into the issue and working on a permanent fix. We may experience interruptions while working on this.
Posted Aug 09, 2023 - 10:23 PDT
Update
IMPORTANT: I'm still seeing issues with line items not saving properly in the POS. Following up with our developers, but it seems the issue may not be completely resolved yet. I'd still recommend not using the POS or keeping track of the line items separately.
Posted Aug 09, 2023 - 09:55 PDT
Monitoring
We fixed the issue, and inventory is saving again. I'll work on getting more details from our engineering team for a further update. Right now, they are still working on diagnosing what caused the issue in the first place. Please contact support to let us know if you do still see any issues.
Posted Aug 09, 2023 - 09:28 PDT
Investigating
The main problem we are seeing is that inventory changes cannot be saved. We recommend either not using the POS, or at least make sure to keep track separately of any items on orders placed through the POS, because line items will fail to save onto the orders.
This is also preventing any website orders from being placed successfully, so we recommend disabling checkout until we have the problem resolved.
We apologize for the trouble this issue is causing, and thank you for your patience. We'll update as soon as we have more info or an ETA.
Posted Aug 09, 2023 - 09:11 PDT
This incident affected: Admin, POS, Frontend, and Batch Update.