404 errors in app admin, storefront forms not loading

Incident Report for Helium Development

Resolved

This incident has been fully resolved. We will continue to monitor for any further issues.

The cause of the outage was due to an unexpected conflict with the IP address used by the app's load balancer. Provisioning a new IP fixed the problem.

The dev team is working on an internal post-mortem and action plan to prevent this type of incident from occurring again. Please contact our support team (support@heliumdev.com) with any questions or concerns.
Posted Sep 22, 2023 - 16:46 PDT

Monitoring

The SSL issue has been resolved and all parts of the app are now operational. We are continuing to monitor for any further issues.
Posted Sep 22, 2023 - 16:04 PDT

Identified

The app is back online after updating the global IP, but there is still a partial outage caused by SSL certificates. The team is still all-hands-on-deck and we'll provide more updates shortly
Posted Sep 22, 2023 - 15:45 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 22, 2023 - 15:27 PDT

Identified

The issue has been identified and a fix is being implemented.
Posted Sep 22, 2023 - 14:29 PDT

Update

We are continuing to investigate the issue.

Errors first started to appear around 13:35 PDT. This outage is of utmost importance and all engineers are actively working to address the problem
Posted Sep 22, 2023 - 14:21 PDT

Investigating

We are currently investigating this issue.
Posted Sep 22, 2023 - 14:14 PDT
This incident affected: Customer Fields app for Shopify (Customer Fields: Admin UI, Customer Fields: Store embeds, Customer Fields: API, Customer Fields: Shopify webhooks, Customer Fields: Email Notifications).