tag:goassemble.statuspage.io,2005:/historyAssemble Status - Incident History2024-03-08T18:12:16+00:00Assembletag:goassemble.statuspage.io,2005:Incident/173661812023-05-25T18:09:19+01:002023-05-25T18:09:19+01:00Intermittent performance issues<p><small>May <var data-var='date'>25</var>, <var data-var='time'>18:09</var> BST</small><br><strong>Resolved</strong> - This has now been resolved</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>17:00</var> BST</small><br><strong>Monitoring</strong> - Due to an increased number of connections we were hitting high CPU usage on database cluster. We have lowered the number of internally made connections and CPU usage has returned to normal.<br /><br />We will continue to monitor this.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>16:15</var> BST</small><br><strong>Investigating</strong> - We are aware of some intermittent performance issues with Assemble. Our engineers are actively working on this.</p>tag:goassemble.statuspage.io,2005:Incident/144068282022-12-03T02:00:00+00:002022-12-05T12:05:10+00:00Degraded Performance<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>02:00</var> GMT</small><br><strong>Resolved</strong> - A badly configured third-party tool was making a large amount of heavy requests which caused the Assemble systems to scale up to their maximum levels. Whilst this tool was running and before it was blocked, some requests may have failed.</p>tag:goassemble.statuspage.io,2005:Incident/132801112022-11-17T12:50:00+00:002022-11-17T15:17:07+00:00Assemble outage<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>12:50</var> GMT</small><br><strong>Resolved</strong> - Assemble API is down</p>tag:goassemble.statuspage.io,2005:Incident/99102712022-05-04T10:30:29+01:002022-05-04T10:47:50+01:00Email issues<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>10:30</var> BST</small><br><strong>Resolved</strong> - The issue has been resolved and emails (including 2FA) should now be working as expected</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>09:58</var> BST</small><br><strong>Investigating</strong> - Assemble is currently experiencing an issue with emails. Our team is actively looking into this and we will be providing a fix as soon as we can. <br /><br />Apologies for any inconvenience this may cause.</p>tag:goassemble.statuspage.io,2005:Incident/92559592022-03-05T02:33:28+00:002022-03-05T02:33:28+00:00Assemble Maintenance<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>02:33</var> GMT</small><br><strong>Completed</strong> - The maintenance work has been completed and Assemble is now fully available.<br /><br />Thank you for your patience through this maintenance period.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>01:00</var> GMT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>00:59</var> GMT</small><br><strong>Update</strong> - All data migration has completed successfully and passed automated tests. Manual testing is underway. All in order so far.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>22:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>12:14</var> GMT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:goassemble.statuspage.io,2005:Incident/86602402021-11-22T13:01:35+00:002021-11-22T13:01:35+00:00Unusually high and slow email queues.<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>13:01</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>12:39</var> GMT</small><br><strong>Update</strong> - The queue has now been fully processed and the additional resources will stay online until the root cause for the backlog is found. Emails should be delivered instantly now.<br /><br />If you had previously requested your authentication code to log in, you can now successfully request this again.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>12:20</var> GMT</small><br><strong>Update</strong> - Extra resources have been brought online and the queues are going down faster now.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>12:00</var> GMT</small><br><strong>Investigating</strong> - We are currently experiencing unusually high and slow email queues. The team are now looking into this and hope to bring services back to normal as soon as we can.</p>tag:goassemble.statuspage.io,2005:Incident/69646802021-05-10T17:36:21+01:002021-05-10T17:44:43+01:00Drops in performance<p><small>May <var data-var='date'>10</var>, <var data-var='time'>17:36</var> BST</small><br><strong>Resolved</strong> - Thank you for your continued patience while we investigated the dips in performance, services have now returned to normal.<br /><br />The issue was identified to be one of our upstream providers who have fixed the issue. Once we receive a full report from them, we will be looking at ways to mitigate against them being a bottleneck in the future.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>16:45</var> BST</small><br><strong>Update</strong> - We have identified the source of the issue with one of our suppliers and currently working with them to resolve the issue. <br /><br />Thank you again for your patience during this time.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>16:02</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>15:21</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue and our resource upgrade is in the process of deploying.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>14:30</var> BST</small><br><strong>Update</strong> - We have now increased our resources and performance should shortly return to normal.<br /><br />Again we apologies for any inconveniences caused.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>14:00</var> BST</small><br><strong>Update</strong> - We will be temporarily increasing our resources to improve performance levels while we continue to investigate the cause of the issue.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>13:30</var> BST</small><br><strong>Investigating</strong> - We are currently experiencing some temperamental drops in performance. Our team are investigating and will be bringing services back to normal as soon as we can.<br /><br />Apologies if this has affected you in any way.</p>tag:goassemble.statuspage.io,2005:Incident/66863262021-04-06T14:21:31+01:002021-07-06T11:37:42+01:00Degraded performance<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>14:21</var> BST</small><br><strong>Resolved</strong> - Having monitored performance over the last hour, we believe services to be running smoothly again. As always we will continue to keep an eye on things.<br /><br />Apologies if this incident has affected you in any way.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>13:13</var> BST</small><br><strong>Monitoring</strong> - The configuration changes have been implemented and services should have returned to normal. We will continue to monitor this.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>12:38</var> BST</small><br><strong>Update</strong> - We are currently in the process of making some configuration changes in an attempt to reduce the load and improve performance, You may experience a short downtime as this change ticks over.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>12:03</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue. This is currently affecting user login and general navigation around Assemble.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>11:00</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:goassemble.statuspage.io,2005:Incident/64428482021-03-03T20:39:57+00:002021-03-04T12:10:11+00:00Data Processing Delays<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>20:39</var> GMT</small><br><strong>Resolved</strong> - All systems have been monitored for a few hours and are working as expected.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>14:43</var> GMT</small><br><strong>Identified</strong> - The root cause of the additional load on the infrastructure has been identified and a patch is currently being tested on our pre-production environment.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>13:28</var> GMT</small><br><strong>Monitoring</strong> - We have now increased our resources and made the config change. Services should be running as normal but we will continue to monitor things.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>12:44</var> GMT</small><br><strong>Update</strong> - We will be increasing resources and making a config change to help us overcome the performance issue. This will mean a downtime of a couple of minutes as we switch resources. This should happen within the next 30 minutes.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>10:30</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>09:30</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>08:30</var> GMT</small><br><strong>Investigating</strong> - Our data processing infrastructure is running behind which is causing slower load times and timeouts. Our developers are investigating the issue and the system should be caught up shortly.</p>tag:goassemble.statuspage.io,2005:Incident/64426712021-03-03T07:00:52+00:002021-03-03T09:22:23+00:00Assemble - Login is Down<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>07:00</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.<br /><br />Down since: March 3, 2021 7:05 AM GMT<br />Available since: March 3, 2021 7:11 AM GMT<br />Down Duration: 11 Mins 29 Secs</p>tag:goassemble.statuspage.io,2005:Incident/64354382021-03-01T15:00:00+00:002021-03-03T09:55:12+00:00Assemble down - 01/03/2021<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>15:00</var> GMT</small><br><strong>Resolved</strong> - Down since: March 1, 2021 2:47 PM GMT<br />Available since: March 1, 2021 2:57 PM GMT<br />Down Duration: 8 Mins 27 Secs</p>tag:goassemble.statuspage.io,2005:Incident/16287542018-03-08T20:42:05+00:002018-03-08T20:42:05+00:00Scheduled Maintenance<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>20:42</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>19:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>18:30</var> GMT</small><br><strong>Scheduled</strong> - Scheduled Maintenance</p>tag:goassemble.statuspage.io,2005:Incident/16000192018-02-12T18:00:00+00:002018-02-12T21:09:35+00:00Elevated API Errors<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>18:00</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.
<br />
<br />Our message queue got very large very quickly and was causing memory issues on one of our instances. Additional instances were brought online to clear down the queue and the faulty instance removed. Additional capacity has been provisioned to handle such an incident.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>17:51</var> GMT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:goassemble.statuspage.io,2005:Incident/15604992018-01-10T02:06:00+00:002018-01-10T10:01:11+00:00Elevated API Errors<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>02:06</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.
<br />
<br />The cache infrastructure has been restarted and API servers brought back up.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>01:40</var> GMT</small><br><strong>Investigating</strong> - A problem with our caching and queuing database is causing errors in the API servers.</p>