Issue Summary:
On 07 May at 08:34 AM, the alerting system detected issues with the central DP management service. The TechnologyOne Team took corrective action to restore normal operation of the DP service. During the incident, users experienced delays in processing DP jobs and errors when attempting to submit them.
Root Cause Analysis:
Increased latency and high error rate of DP management service due to overloading of underlying platform components.
Corrective Measures:
Optimised scaling on the system to reduce the load on the DP processing to restore the service.
Preventive Measures:
As part of the ongoing project the following was delivered as part of this incident response:
Changes implemented to the central DP management service to increase isolation and reduce load on underlying platform components.
Further enhancements to the resilience and availability of the DP service have been accelerated.
Posted May 20, 2025 - 16:54 AEST
Resolved
After 1 hour monitoring this incident is now resolved.
We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion.
We apologise for how you and your business may have been affected by this incident.
Posted May 07, 2025 - 17:38 AEST
Monitoring
Our team has verified the implementation of a fix is complete for ANZ and UK Region for all Releases.
We will monitor the logs for the next 1 hour to ensure no further customers are impacted.
Posted May 07, 2025 - 16:33 AEST
Update
We are seeing improvements in the processing of background jobs and are still continuing to work through further stabilisation.
The next update will be provided in 60 minutes, or sooner if new information becomes available.
Posted May 07, 2025 - 15:52 AEST
Update
We are seeing improvements in the processing of background jobs and are continuing to work through further stabilisation.
The next update will be provided in 60 minutes, or sooner if new information becomes available.
Posted May 07, 2025 - 14:49 AEST
Update
The remediation has been successfully implemented, and the priority DP jobs are back to normal.
The background DP jobs backlog is being cleared out in the coming hours.
The next update will be provided in 60 minutes, or sooner if new information becomes available.
Posted May 07, 2025 - 14:10 AEST
Update
The implementation of remediation is in progress.
The next update will be provided in 60 minutes, or sooner if new information becomes available.
Posted May 07, 2025 - 12:52 AEST
Update
We are continuing to work through stabilising Priority and background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.
Posted May 07, 2025 - 11:49 AEST
Identified
Our team has implemented the intended fix. We are working through a confirmation and verification to ensure the fix is complete.
The next update will be provided in 60 minutes, or sooner if new information becomes available.
Posted May 07, 2025 - 10:52 AEST
Investigating
We are investigating an issue impacting DP service for ANZ Region for All Releases.
Impact/Error/How to verify: A subset of customers in ANZ Region for all Releases are experiencing degradation of DP service
Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.
Posted May 07, 2025 - 09:48 AEST
This incident affected: Software as a Service - Australia & New Zealand (Database Services) and Software as a Service - United Kingdom & Europe (Database Services).