Summary:
ITS is aware of and working on Sakai performance issues. Issues have appeared sporadically since the afternoon of Sunday, April 26.
We apologize for the inconvenience that this is causing our students and faculty. Our staff is working hard to address issues. We are working directly with our database provider to develop a solution as quickly as possible.
Updates will be posted here and timestamped as they are available. We will also post updates to Twitter, @UNCTarHeelTech.
Log of Service Disruptions
| Date | Time/Length of Disruption | Type of Disruption |
|---|---|---|
| 4/26/2015 | 2:30 – 9:48 PM | Degraded performance |
| 4/26/2015 | 9:48 – 10:30 PM | Total outage |
| 4/27/2015 | 11:00 AM – 12:28 PM | Degraded performance |
| 4/27/2015 | 12:28 – 12:46 PM | Total outage |
| 4/28/2015 | 12:00 – 1:04 AM | Degraded performance |
| 4/28/2015 | 1:04 – 1:24 AM | Total outage |
**During periods of degraded performance, actions/functions in Sakai varied by action/function between sluggish and non-responsive.**
Student, faculty and staff who experience issues are strongly encouraged to call 962-HELP to report problems.
Communications Log
[Update 4/28 – 1:45 AM – Web/Social]
Around midnight Sakai began experiencing performance issues, most noticeably slow page load times. The system was restarted at 1:04AM and page times have now returned to normal. We are closely monitoring performance and may reboot the system as needed.
[Update 4/27 – 3:30 PM – Formal notice to students]
Students,
As you all are well aware, yesterday Sakai began experiencing significant performance issues, including downtime during the evening. We restored Sakai service late Sunday evening, and ITS has been closely monitoring performance since. If you have any issues, please report them to 962-HELP. This is the single most important tool we have for hearing user problems and identifying trends early.
–What we are doing now–
Sakai experienced intermittent slow page load times at 11:10AM today. In an effort to mitigate the impact, ITS took the system offline at 12:28PM, applied additional technical changes and returned the system to service at 12:46PM. This has helped resolve some issues. To maintain the reliability of Sakai during the next two weeks we will be performing continued configuration adjustments. We expect to take maintenance windows for Sakai daily between 3:00-4:00AM. We ask that you adjust your usage as Sakai may be unavailable during this time.
–What happened yesterday–
On Sunday afternoon at around 2:30PM, Sakai began experiencing significant performance issues. ITS immediately identified and planned a response. As part of that activity, ITS consulted with Oracle Enterprise Support administrators, our database support vendor, and opened a critical support ticket at approximately 6:00PM. We applied configuration changes at approximately 9:46PM, and Sakai returned to an operational state by 10:30PM.
We apologize for the significant stress and anxiety that these Sakai outages have caused for you. I assure you that ITS has done and continues to do everything possible to repair Sakai. I encourage you to follow us on Twitter (@UNCTarHeelTech) for the most up-to-date information regarding Sakai and other IT announcements.
Sincerely,
Chris Kielt
Vice Chancellor for Information Technology & CIO
[Update 4/27 – 3:35 PM – Formal notice to faculty/staff]
Faculty and staff,
Last night after making configuration changes and restarting the system, Sakai returned to regular operation. These changes were made based on an ITS review of issues, as well as engaging emergency assistance from Oracle, our database support vendor.
Around 11:00AM today, we observed that the system was again lagging. We felt an important and immediate step to take was to make additional configuration changes and once again restart the system, which resulted in an outage from 12:28-12:46PM this afternoon. We had to make the difficult decision to impact those students who were taking exams to solve the issue for the larger population. As a result, we are seeing significantly improved performance. We are encouraging any students, staff or faculty who experience issues with Sakai to report them by calling 962-HELP. This is the single most important tool we have for hearing user problems and identifying trends early.
We believe that much of the reason for the system contention has grown out of end-of-semester activity. I know that this has been disruptive and understand the importance of having Sakai up and running, especially at this critical time of year. I have engaged all members of our team and our vendor resources to resolve the issue as quickly as possible.
We will continue to update the campus community as information becomes available. Regular updates are being posted to the ITS website (its.unc.edu).
Sincerely,
Chris Kielt
Vice Chancellor for Information Technology & CIO
[Update 4/27, 12:30AM – Formal Notice sent to faculty and staff] Faculty and staff, On Sunday afternoon at around 2:30PM Sakai began experiencing significant performance issues with intermittent outages. ITS began work shortly thereafter to address and fix the issue. Issues appear to be resolved as of Sunday evening at 10:45PM. We will continue to monitor closely through the night and Monday. ITS consulted with top system administrators from Oracle, our database vendor. We opened a critical support ticket at approximately 6:00PM and although identifying issues was difficult given the volume and diversity of activity on Sakai, we were able to determine possible solutions. With configuration changes in place at approximately 10:00PM, we restarted the system. As of 12:30AM Monday morning, Sakai remains operational. We are acutely aware that the timing of this unplanned outage is an inconvenient and stressful experience for both our faculty and students. We are deeply sorry for the disruption. We take such events with the utmost seriousness and will continue to monitor these circumstances. Regards,
Chris Kielt
Vice Chancellor for Information Technology & CIO