For the first time, ITS has automated vendor quarterly patching for two of the platforms that run core critical business applications for the University. This significant achievement expedites performing security updates and fixes and saves considerable staff time.
In May and June, ITS Infrastructure & Operations (I&O) fully rolled out this automated patching for the Oracle databases and PeopleSoft WebLogic middleware servers. Oracle databases and middleware are the foundational platforms for running the core critical business applications that support the University’s administrative processes such as student registration, finance and human resources processes, and academic initiatives such as Research Computing and NC TraCS.
Furthers automation
“Aligning with the Carolina Next strategic framework, we have built upon our automation journey by identifying opportunities for continuous improvement through automation and innovative thinking,” said Scotia Roopnarine, ITS’ Applications Infrastructure Director.
That automation journey started last summer when ITS began upgrading the University’s Oracle software to a newer version, Oracle 19c. The upgrade monumentally changed the way ITS configures the infrastructure of Carolina’s databases — it introduced automation. The automation ability of Oracle 19c roughly cut in half the time to update PeopleSoft, the software on which ConnectCarolina operates. An update that previously took six months can now be completed in three and a half months. That was the start of the automation journey.
Manual is cumbersome
Now consider this second journey and the work that these quarterly patches require without automation. Every quarter, Oracle Corp. releases a quarterly critical patch update as security updates and fixes to flaws for remediation of security vulnerabilities in the Oracle software. The vendor recommends that customers — in this case, ITS — apply these software patches when they become available to ensure proper security measures and address any known security vulnerabilities. But with the manual method, applying the patches for a database entails many steps, and they must be repeated for each database.
“With roughly 180 databases in the portfolio, this becomes cumbersome very quickly,” Roopnarine said.
Security vulnerabilities, said Vetri Thiagarajan, Architect and Director of Enabling Tools for ITS Enterprise Applications, “are lurking out there in all types of software that we use and typically ensnare us before we can react. That is why software vendors release quarterly patch updates for common vulnerabilities and exposures and encourage their customers to apply these patches as soon as possible.”
So, he added, “when it comes to managing Enterprise Applications software like Oracle and PeopleSoft, a significant part of IT administration is risk mitigation. In the past, we had to manually apply these patches, and that took us somewhere between six months to a year or more in some cases to find the right maintenance window to apply these patches and thereby putting the organization’s data at risk of exposure and exploitation.”
Promptness boosts security
Doing these patches manually, I&O was about eight months behind schedule, Roopnarine said. With the latest cybersecurity breaches and ransomware attacks, applying these security patches in a timely manner is ever more important.
In its strategic roadmap for 2020-2021, I&O identified the automation of these patches as a top priority. Ongoing, immediate, and systematic application of every security patch for all production systems are difficult due to the complexity and size of the implementations at Carolina. I&O tackled this automated patching for the Oracle databases and PeopleSoft WebLogic middleware servers by building on last year’s efforts to simplify the architecture.
I&O “used an incremental approach with small gains to keep the forward momentum,” Roopnarine said.
First, the team built a small testing product to demonstrate the technical viability of the idea. Next, the team rolled out the automation into live environments. That pilot phase for the first database began in the first quarter of this year. The pilot phase for the PeopleSoft WebLogic database occurred in the second quarter. With that success, ITS began full implementation for both. I&O expects to complete the patching later in the summer and early fall.
Significant achievement
“What this automation brings is a completely new way of patching, one where we issue one command that kicks off the process, akin to clicking the ‘install button’ for installing Microsoft or Apple patches on your desktop,” Roopnarine said.
“The automated vendor patching solution,” Thiagarajan said, “helps mitigate the risk since it cuts down the patch application time to around 30-60 minutes for an environment compared to four to six hours while doing it in a manual fashion, thereby allowing us to reduce disruption to business. The best part about automation is that it is more or less a rinse/repeat procedure requiring minor tweaks from time to time to maintain it. ITS has been looking to do this for a while and having now this capability is an added arsenal in our infrastructure and security setup in the constant fight to keep the bad actors at bay.”
This achievement is truly significant, Roopnarine said.
“Look at the services that are enabled by the Oracle databases and PeopleSoft middleware software at UNC — ConnectCarolina, Research Computing, the enterprise data warehouse, Eshelman School of Pharmacy, School of Nursing and others,” he said. “These systems are more secure as a result of this initiative.”