The Carolina CloudApps team updated its underlying software, Red Hat’s OpenShift Container Platform, from version 3 to version 4 in April.
“During the week of April 19, we successfully migrated 1,613 projects with minimal downtime and minimal issues,” said Patrick Casey, Director of ITS Middleware Services and Linux Administration. “Numerous high visibility and mission critical applications reside on the Carolina CloudApps platform, and the Middleware/Linux team did everything they could to perform this migration as seamlessly as possible.”
To fully understand everything entailed in this migration, we’d need one heck of a long story! But we’ll cover the highlights.
First, OpenShift version 4 runs on a new operating system, Red Hat CoreOS. “So, we had to familiarize ourselves with the new offering and work to fit this within our existing o/s support structure,” Casey said.
CoreOS is an immutable operating system that enables improved processes such as over-the-air updates for patching that operating system.
Changed underlying storage
Secondly, the Middleware/Linux team replaced the traditional host/path-based NFS mounts with a new storage platform in NetApp Trident.
“This change in underlying storage brought major improvements to how we implement persistent storage volumes,” Casey said. “The outcome is that we can now right size storage and grow volumes without requiring customers to migrate data between host/path mounts.”
In both of these updates, the Middleware/Linux team worked with Matthew Conley’s Systems Infrastructure team members, who were instrumental in achieving success in both of these areas.
Thirdly, the underlying technologies used by OpenShift changed significantly, including the container runtime environment moving from Docker to CRIO, major updates to Kubernetes, and major user interface changes to the platform.
Four days vs. a year
Lastly, there was the migration itself. To fully understand the migration, it helps to think about the last major OpenShift upgrade.
“As with our recent upgrade, the version 2 to version 3 upgrade also involved migrating applications,” Casey said. “To that end, manual migrations were done, project by project, and it took nearly 12 months using two team members to migrate all project workloads. Red Hat had heard from many customers, including UNC, that this was an untenable solution and they engaged key customers (again, including UNC) on requirements for a migration tool.”
To that end, Red Hat released a migration toolkit. However, that tool is written in some ways that are not achievable for user bases in which every user is not a professional developer.
“So, we enhanced the migration process by adding pre- and post-processing to the migration that allowed us to migrate storage from NFS to Trident on the fly, verify before/after statuses of applications as they migrated, send communications to the project administrators, allow application workloads to run in version 3 and version 4 simultaneously, update the routing to running applications on the fly via scripting, and then finally have the full application migration complete to version 4,” Casey said. “Of the 1,613 projects migrated over four days, only a single project had downtime due to the migration in excess of two hours.”
‘Helpful every step of the way’
Jason Whitley of the UNC Eshelman School of Pharmacy IT appreciated how the CloudApps team handled the migration. The team, he said, responded quickly to his questions throughout the process. Since 2017, the MEDIA team at the School of Pharmacy has migrated more than two dozen educational applications to UNC CloudApps.
“The CloudApps team has always been more than helpful every step of the way, helping us overcome any number of hurdles to make the migration as easy as possible on our team’s small number of developers,” Whitley said. “The latest upgrade is no exception. The CloudApps team prepared us well in advance to update any potentially breaking changes and then seamlessly performed the upgrade. They quickly solved any small remaining issues after the upgrade. I personally appreciate their approach of teaching users how to solve issues on their own.”
CloudApps, he said, “is perhaps the only ITS-supported platform that developers can use to work in almost any coding language on live servers at UNC. Performing the upgrade almost flawlessly across the entire University is a terrific accomplishment. They do a fantastic job.”
Biology Professor Allen Hurlbert, meanwhile, has two separate applications hosted on CloudApps.
“The Caterpillars Count! migration went without a hitch,” he said. “The Avian Diet Database website went down briefly, but only because the migration revealed some inappropriately specified static IP addresses on our end and an out-of-memory error. The support staff — especially Stephen Braswell and William Mowery — were on it immediately when I raised the issue, and not only got the site back up in no time, but provided useful tips for future monitoring and fiddling in the new CloudApps environment.”
Teamwork key to ‘monumental success’
Each member of the Middleware/Linux team provided valuable skills, insight and implemented various required portions of the project, Casey said. The full project team was made up of Braswell, Mowery, Chris Conklin, Jonathan Towne and Bain Heffner.
“The tireless dedication of the Middleware/Linux staff to achieving this monumental success cannot be expressed enough,” Casey said. “A special thanks should also be made to Stephen and William for their tireless efforts on those amazing four days of the migration as they worked through any issues that arose during in-flight migrations. I couldn’t be prouder of the team for their triumphant upgrade/migration efforts.”