FreeIPA upgrade - Post Mortem

My homelab runs FreeIPA as a source of identity. It’s the internal source of truth on that, and it allows me to have centralised authentication. That’s what it does. I don’t have to automate SSH keys, I don’t have to automate some certificate tracking - even though this one will change in the future. Well, that was until May 2026, when I’ve decided that I wanted to have FreeIPA running on some CentOS 10 Stream nodes.

As a bit of context, I had FreeIPA running off some CentOS 8 Stream nodes. They’re EOL, have been for a while, and this being homelab, a home infrastructure, I never got around to actually doint the migration, only thinking about it. I decided to get on with it, and thus started the wrestling.

Deploying the new systems

This step was actually pretty straightforward. It’s just a matter of ISO deploy, nothing fancy. Install the software. From here, it got complicated.

The plan was the following :

• Deploy the servers
• Install the applicative
• Run the automation playbook (limited)
• Perform a replica install Replica Setup
• Wait for a bit, then test the replica
• Failover then phase out the old systems

As the title implies, this went hastily off-track and the plan had to be terminated at some point.

Timeline

This incident didn’t actually have a start date, nor a date from which impact started. It’s the result of a problem that compounded over time and never got addressed, resulting in technical debt. It is currently still ongoing but is now controlled. An initial migration had been planned early 2024 before the EOL of CentOS 8, so let’s consider this fictional date as the pivot point.

Root causes

The FreeIPA setup, due to lack of maintenance got in a state that was difficult to migrate to integrate with newer systems.

The first issue was that the KRA certificate had silently expired. Due to lack of monitoring, this one flew under the radar and never got detected until the replica-install process, which requires the KRA to be reachable to deploy another node in the cluster.

Another issue existed in earlier installs, in which the available SID ranges were limited in size. This was not a cause for concern at the moment, but adding systems deployed with newer CentOS 10 FreeIPA was not straightforward as it required availability in the SID range, and the expansion process within CentOS 8 is broken.

Impact

Attempting to deploy new nodes in the cluster would fail, requiring the issue to be fixed before any expansion or migration could be considered.

ipa-replica-install --unattended --add-sids --setup-dns --setup-kra --setup-ca --no-sshd --forwarder 192.168.100.8 --forwarder 192.168.2.200 --domain=home.epicfail.be -w $(cat pass.txt)

Note the –add-sids switch. Per the documentation

--add-sids Add SIDs for existing users and groups as the final step

This required planning around the issue.

Actions taken

For the KRA certificate, this was the process to follow. Initially, it required digging into the directory structure to locate the configuration, or knowing it is tomcat-pki, which can be inferred from ipactl status.

# These need to have the same output, certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'transportCert cert-pki-kra' -a | tail -n +2 | head -n -1 | tr -d '\r\n' grep 'ca.connector.KRA.transportCert' /etc/pki/pki-tomcat/ca/CS.cfg # If not matching, edit CS.cfg to have the certificate above. Then restart IPA ipactl restart

This will fix the above issue, it is a simple one. Don’t forget to verify connectivity, just in case :

ipa-replica-conncheck

The second issue is the DNA range.

The forwarders do not point to other FreeIPA nodes. This will be the object of another blogpost, as this isn’t part of the incident. The FreeIPA servers are still reachable though, so this is not cause for concern. This note exists to make explicit that this is not a problem.

Attempting to do it straight away will fail, FreeIPA requires to have an extended DNA range. 389-DS will evaluate that the range overflows and fail, causing the entire process to fail. Thus, add another range. This time I did from the web UI, from the CLI is as similar process.

Then run the replica-install, it’ll succeed.

Lessons learned

Some incidents will happen fast and hit you straight in the face like a boulder, no question asked. Some are the result of compounded lack of maintenance. The cause of the lack of maintenance here doesn’t matter, and maintenance had to be postponed for imperative reasons, from the initial date. This, however, doesn’t change the fact that by 2024 it was already falling apart due to lack of maintenance, and the issue would probably have happened then. It should have happened earlier.

The lesson learned is the following list of statements :

• Plan early
• Discpline before exciting
• Schedule small windows to evaluate if the tool needs migrating soon or not

Planning early could have prevented the issue from compounding into what it had become. Discipline could have prevented pushing back the issue “until later”. Scheduling earlier an evaluation to migrate out of CentOS 8 would have done the same, as well. Also, note that this is a homelab setting, with a bit of homeprod. I treated it as “not really critical”, and then it compounded into a mess. That was underestimating the issue and its potential impact. This is a reminder that risk should be evaluated with upper and lower boundaries and its potential for deviation should be taken into consideration even for systems that are considered to be less critical.