QuattorFreeIPA

Why are we interested in FreeIPA ?

SINDES project seems to be abandoned since a few years now, and we didn't manage to keep it working in our deployment system.

In the process of looking for a replacement solution to SINDES, the main question was : how to securely deliver XML profiles to the machines during the installation process ? SINDES was bringing a nice solution to this problem : during the post-reboot, the machine fetches its own certificate signed by the AII server, and it is then able to download its profile through SSL during the ccm-fetch step.

Finally, after a discussion with people involved in the management of the VSC Tier1 at Ghent (April 2014), we decided to test the solution they had developed based on FreeIPA.

FreeIPA provides an easy way to set up a MIT Kerberos realm with many integrated services, like : 389 Directory Server (LDAP), NTP, DNS, Dogtag certificate system, SSSD, etc.

How Quattor and FreeIPA work together ?

The solution developed by people from Ghent is based on aii hooks. These hooks are documented in the manpages :

(on the aii server)
man /usr/share/doc/man/man8/aii-hooks.8.gz

During the "aii-shellfe --configure" step, a host entry is created on the IPA server, and as a result, a one-time password is generated (OTP), and a "ipa-client-install" (with the previous OTP) is added to the post-reboot part of the Kickstart file.

As we wanted to keep SSL to secure the ccm-fetch, we have slightly modified the freeipa.pm so that the following things are done in post-reboot step, just after the ipa-client-install : the machine fetches its own cert and private key, and the CA certificate, and puts everything in a directory /etc/ipa/quattor/certs. This will be described in details below.

FreeIPA server installation

The FreeIPA server was installed on a CentOS 6.5 virtual machine in the private network (wn.iihe.ac.be).

First of all, SELinux was disabled ("sentenforce 0" and modify /etc/sysconfig/selinux so that SELINUX variable is set to "disabled").

Then, you have to enable homedir creation at log in :

service messagebus restart
authconfig --enablemkhomedir --update

We can now launch the ipa server install script :

ipa-server-install

The script will ask you a few simple questions. It's worth mentioning that we didn't use the option "--setup-dns", because we would like to keep the private DNS out of the freeipa control.

Last step is to open some ports by adding these lines to /etc/sysconfig/iptables :

-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 389 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 636 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 88 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 464 -j ACCEPT
-A INPUT -p udp -m udp --dport 88 -j ACCEPT
-A INPUT -p udp -m udp --dport 464 -j ACCEPT
-A INPUT -p udp -m udp --dport 123 -j ACCEPT

and then restart iptables to apply these new rules :

service iptables restart

Things to do on the AII server

The AII server must join the newly created Kerberos realm :

yum install ipa-client oddjob-mkhomedir
service messagebus start
authconfig --enablemkhomedir --update
ipa-client-install --server=freeipa.wn.iihe.ac.be --domain=wn.iihe.ac.be --hostname=qclig.wn.iihe.ac.be

Before using aii-shellfe commands, you will have to create a ticket :

kinit admin

In order to add freeipa features to aii, you will have to add the two following files :

/usr/lib/perl/AII/freeipa.pm
/usr/lib/python2.6/site-packages/ipalib/plugins/aii.py

You will find the content of these files on the Quattor Github.

To secure access to profiles with SSL, you have to modify slightly freeipa.pm by adding the following lines just after the ipa-client-install part :

mkdir -p /etc/ipa/quattor/certs
certutil -L -d /etc/pki/nssdb -a -n "IPA CA" > /etc/ipa/quattor/certs/ca.pem
certutil -L -d /etc/pki/nssdb -a -n "IPA Machine Certificate - $hostname.$domainname" > /etc/ipa/quattor/certs/hostcert.pem
certutil -K -d /etc/pki/nssdb -a -n "IPA Machine Certificate - $hostname.$domainname"
pk12util -o keys.p12 -n "IPA Machine Certificate - $hostname.$domainname" -d /etc/pki/nssdb -W ""
openssl pkcs12 -in keys.p12 -out /etc/ipa/quattor/certs/hostkey.pem -nodes -password pass:''
chmod 600 /etc/ipa/quattor/certs/hostkey.pem
rm -f keys.p12

The goal of these commands is to extract the host certificate and key and the IPA CA certificate from nssdb, and to copy them in the /etc/ipa/quattor/certs directory.

To restrict access to profiles in Apache, you need to create the file /etc/httpd/conf.d/quattor_ssl.conf with the following content :

NameVirtualHost *:444
Listen 0.0.0.0:444

<VirtualHost _default_:444>
    HostnameLookups On
    SSLEngine On
    SSLCertificateFile /root/certs/hostcert.pem
    SSLCertificateKeyFile /root/certs/hostkey.pem
    SSLCACertificateFile /root/certs/ca.pem

    DocumentRoot /var/www/https

    RewriteRule ^/profiles/.*$ /profiles/profile_%{REMOTE_HOST}.xml

    ErrorLog logs/quattor_ssl_error_log
    TransferLog logs/quattor_ssl_access_log

    <Location "/profiles">
        Options +Indexes
        SSLOptions +StdEnvVars
        SSLRequireSSL
        SSLVerifyClient require
        SSLOptions +StrictRequire
        SSLVerifyDepth 1
        SSLOptions +OptRenegotiate
        SSLRequire %{SSL_CLIENT_S_DN_CN} eq %{REMOTE_HOST}
    </Location>
</VirtualHost>

Things to do in your SVN

First, you need to add the schema files for freeipa under cfg/core/<quattor_release>/quattor/aii/freeipa. The two files (default.pan and schema.pan) can be found in the Quattor Github.

To really make use of what has been done before, you need to add the following lines in the config of your site/cluster :

variable QUATTOR_PROFILE_URL ?= "https://qclig.wn.iihe.ac.be:444/profiles";

variable FREEIPA_AII_DOMAIN ?= 'wn.iihe.ac.be';
variable FREEIPA_AII_SERVER ?= 'freeipa.wn.iihe.ac.be';
variable FREEIPA_AII_REALM ?= 'WN.IIHE.AC.BE';
include {'quattor/aii/freeipa/default'};

'/software/components/ccm/cert_file' = '/etc/ipa/quattor/certs/hostcert.pem';
'/software/components/ccm/key_file' = '/etc/ipa/quattor/certs/hostkey.pem';
'/software/components/ccm/ca_file' = '/etc/ipa/quattor/certs/ca.pem';

Testing the deployment

After the "runcheck", don't forget to create a Kerberos ticket with "kinit admin" before issuing the aii-shellfe commands. Also keep in mind that the one-time password (OTP) for the automatic enrollment of the machine will be generated during the "aii-shellfe --configure", and if the entry already exists in the realm for the machine you are trying to deploy, you will not receive the OTP. So, you are oblige to issue "aii-shellfe --remove" before the "--configure" and the "--install".

Testing the Web UI

You need to be logged in on a machine that is in the private network and that has firefox installed on it. Of course, you must log in in X forwarding mode (ssh -X). Then, you must first create a Kerberos ticket with "kinit admin". If you are outside the Kerberos realm, then you have to copy the /etc/krb5.conf of a machine that is in the realm. If kinit is not installed, then "yum install krb5-workstation". The URL to access the Web UI is :

https://freeipa.wn.iihe.ac.be/ipa/ui

Testing SSH login using an account created in the realm

Let's say you are logged in on qclig.wn and you would like to log in as 'toto' (an account created with the Web UI in the realm) on node19-101.wn. You will first have to do the following things on node19-101.wn :

yum install sssd
service sssd start
service messagebus start
authconfig --enablemkhomedir --update

Moving the management of accounts from Quattor to FreeIPA

We are investigating the possibility to completely removed the management of users from Quattor, and to move it entirely into FreeIPA, but there is still a solution to find for the dCache part (creating users in Quattor is mandatory for CMS users because they need their DN to be mapped correctly otherwise they don't have write access to dCache).

By the time we find a solution for dCache, during the transition period, we need a way to keep the FreeIPA users database up-to-date :

1. Connect to qclig and create a Kerberos ticket for admin user
2. Run the following commands :

./collect_ssh_keys_ipa.pl > users_and_keys_ipa.lst
./collect_ssh_keys_quattor.pl > users_and_keys_quattor.lst
diff users_and_keys_quattor.lst users_and_keys_ipa.lst

This way, you can check which users defined in the Quattor SCDB are not yet defined in the FreeIPA system, and you can also detect if some ssh-keys are missing on the FreeIPA side for already existing users.

Troubleshooting

Cannot obtain CA certificate

I've tried to enroll m10 with the following command :

 ipa-client-install --server=freeipa.wn.iihe.ac.be --domain=wn.iihe.ac.be --hostname=m10.wn.iihe.ac.be

but it failed with the following message :

Cannot obtain CA certificate
'ldap://freeipa.wn.iihe.ac.be' doesn't have a certificate.

The workaround is :

wget -O /etc/ipa/ca.crt http://freeipa.wn.iihe.ac.be/ipa/config/ca.crt
ipa-client-install --server=freeipa.wn.iihe.ac.be --domain=wn.iihe.ac.be --hostname=m10.wn.iihe.ac.be --ca-cert-file=/etc/ipa/ca.crt

Problems with certificates with EL7 machines

The new ipa-client-install command doesn't create automatically a certificate for the machine. This is due to the new FreeIPA 4.1. As it is said in the release notes :

New option --request-cert to optionally request host certificates on FreeIPA clients (to /etc/ipa/nssdb/)

And the commands to retrieve the certificate to the machine have also changed... For the moment, we don't have any solution to provide a certificate to an EL7 machine. This means we cannot use ccm-fetch over https with EL7 machines.

Freeipa admin account locked-out

If you exceed the max. number of attempts to type the correct admin password, the account will be locked. The solution to remove the lock is the following :

1. Create an ldif with the following content :

dn: cn=global_policy,cn=WN.IIHE.AC.BE,cn=kerberos,dc=wn,dc=iihe,dc=ac,dc=be
changetype: modify
replace: krbMinPwdLife
krbMinPwdLife: 0

2. Run the command :

ldapmodify -h localhost -x -W -D "cn=directory manager" -f /root/test/krb_test.ldif

(Of course, you need to remember at least the LDAP password. If you can't... well... you've got a serious problem !)

3. Now reset the password through kadmin.local:

kadmin.local
Authenticating as principal admin/admin@WN.IIHE.AC.BE with password.
kadmin.local: change_password -pw secret123 admin@WN.IIHE.AC.BE
Password for "admin@WN.IIHE.AC.BE" changed.
kadmin.local: q

4. Run this command to clear cache :

kdestroy

User was changed on freeipa but changes were not yet propagated

changes take some time to propagate. Simply issue the following command:

sss_cache -u <user_login>⁠⁠⁠⁠

Sources

• Discussion with people from Ghent (a big THANKS to Stijn and Luis for the work they have done !).
• SINDES official website
• FreeIPA official website