Toger Blog

Cloud-Init With CentOS7, Chef, and SELinux

The CentOS 7 AMI in Amazon comes with Cloud-Init (cloud-init-0.7.5-10.el7.centos.1.x86_64). This is quite handy as it assists in automating several bootup tasks. One of these tasks is to install and bootstrap Chef. Unforunately, when SELinux is installed the Chef handler will fail.

Sample error:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

[CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /var/lib (recursive=True)
[CLOUDINIT] util.py[DEBUG]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py'>) failed
       Traceback (most recent call last):
         File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 658, in _run_modules
           cc.run(run_name, mod.handle, func_args, freq=freq)
         File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 63, in run
           return self._runners.run(name, functor, args, freq, clear_on_fail)
         File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 197, in run
           results = functor(*args)
         File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py", line 54, in handle
           util.ensure_dir(d)
         File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1291, in ensure_dir
           os.makedirs(path)
         File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 167, in __exit__
           self.selinux.restorecon(path, recursive=self.recursive)
         File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in restorecon
           for fname in fnames]), None)
         File "/usr/lib64/python2.7/posixpath.py", line 246, in walk
           walk(name, func, arg)
         File "/usr/lib64/python2.7/posixpath.py", line 238, in walk
           func(arg, top, names)
         File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in <lambda>
           for fname in fnames]), None)
         File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon
           status, context = matchpathcon(path, mode)
       OSError: [Errno 2] No such file or directory

This proved to be most frustrating. It was obviously failing while it was fixing the SELinux parameters under /var/lib, but it was not clear why. There were no denied messages in the audit log. Of course /var/lib and /var/lib/chef existed. /var/lib/chef is a default directory listed in /usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py.

Adding in several debug statements led to determining that the issue was specific to /var/lib/nfs/rpc_pipefs. The file existed and had no special permissions, although it had a generic SELinux label as opposed to a nfs-specific value. Further investigation showed that the root of the issue was in libselinux, which is looking up the intended label in /etc/selinux/targeted/contexts/files/file_contents:

1
2

/usr/share/Modules/init(/.*)?   system_u:object_r:bin_t:s0
/var/lib/nfs/rpc_pipefs(/.*)?   <<none>>

The selinux python library is expecting a label as above, and returns a Errno 2 ‘No such file or directory’ if it hits the ‘none’ value. This error is returned to cloud-init and causes the chef handler to bomb out. Adding a label to the file_contents file works around the issue but likely breaks NFS (if it were in use). The better approach would be for util.py to ignore the case where a new label cannot be found. If CentOS7 updated to the latest cloud-init there is a decent chance this is fixed.

Filed as https://bugs.centos.org/view.php?id=10990.