TheBonsai's Blog

About the days and nights of TheBonsai

Archive for the 'Linux' Category

Fingerprint daemon tickles DBUS daemon

March 2nd, 2018 by TheBonsai

On a customer Linux system, OEL (RHEL) 7.3 we noticed a huge load of open files for the dbus-daemon process. After reaching the open file limits, the process began to use one whole CPU core and weird effects took place. All open files pointed to different /proc/PID/cmdlinefiles, all of processes not existing anymore.

The problem was detected a while after a job with a ten minute frequency was deployed. This initially means nothing, but the time correlation is high enough to keep that in mind.

After some hours of troubleshooting, log analysis, process analysis and some common sense, I found a closed bug report in the RH bugzilla about a very similar – or the same – problem. This pointed me in the right direction. The key is, that the mode this job is being ran involves PAM loading (it actually doesn’t switch, but it could switch user on start). Due to the RH standard PAM configuration, this involves loading if installed (even if unused). This leads to a request to start the fprintd daemon, which goes from systemd-logind over the DBUS to the dbus-daemon. The fprintd daemon process is started and terminates again. This constellation leads to dbus-daemon holding open file descriptors pointing to /proc/PID/cmdline of the job process (which terminates afterwards) that are never closed by the dbus-daemon itself. A typical file descriptor leakage situation.

The workaround was just to disable the fingerprint PAM module in the relevant PAM configuration file(s) and restart the DBUS daemon and some DBUS-using processes that don’t recover themselves (like systemd-logind and NetworkManager).

Unfortunately I wasn’t able to understand the exact anatomy of the problem, since another referenced bug in the bug report I linked above isn’t available¬† to me.

Category: Linux, Work | 2 Comments »