TheBonsai's Blog

About the days and nights of TheBonsai

Fingerprint daemon tickles DBUS daemon

March 2nd, 2018 by TheBonsai

On a customer Linux system, OEL (RHEL) 7.3 we noticed a huge load of open files for the dbus-daemon process. After reaching the open file limits, the process began to use one whole CPU core and weird effects took place. All open files pointed to different /proc/PID/cmdlinefiles, all of processes not existing anymore.

The problem was detected a while after a job with a ten minute frequency was deployed. This initially means nothing, but the time correlation is high enough to keep that in mind.

After some hours of troubleshooting, log analysis, process analysis and some common sense, I found a closed bug report in the RH bugzilla about a very similar – or the same – problem. This pointed me in the right direction. The key is, that the mode this job is being ran involves PAM loading (it actually doesn’t switch, but it could switch user on start). Due to the RH standard PAM configuration, this involves loading pam_fprintd.so if installed (even if unused). This leads to a request to start the fprintd daemon, which goes from systemd-logind over the DBUS to the dbus-daemon. The fprintd daemon process is started and terminates again. This constellation leads to dbus-daemon holding open file descriptors pointing to /proc/PID/cmdline of the job process (which terminates afterwards) that are never closed by the dbus-daemon itself. A typical file descriptor leakage situation.

The workaround was just to disable the fingerprint PAM module in the relevant PAM configuration file(s) and restart the DBUS daemon and some DBUS-using processes that don’t recover themselves (like systemd-logind and NetworkManager).

Unfortunately I wasn’t able to understand the exact anatomy of the problem, since another referenced bug in the bug report I linked above isn’t available¬† to me.

This entry was posted on Friday, March 2nd, 2018 at 12:46 and is filed under Linux, Work. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

2 responses about “Fingerprint daemon tickles DBUS daemon”

  1. M. Pausch said:

    Yaaay, the first blog post since almost 2 years. And I’m related to that :-)

    A few additional information:
    – operating system is Oracle Linux 7.3 (but very likely that RHEL 7.x is affected also)
    – problem also occurs on OL 7.4 with most current patches as today

    – fprintd stuff doesn’t get installed by default when OS gets installed with “Minimal Installation” option, but get installed with e.g. “Server with GUI” option.

    – the Oracle database has a DBMS_SCHEDULER job for rman backups, not a cron job.
    therefor you have to define some credentials, because it has to do OS authentication:

    exec DBMS_CREDENTIAL.create_credential(
    credential_name => ‘BACKUP_CRED’,
    username => ‘[OS user that owns the Oracle installation]’
    password => ‘[password of OS user]’
    );

    DBMS_SCHEDULER jobs looks like this:
    exec DBMS_SCHEDULER.CREATE_JOB (
    job_name => ‘”SYS”.”DB_BACKUP_FULL”‘,
    job_type => ‘BACKUP_SCRIPT’,
    job_action => ‘connect target /
    [do the backup stuff}’
    credential_name => ‘”SYS”.”BACKUP_CRED”‘
    [some further irrelevant job attributes]

    – a tested workaround: removing the line “auth sufficient pam_fprintd.so” from /etc/pam.d/system-auth
    – a untested workaround: “yum erase fprintd-pam fprintd libfprint” (but very likely that this works also )

    – because we’ve seen it in on Oracle Linux with Oracle database and we have a valid CSI, I opened a SR which led to Bug 27693223 (currently not publicly available)

    Let’s see what happens next :-)

  2. M. Pausch said:

    A little update on this:
    Updating Oracle Linux to version 7.5 pushes dbus from 1.6 to 1.10.
    After the update the problem doesn’t occur anymore.

Leave a Reply