Had an sssd process spinning and using 100% cpu. Did an strace on it and saw that it was complaining about too many open files.
pid accept(24, 0xaddress, [110]) = -1 EMFILE (Too many open files)
getting the number of open files for the process.
# lsof -p $(pidof sssd_pam) |wc -l
1065
Looking at the limits for sssd, I saw that the nofile was set to 1024, which appears to be the default everywhere I tried.
# cat /proc/$(pidof sssd_pam)/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 5788 5788 processes
Max open files 4096 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 5788 5788 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
What I didn't know is that you can change it just by echoing the correct string to that limits file.
# echo -n "Max open files=8192:8192" >/proc/$(pidof sssd_pam)/limits
# cat /proc/$(pidof sssd_pam)/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 5788 5788 processes
Max open files 8192 8192 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 5788 5788 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
This works on a RHEL6 system, didn't work on my Fedora system. On Fedora there is a program called prlimit which can be used to modify the limits.
[root@burnaby 907]# prlimit --pid=907 --nofile=8192:8192
[root@burnaby 907]# cat limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 62617 62617 processes
Max open files 8192 8192 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 62617 62617 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us