some false positive reports

[ohauer]

Thanks, this seems to be a really useful tool, specially if someone manage hosts with automation like salt/pupet/...

I suspect this are some false positives, specially if process is started via daemon or forked process changes the UID to unprivileged user

   pid    jid stat command
43919      0 miss /usr/local/openjdk8/bin/java
lsop: sysctl: kern.proc.pathname: 48638: No such file or directory
 48638      0 miss (master)
 44015      0 miss /usr/local/sbin/httpd
 85311      0 miss /usr/local/sbin/httpd
 85310      0 miss /usr/local/sbin/httpd
 85309      0 miss /usr/local/sbin/httpd

ps -waux | grep java
jenkins 43918   0.0  0.0   14496    2044  -  Is   10:22AM      0:00.00 daemon: /usr/local/openjdk8/bin/java[43919] (daemon)
jenkins 43919   0.0  5.7 8280092 1442980  -  I    10:22AM      0:30.90 /usr/local/openjdk8/bin/java -DJENKINS_HOME=/usr/local/jenkins -jar ...

ps -waux | grep 48638
root    48638   0.0  0.0   69300     748  -  Ss   30Aug16      0:03.03 /usr/local/libexec/postfix/master -w

ps -waux | grep -e 44015 -e 85311 -e 85310 -e 85309
www     44015   0.0  0.2  417276   53132  -  S    29Aug16      1:41.36 /usr/local/sbin/httpd
www     85309   0.0  0.1  374952   29892  -  S    29Aug16      1:23.65 /usr/local/sbin/httpd
www     85310   0.0  0.2  430328   51092  -  S    29Aug16      1:37.66 /usr/local/sbin/httpd
www     85311   0.0  0.1  416700   33924  -  S    29Aug16      1:26.33 /usr/local/sbin/httpd

[606u]

lsop scans processes' virtual memory mappings, searching for ones with missing paths. Can you post output from procstat -v <pid> for java, postfix and any of httpd processes? Which HTTP server is that? Apache?

[ohauer]

Since the output is 2000+ lines, I've place the file for you here:
https://people.freebsd.org/~ohauer/diffs/lsop/lsop_false_positive_procstat.log

Additional I've created a port prototype:
https://people.freebsd.org/~ohauer/diffs/lsop/lsop.shar

[606u]

I've installed jenkins and reproduced 1st of your cases. It might be that JRE creates some executable code on-the-fly (possibly by JIT), mmaps it for read+execute, then unlinks the underlying file.

Sadly once file is gone, its file-system path can no longer be recovered (or at least I have no knowledge how to do so); only information left is file system id and an inode, but find / -inum ### finds nothing.

I couldn't reproduce Apache case, but it seems your configuration is pretty large and loads lots of modules.

Any suggestions how to deal with it?

[606u]

This is what I came up with: lsop could use whitelisting rules -- to be told how many such mappings to be allowed for specific binaries -- via some sort of a state file. For example:

/usr/local/openjdk8/bin/java    1
/usr/local/sbin/httpd           1

Even better, a new feature might be implemented, allowing lsop to generate those rules by scanning a running system and recording which processes do such tricks.

[ohauer]

I just finished checking some other systems where apache is running and none of the other systems was showing the same output, so at the moment I assume lsop was correct in case of the httpd process.
After disabling some third party modules (mod_wsgi, mod_dav_svn) everything showed OK.

In case of jenkins, I suspect that's something from /usr/sbin/daemon will do some additional tests.
I think the only real false positive is postfix that controls the lifetime of child process (the reported master line)

[606u]

I've made an initial implementation of whitelisting (a separate branch), perhaps you can give it a try...?

[ohauer]

Not sure if I used the tool the correct way, but it fails on the postfix process.

$ touch ./whitelist
$ ./work/stage/usr/local/sbin/lsop -c ./whitlist.txt
  pid    jid stat command
 43919      0 miss /usr/local/openjdk8/bin/java
lsop: sysctl: kern.proc.pathname: 48638: No such file or directory
lsop: whitelist not created: cannot get process 48638 path

$ ./work/stage/usr/local/sbin/lsop
   pid    jid stat command
 43919      0 miss /usr/local/openjdk8/bin/java
lsop: sysctl: kern.proc.pathname: 48638: No such file or directory
 48638      0 miss (master)

$ postfix stop
$ ./work/stage/usr/local/sbin/lsop -c ./whitelist.txt
$ cat ./whitelist.txt
/usr/local/openjdk8/bin/java    1

$ ./work/stage/usr/local/sbin/lsop -w ./whitelist.txt ; echo $?
0
$ postfix start
$ ./work/stage/usr/local/sbin/lsop -w ./whitelist.txt ; echo $?
0
$ ./work/stage/usr/local/sbin/lsop
   pid    jid stat command
 43919      0 miss /usr/local/openjdk8/bin/java

I know postfix recycles periodically child processes, but I'm sure I've started / stopped postfix also in previous tests. will wait some time and then look again after postfix recycled the child processes

[606u]

Sadly, after reproducing your case with postfix, and digging through the source code, it turned out lsop can never work reliably as currently designed: for memory mapped files, including binaries and shared objects, kernel only cares about vnode and uses vn_fullpath(9) to recover the path from vnode; however:

This process [vn_fullpath] is necessarily unreliable for several reasons: intermediate entries in the path may not be found in the [VFS name] cache; files may have more than one name (hard links), not all file systems use the name cache (specifically, most synthetic file systems do not); a single name may be used for more than one file (in the context of file systems covering other file systems); a file may have no name (if deleted but still open or referenced).

Whereas lsop cares for the last case -- deleted but still open or referenced -- 1st case is perfectly valid and I can bet this is the reason why initially lsop reports everything is okay, but few hours later it marks postfix as outdated: because entries have been dropped from VFS name cache.

[ohauer]

I think it is good to know the limitations, lsop is in every case a fantastic utility to check which process needs a restart after updating installed packages or applying OS patches (I've already deployed it on 40+ systems).

What do you think about the port prototype (https://people.freebsd.org/~ohauer/diffs/lsop/lsop.shar)?
(To use the current code, replace s/fb5858e/c097c33/)

[606u]

I have some ideas about how to bypass this restriction, although it requires a major rewrite and will take awhile:

Basically lsop 0.2 :-) will be given a list of directories, where binaries are (/bin, /lib, /usr/lib, etc.); then it will create a small index/snapshot containing file paths, file system ids and inode numbers. When started at a later time it can scan those directories again to see if any files have been updated or deleted (inode would change in the former case, file would disappear in the latter case). Only then it can scan running processes to see if any of updated/deleted objects are in use, but this time it will detect those using file system id + inode, instead of file paths.

Thanks for the port prototype, but I'll have to read a manual or two first, as I've no idea what is the next step to follow.

[grahamperrin]

0.2

https://cgit.freebsd.org/ports/commit/?id=7e31d734fd42fbfd2500ddebb96fc5b63407a35d