Bruning Questions: SmartOS /proc Tools

April 05, 2013 - by Mr. Max Bruning

ask-mr-bruning-logo

I am currently working on an online course that is aimed at Linux system administrators and developers who want to use SmartOS. One of the many differences between Linux and Solaris-based systems is the /proc file system. On Linux, this is documented proc(5), and for SmartOS, proc(4).

On Linux, /proc includes information about processes, as well as many other pieces of information about the system. Many of the files are readable (i.e., you can cat them), and some are writeable allowing you to modify some aspects of the system. On SmartOS, /proc only contains information about processes. Most (all?) of the data in the files are accessed via ioctl(2) system calls, as explained in the man page mentioned above.

In this blog, I'll just talk about a set of tools available on SmartOS for accessing data in/proc. Some of these tools can also be used on core files to get information about the environment of a process at the time it generated a core file. Most of the tools I am referring to are documented in proc(1). People familiar with Linux will find that most of the information the tools give you on SmartOS are available in some form by cat-ing the appropriate file in /proc.

First, a list of the tools. Afterwords, we'll take a look at some examples.

  • pargs(1) - Display process arguments, environment variables, and auxillary information. Also works on core files.
  • pgrep(1) - Find process(es) by name and other attributes
  • pkill(1) - Kill a process with name or attribute. Note that on Linux, the killall(1) command can be used instead of pkill, and on SmartOS, the killall(1M) command kills everything. Not too great when porting shell scripts containing killall from Linux to SmartOS.
  • pcred(1) - Print or set credentials (user/group ids) of a process. Usable on core files.
  • pldd(1) - Show dynamic shared libraries used by a process. Includes libraries linked via dlopen(3C). Usable on core files.
  • psig(1) - Show disposition of signals for a process.
  • pstack(1) - Show stack backtraces for all threads (or a specific thread) within a process. This can also be used to examine stack traces from core files.
  • pfiles(1) - Show information about open files/sockets of a process. For network endpoints, shows local (and remote, if connected) addresses.
  • pmap(1) - Display process address space.
  • pwdx(1) - Print current working directory of a process.
  • pstop(1) - Stop process or thread.
  • prun(1) - Inverse of pstop
  • pwait(1) - Wait for process(es) to terminate.
  • ptime(1) - Time process using microstate information. Like time(1), but does not count time spent by child processes.
  • ptree(1) - Print process tree (parent/child relationships). Can be used for specific pid, zone, or user.

Now we'll go through some examples.

# pgrep sh
35668
91114
91113
91117
#

Note that this will match, for instance, sh, bash, and sshd.

# pgrep a.out
91145
# pkill a.out
# [1]+  Terminated              ./a.out
#

Here is pargs(1) output showing command line and environment variables.

# pargs -ea $$
91117: -bash
argv[0]: -bash

envp[0]: MANPATH=/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man:/opt/local/gnu/man:/opt/local/man:/usr/share/man
envp[1]: TERM=dumb
envp[2]: SHELL=/usr/bin/bash
envp[3]: SSH_CLIENT=10.88.88.162 43182 22
envp[4]: SSH_TTY=/dev/pts/3
envp[5]: USER=root
envp[6]: COLUMNS=80
envp[7]: PAGER=less
envp[8]: MAIL=/usr/mail/root
envp[9]: PATH=/opt/local/gnu/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbin
envp[10]: PWD=/var/tmp
envp[11]: LANG=C
envp[12]: TZ=UTC
envp[13]: LINES=24
envp[14]: SHLVL=1
envp[15]: HOME=/root
envp[16]: LOGNAME=root
envp[17]: SSH_CONNECTION=10.88.88.162 43182 8.17.87.25 22
envp[18]: FTPMODE=auto
envp[19]: _=/usr/bin/pargs
envp[20]: OLDPWD=/root
#

The same for a core file.

# pargs -ea core.91145
core 'core.91145' of 91145:       ./a.out
argv[0]: ./a.out

envp[0]: MANPATH=/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man:/opt/local/gnu/man:/opt/local/man:/usr/share/man
envp[1]: TERM=dumb
envp[2]: SHELL=/usr/bin/bash
envp[3]: SSH_CLIENT=24.6.92.162 43182 22
envp[4]: SSH_TTY=/dev/pts/3
envp[5]: USER=root
envp[6]: COLUMNS=80
envp[7]: PAGER=less
envp[8]: MAIL=/usr/mail/root
envp[9]: PATH=/opt/local/gnu/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbin
envp[10]: PWD=/var/tmp
envp[11]: LANG=C
envp[12]: TZ=UTC
envp[13]: LINES=24
envp[14]: SHLVL=1
envp[15]: HOME=/root
envp[16]: LOGNAME=root
envp[17]: SSH_CONNECTION=24.6.92.162 43182 8.17.87.25 22
envp[18]: FTPMODE=auto
envp[19]: _=./a.out
envp[20]: OLDPWD=/root
#

Some pldd(1) output.

# pldd $$
91117:       -bash
/usr/bin/bash
/lib/libcurses.so.1
/lib/libsocket.so.1
/lib/libnsl.so.1
/lib/libdl.so.1
/lib/libc.so.1
#
# pldd core.91145
core 'core.91145' of 91145:  ./a.out
/lib/amd64/libc.so.1
#

And psig(1).

# psig $$
91117: -bash
HUP        caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
INT    caught  sigint_sighandler       0
QUIT   ignored
ILL      caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
TRAP   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
ABRT   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
EMT    caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
FPE    caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
KILL   default
BUS      caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
SEGV   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
SYS    caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
PIPE   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
ALRM   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
TERM   ignored
USR1     caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
USR2   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
CLD    blocked,caught  sigchld_handler 0
PWR    default
WINCH    caught  sigwinch_sighandler     0
URG    default
POLL     default
STOP     default
TSTP     ignored
CONT     default
TTIN     ignored
TTOU     ignored
VTALRM   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
PROF   default
XCPU     caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
XFSZ   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
WAITING        default
LWP      default
FREEZE   default
THAW     default
CANCEL   default
LOST     caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOST
XRES   default
...
#

And pstack(1). Note this is very useful to help determine where/if a process is hung, and, on core files, where the process died.

# pstack 28963
28963:       /opt/local/sbin/mysqld --user=mysql --basedir=/opt/local --datadir=/va
-----------------  lwp# 1 / thread# 1  --------------------
 fffffd7fff2601da pollsys  (fffffd7fffdff480, 2, 0, 0)
 fffffd7fff1e704a poll () + 62
 0000000000613276 _Z26handle_connections_socketsv () + b6
 0000000000615535 _Z11mysqld_mainiPPc () + e45
 000000000060d64c _start () + 6c
-----------------  lwp# 2 / thread# 2  --------------------
 fffffd7fff2575e7 lwp_park (0, 0, 0)
 fffffd7fff250960 cond_wait_queue () + 68
 fffffd7fff250f33 __cond_wait () + 7b
 fffffd7fff250f8b cond_wait () + 23
 fffffd7fff250fc1 pthread_cond_wait () + 9
 000000000099e7f6 os_event_wait_low () + 46
 000000000099d3a6 os_aio_simulated_handle () + 6d6
 000000000095f375 fil_aio_wait () + 45
 00000000008ed758 io_handler_thread () + 58
 fffffd7fff2572d4 _thrp_setup () + bc
 fffffd7fff2575a0 _lwp_start ()
...
#

pfiles(1)

# pfiles 93362
93362:       /usr/lib/ssh/sshd
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:532,0 ino:19922952 uid:0 gid:3 rdev:38,2
      O_RDWR
      /devices/pseudo/mm@0:null
      offset:0
   1: S_IFCHR mode:0666 dev:532,0 ino:19922952 uid:0 gid:3 rdev:38,2
      O_RDWR
      /devices/pseudo/mm@0:null
      offset:0
   2: S_IFCHR mode:0666 dev:532,0 ino:19922952 uid:0 gid:3 rdev:38,2
      O_RDWR
      /devices/pseudo/mm@0:null
      offset:0
   3: S_IFDOOR mode:0444 dev:542,0 ino:52 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[1614]
      /var/run/name_service_door
   4: S_IFSOCK mode:0666 dev:540,0 ino:5398 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
       SOCK_STREAM
     SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(128872)
    sockname: AF_INET6 ::ffff:10.0.110.25  port: 22
 peername: AF_INET6 ::ffff:10.0.110.7  port: 54326
   5: S_IFIFO mode:0000 dev:530,0 ino:5480480 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK FD_CLOEXEC
...
#

pmap(1) output can give you a good idea of how the address space of a process is being used, also on core files.

# pmap -x $$
91117:        -bash
 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
08045000      12      12       4       - rw---    [ stack ]
08050000     724     668       -       - r-x--  bash
08114000      24      24       8       - rwx--  bash
0811A000     160     160      44       - rwx--    [ heap ]
FEAF0000     456     456       -       - r-x--  libnsl.so.1
FEB72000       8       8       4       - rw---  libnsl.so.1
FEB74000      20      12       -       - rw---  libnsl.so.1
FED50000      64      16       -       - rwx--    [ anon ]
FED6D000       4       4       -       - rwxs-    [ anon ]
FED70000      24      12       4       - rwx--    [ anon ]
FED80000       4       4       -       - rwx--    [ anon ]
FED90000    1216    1216       -       - r-x--  libc.so.1
FEEC0000      36      36      16       - rwx--  libc.so.1
FEEC9000       8       8       -       - rwx--  libc.so.1
FEED0000       4       4       -       - r-x--  libdl.so.1
FEEE0000       4       4       4       - rwx--    [ anon ]
FEEF0000      56      56       -       - r-x--  libsocket.so.1
FEF0E000       4       4       -       - rw---  libsocket.so.1
FEF10000       4       4       4       - rwx--    [ anon ]
FEF20000     180     180       -       - r-x--  libcurses.so.1
FEF5D000      28      28       -       - rw---  libcurses.so.1
FEF64000       8       4       -       - rw---  libcurses.so.1
FEF70000       4       4       -       - r--s-  ld.config
FEF80000       4       4       4       - rw---    [ anon ]
FEF90000       4       4       -       - rw---    [ anon ]
FEFA0000       4       4       4       - rwx--    [ anon ]
FEFB0000       4       4       -       - rwx--    [ anon ]
FEFB7000     208     208       -       - r-x--  ld.so.1
FEFFB000       8       8       4       - rwx--  ld.so.1
FEFFD000       4       4       -       - rwx--  ld.so.1
-------- ------- ------- ------- -------
total Kb    3288    3160     100       -
#
#
# pmap core.91145
core 'core.91145' of 91145:     ./a.out
0000000000400000          4K r-x--  /var/tmp/a.out
0000000000410000          8K rw---    [ heap ]
FFFFFD7FFF170000         24K rwx--    [ anon ]
FFFFFD7FFF180000          4K rwx--    [ anon ]
FFFFFD7FFF190000       1584K r-x--  /lib/amd64/libc.so.1
FFFFFD7FFF32C000         52K rw---  /lib/amd64/libc.so.1
FFFFFD7FFF339000          8K rw---  /lib/amd64/libc.so.1
FFFFFD7FFF350000          4K rwx--    [ anon ]
FFFFFD7FFF360000          4K rwx--    [ anon ]
FFFFFD7FFF370000          4K rw---    [ anon ]
FFFFFD7FFF380000          4K rw---    [ anon ]
FFFFFD7FFF390000          4K rwx--    [ anon ]
FFFFFD7FFF393000        348K r-x--  /lib/amd64/ld.so.1
FFFFFD7FFF3FA000         12K rwx--  /lib/amd64/ld.so.1
FFFFFD7FFF3FD000          8K rwx--  /lib/amd64/ld.so.1
FFFFFD7FFFDFE000          8K rw---    [ stack ]
         total         2080K
#

Some ptime(1) output.

# ptime dd if=/dev/zero of=/dev/null bs=8k count=1024
1024+0 records in
1024+0 records out
8388608 bytes (8.4 MB) copied, 0.00246212 s, 3.4 GB/s

real        0.010517277
user        0.002187369
sys         0.007724275
#

And here is ptree.

# ptree $$
17701 zsched
  35668 /usr/lib/ssh/sshd
    91113 /usr/lib/ssh/sshd
      91114 /usr/lib/ssh/sshd
        91117 -bash
          93577 ptree 91117
#

There are other tools using /proc, including ps(1), truss(1), and sotruss(1).

For those interested, I am teaching a DTrace course in San Francisco, May 14 through May 16. Details can be found here. Hope to see you there!

:

Sign up now for Instant Cloud Access Get Started