Make Peace With The Linux Process Table

Click the link Linux Server Tutorial if you find this version difficult to read.

A fundamental design feature of Unix-like Operating Systems is that many of a system’s resources are accessible via the filesystem, as a file. For example the “procfs” pseudo-filesystem offers us access to all kinds of valuable treasures.

By querying files present in “/proc” you can quickly find out the intricacies of your network connections, the server’s CPU vendor and look up a mountain of useful information such as the command line parameters passed to your latest-and-greatest application when it fired up. This is thanks to the fact that many of the functions of a server, such as a network connection are really just another stream of data and in most cases can be presented as a file on your filesystem.

Let’s jump ahead of ourselves for a moment in case you’re not too familiar with the “/proc” filesystem. If you knew that your Process ID was “16651” for example then you could run this command in order to find out what was sent to the Puppet process to start it up:

# xargs -0 < /proc/16551/cmdline

The output from that command is:

/usr/bin/ruby /usr/bin/puppet agent

As you can see Puppet’s agent is using the Ruby programming language in this case and the binary “/usr/bin/puppet” is passed the parameter “agent” to run it as an “agent” and not a “master”.

The “Everything Is A File” philosophy makes total sense if you think about it. The power harnessed within Unix’s standard tools (usually used for manipulating data held in the more common text files) such as “grep”, “awk” and “sed” are a major strength of the Operating System. But most importantly you can have a system’s components integrate very efficiently and easily if many things are simply files.

If you have you ever tried to look into a process running on a Unix-like machine then you’ll know that if anything the abundance of information adds confusion, rather than assists, if you don’t know where to look. There are all sorts of things to consider when you are eagerly trying to track down a rogue process on production machine.

In this chapter I will attempt to offer a broad insight into how the Process Table can be accessed by the ps command and in combination with “/proc” and “/dev” how it can help you manipulate your systems.


There are a few legacy stumbling blocks when it comes to looking up a process on different types of Unix boxes but thankfully we can rely on the trusty "ps" command to mitigate some of these headaches automatically for us.

For example, Unix used the "ps" command by grouping its parameters together and prepending a hyphen. Whereas BSD enjoyed grouping switches together but, for one reason or another, fell out with the hyphen entirely.

Throwing another spanner in the works however was good, old GNU's preference; where its long options used two dashes. Now that you've fully committed those confusing differences to memory let's assume that the ps command does as much as it can for us by mixing and matching the aforementioned options in an attempt to keep everyone happy.

Be warned that occasionally sometimes oddities can occur so keep an eye out for them just in case. I’ll try and offer alternative commands as we continue to act as a reminder that not all is to be taken exactly as read. For example a very common use of the ps command is:

# ps -aux

It should be noted however that this is indeed different to:

# ps aux

You might suspect, and would be forgiven for thinking as much, that this is purely to keep everyone on their toes. However according to the ps command’s manual this is apparently because POSIX and UNIX insist that they should cater for processes owned by a user called "x". However, if I’m reading the information correctly, then should the user "x" not exist then "ps aux" is run. I love the last sentence of the manual’s definition and draw your attention to it as a gentle warning: “It is fragile, subject to change, and thus should not be relied upon.”

   Process Tree

Enough eye-strain for a moment and let’s begin by looking at the ps command and what it is capable of helping with in relation to querying the Process Table.

For starters (and I won’t be betting the ranch on this statement) it's relatively safe to assume that upper and lower case mean the same thing.

If you’ve never seen the output of a Process Tree then it might help with understanding “child” threads which live under a “parent” process. In this case the command is simply:

# pstree

The not-so-impossible-to-decipher output from that command is shown in Listing One (on a server not running “systemd but good, old “init” (which is always Process ID (PID) number one, as an aside):






     |                    `-hald-addon-inpu



     |        `-qmgr







Listing One: Output from the “pstree” command showing parent processes and their children

You can make your screen output much messier by adding the “-a” switch. Doing so will add command line arguments (pulled from the /proc filesystem in the same way that our example did earlier). This is very useful but you might want to do something like “grep” a specific process name from the output, as follows:

# pstree -ap | grep ssh


  |   `-sshd,32365

  |       `-sshd,32370

  |               |-grep,8401 ssh

  |   |-sssd_ssh,1143 --debug-to-files

Listing Two: The command “pstree” showing only SSH processes with command line arguments and PIDs

As we can see from Listing Two, the command which we are querying with is also shown (starting with “grep”) in the output so try not to let that trip you up on your travels. I’ve added the “-p” switch to display the PIDs too for ease.

One final look at that example is seen in Listing Three. Here the all-pervasive “-Z” switch offers us any SELinux config associated with the parent and child detail displayed in our process table tree. That command for reference was:

# pstree -aZp | grep ssh


  |   `-sshd,32365,`unconfined_u:system_r:sshd_t:s0-s0:c0.c1023'

  |       `-sshd,32370,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023'

  |               |-grep,8406,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023' ssh

  |   |-sssd_ssh,1143,`system_u:system_r:sssd_t:s0' --debug-to-files

Listing Three: The output now includes SELinux detail, command line arguments and PIDs

   Alternative View

Back to school for a moment and we’ll pick up some of the basics that will help us increase our knowledge later. We will (very loosely) mimic the Process Tree output we saw earlier with just the ps command now. You might stumble across “forest” puns in manuals with regards to “trees” in case it’s confusing.

The “-e” switch is also equal to “-A” and it will dutifully display ALL of the processes. We’re going to combine that with “-j” which should give us a “jobs format”. I’m sorry to potentially give you more headaches and I encourage to try a few of these of alternative options but one small caveat would be that running “j” and “-j” gives different levels of information. The non-hyphenated version giving more output in this case.

We will also add the “-H” to display the hierarchical output and we can achieve this by running:

# ps -ejH

Or, in a parallel universe, alternatively you can also use for vaguely similar results:

# ps axjf

Try it yourself. It’s far from identical to the “pstree” command but there are similarities in the output due to the hierarchies I’m sure that you will agree.


It is easy to get overwhelmed by the level of detail that the ps command can provide. My favourite process command is:

# ps -ef

With “-e” this gives us a display of all the processes in the Process Table and the newly added “-f” switch gives us what’s known as full-format listing. Apparently we can add “-L” to this concoction in order to offer us process thread information. I find this command very easy to remember around Christmas time.

# ps -eLf

Et voilà, as requested two new columns NLWP (number of threads) and LWP (thread ID) are now added to the now busying display.

If you only wanted to print the name of a process (which might be ideal for scripting) then you can sort the wheat from the chaff by using this command for process number "37":

# ps -p 37 -o comm=

An option that I've used in the past a few times is checking what command my terminal is currently

responsible for. This is useful if you're stressing your server or worrying your workstation with lots of extra load. The simple (non-hyphenated) "T" switch lets us view this.

# ps T

You can test this by running something unusual, like a “for” loop with a pause in it using“sleep”, or anything odd that stands out such as this command.

# echo {1..999999} &

This simply runs a counter process in the background and when we run “ps T” we can see processes associated with the terminal in question.

   Built-In Features

Let’s look at a few other built-in basics of the pliable ps command.

You can reverse the output of a command with a non-hyphenated “N” switch, which stands for negate. Or more precisely from the manual “Select all processes except those that fulfill the specified conditions. (negates the selection) Identical to --deselect.” All is revealed in Listing Four. As we can see, there isn’t any mention of “ntp” except from our ps command.

# ps N ntp


 1414 tty1     Ss+    0:00 /sbin/mingetty /dev/tty1

 1416 tty2     Ss+    0:00 /sbin/mingetty /dev/tty2

 1418 tty3     Ss+    0:00 /sbin/mingetty /dev/tty3

 1420 tty4     Ss+    0:00 /sbin/mingetty /dev/tty4

 1426 tty5     Ss+    0:00 /sbin/mingetty /dev/tty5

 1430 tty6     Ss+    0:00 /sbin/mingetty /dev/tty6

 9896 pts/1    S       0:00 sudo -i

 9899 pts/1    S       0:00 -bash

10040 pts/1   R+     0:00 ps N ntp

Listing Four: Demonstrates running processes without including “ntp” (note our “ps” command again being seen)

Imagine that you again wanted to see what SSH activity was going on and also the Hardware Abstraction Layer daemon, “hald”. Hardly related I agree but you can never account for strange scenarios when it comes to computers.

The command which follows is a way of searching for a list of processes with a certain name. Separated without a space and just a comma in this case.

# ps -C sshd,hald

If you need to check any processes run by a particular system group you can achieve this with the

following command:

# ps -G ntp


Although not always the case the modern ps command cleverly mitigates our migraine-inducing compatibility headaches by letting us run a simple command in a number of ways where possible.

If you wanted to select a list of processes which belonged to the superuser, "root", then you can achieve this with the following three commands (which admittedly display ever-so-slightly different outputs):

# ps -u root

# ps U root

# ps --user root

The above commands dutifully offer us what's known as the "EUID" or “effective ID” of a user but

not the "real user ID". In reality, no pun intended, every process actually has two user IDs; just to keep things simple. This also applies to groups too but let's not worry about that.

Apparently the kernel is most concerned with the "effective user ID" for activities such as writing to a file and if a user is allowed to complete a request to do something that requires a privilege.

And, although that’s required for much of the time, there's an important scenario within which the "real user ID" needs to be paid attention to. If someone or something wants to alter the owner of "effective user ID" on an already running process then the kernel needs to check up on both the "real user ID" and the "effective user ID".

Changing the ownership of a process is particularly useful if a new user wants to do essentially the same thing (like write to a file) as the existing owner does. Rather than duplicating the process (adding extra strain to the system and potentially introducing more security aspects to consider) we can avoid duplicating the process and simply reassign it.

What about after a user is finished with their short task? The answer is that we only temporarily give access and then swap it back again to its original owner. If you want to reveal the somewhat elusive "real user IDs" then you can do so with system groups like this (which is the same as "-G"):

# ps --Group ntp

And, not surprisingly, we can do exactly the same thing for users as follows (we do this with "-U"):

# ps --User chrisbinnie

If you want to query a very specific Process ID (because you've spotted it in "top" or a script

has complained about it) then to all intents and purposes these commands all the do the same. I have included some output for comparison as it’s short and easy to read in this case:

# ps -p 1248


 1248 ?      00:00:08 ntpd

# ps p 1248


 1248 ?      Ss    0:08   ntpd -u ntp:ntp -p /var/run/ -g

# ps --pid 1248


 1248 ?      00:00:08 ntpd

If you’ve ever wondered about the efficiencies of a system, here’s something interesting. The kernel has to be able to tidy up when a user logs out (otherwise there would be a plethora of useless processes clogging up the pipes) so Unix-like systems dutifully groups processes into “sessions”. You can test for session IDs by using "--sid" or as below with “-s”:

# ps -s 1526

If it’s of interest a session can have an associated terminal (of "tty" or "Teletype" varieties) controlling it however only one process can be at running in the foreground. All these components are given numbers to keep the system nice and orderly. A a result we have thread IDs, process IDs, process group IDs and session IDs. Here was you thinking that the ps command didn't have much to do.

If you’re interested in reading a little more about sessions this book’s excerpt is intriguing with sentences such as “Consequently, sessions are largely the business of shells. In fact, nothing else really cares about them.”:


Now picture the scene where we want to check for the parents of a process (we'll look at this in more detail in a second). You can achieve this by using this command:

# ps --ppid 21201

This shows us the processes with a parent process of that ID. In other words we can pinpoint processes that are children of process "21201"in this case.

Having said earlier that usually case-sensitivity shouldn't cause too many headaches I'm going to completely contradict myself with a few examples of why that statement isn’t always true.

Try running my favourite ps command again, its abbreviated output is shown underneath:

# ps -ef


apache   23026 22856  0 Feb26 ?        00:00:00 /usr/sbin/apache2

Now try running the full fat version by using an uppercase "F":

# ps -eF


apache   23026 22856  0 44482  3116   0 Feb26 ?        00:00:00 /usr/sbin/apache2

The differences are that the latter includes SZ, RSS and PSR fields. The first two are memory related whereas PSR shows which CPU the process is using. For more information there’s lots more in the manual:

# man ps

Onwards we can look at another alternative to the “-Z” option which we briefly touched on before:

# ps -efM

unconfined_u:system_r:apache2_t:s0 apache  23031 22856  0 Feb26 ?        00:00:00 /usr/sbin/apache2

A useful BSD throwback I quite like the look of is possibly one of the shortest commands known to

mankind. Have a look at Listing Five.

# ps l


4     0  1414     1  20   0   4064   584 n_tty_ Ss+  tty1       0:00 /sbin/mingetty /dev/tty1

4     0  1416     1  20   0   4064   588 n_tty_ Ss+  tty2       0:00 /sbin/mingetty /dev/tty2

4     0  1418     1  20   0   4064   584 n_tty_ Ss+  tty3       0:00 /sbin/mingetty /dev/tty3

4     0  1420     1  20   0   4064   580 n_tty_ Ss+  tty4       0:00 /sbin/mingetty /dev/tty4

4     0  1426     1  20   0   4064   584 n_tty_ Ss+  tty5       0:00 /sbin/mingetty /dev/tty5

4     0  1430     1  20   0   4064   588 n_tty_ Ss+  tty6       0:00 /sbin/mingetty /dev/tty6

4     0  9896  9558  20   0 191392  2740 poll_s S    pts/1      0:00 sudo -i

4     0  9899  9896  20   0 110496  1960 wait   S    pts/1      0:00 -bash

4     0 10776  9899  20   0 108104   980 -      R+   pts/1      0:00 ps l

Listing Five:Shows us the “long formatted” output which can be embellished with other options, harking from BSD origins


Sometimes even the mighty ps command struggles to precisely refine its output. Imagine a scenario where Java processes are filling up the process table and all you want to do is find their parent so that you can stop (or "kill") the process abruptly. To summarise your information you can use the non-hyphenated “S” switch:

# ps S

This helps to find a parent when its child processes only live for a short period of time.

What about when your Process Table is brimming with processes and you need to list a number of process PIDs at once? As you’d expect there are different ways to achieve this as Listing Six shows when we run the following command:

# ps -p "1 2" -p 3,4


    1 ?        00:00:03 init

    2 ?        00:00:01 kthreadd

    3 ?        00:00:01 migration/0

    4 ?        00:00:20 ksoftirqd/0

Listing Six: We can pick and choose the PIDs that we view in a number of ways

  Everything Is A File

As we mentioned earlier the well-considered principle of basing as much as possible on Unix-like systems around files is a well-advised approach. It could be said that this also extends to the Process Table; consider the treasure trove of gems to be found if you delve deeply into the “procfs” pseudo-filesystem, located in root level “/proc” on your filesystem.

Elements of the innards of /proc can only be read from and not written to. The key file here is

“/etc/sysctl.conf” where you can also change many tunable kernel settings so that they persist following a reboot. One not-so-trivial caveat is that, almost magically, any freshly entered parameters into “/proc” are usually set live instantly so be careful!

Clearly this has a number of advantages. There's no messing about with stopping and starting daemons but be warned that if you're in the slightest bit unsure of making a change (especially to servers) then take a deep breath before doing so. Rest assured that a reboot will revert any changes that you make if they are not entered into the file “/etc/sysctl.conf”.

There are zillions of hidden corridors and secret rooms to explore inside /proc and sadly we will only be able to look at a tiny percentage of them. Needless to say that on a test Virtual Machine or development machine you should spend a long time tweaking, fiddling and breaking your current kernel’s procfs settings. If you’re like me then you might even find it vaguely cathartic and it’s safe to say that the immediacy of the changes really appeal to the impatient.

You can look further into a particular process which you've found using the excellent ps command as we've already seen. The path of Process ID “23022” for example is simply “/proc/23022” in relation to /proc.

If we enter that directory then we are shown (after some complaints that we don't have access to parts of the directory structure if we’re not logged in as root) the contents is presented as follows in Listing Seven:

dr-xr-xr-x.   8 apache apache 0 Feb 26 03:15 .

dr-xr-xr-x. 144 root   root   0 Feb 11 13:31 ..

dr-xr-xr-x.   2 apache apache 0 Feb 26 04:03 attr

-rw-r--r--.   1 root   root   0 Feb 28 08:25 autogroup

-r--------.   1 root   root   0 Feb 28 08:25 auxv

-r--r--r--.   1 root   root   0 Feb 28 08:25 cgroup

--w-------.   1 root   root   0 Feb 28 08:25 clear_refs

-r--r--r--.   1 root   root   0 Feb 26 04:03 cmdline

-rw-r--r--.   1 root   root   0 Feb 28 08:25 comm

-rw-r--r--.   1 root   root   0 Feb 28 08:25 coredump_filter

-r--r--r--.   1 root   root   0 Feb 28 08:25 cpuset

lrwxrwxrwx.   1 root   root   0 Feb 28 08:25 cwd -> /

-r--------.   1 root   root   0 Feb 27 14:01 environ

lrwxrwxrwx.   1 root   root   0 Feb 28 08:25 exe -> /usr/sbin/apache2

dr-x------.   2 root   root   0 Feb 26 04:03 fd

dr-x------.   2 root   root   0 Feb 28 08:25 fdinfo

-r--------.   1 root   root   0 Feb 28 08:25 io

-rw-------.   1 root   root   0 Feb 28 08:25 limits

-rw-r--r--.   1 root   root   0 Feb 28 08:25 loginuid

-r--r--r--.   1 root   root   0 Feb 28 08:25 maps

-rw-------.   1 root   root   0 Feb 28 08:25 mem

-r--r--r--.   1 root   root   0 Feb 28 08:25 mountinfo

-r--r--r--.   1 root   root   0 Feb 28 08:25 mounts

-r--------.   1 root   root   0 Feb 28 08:25 mountstats

dr-xr-xr-x.   4 apache apache 0 Feb 28 08:25 net

dr-x--x--x.   2 root   root   0 Feb 28 08:25 ns

-r--r--r--.   1 root   root   0 Feb 28 08:25 numa_maps

-rw-r--r--.   1 root   root   0 Feb 28 08:25 oom_adj

-r--r--r--.   1 root   root   0 Feb 28 08:25 oom_score

-rw-r--r--.   1 root   root   0 Feb 28 08:25 oom_score_adj

-r--r--r--.   1 root   root   0 Feb 28 08:25 pagemap

-r--r--r--.   1 root   root   0 Feb 28 08:25 personality

lrwxrwxrwx.   1 root   root   0 Feb 28 08:25 root -> /

-rw-r--r--.   1 root   root   0 Feb 28 08:25 sched

-r--r--r--.   1 root   root   0 Feb 28 08:25 schedstat

-r--r--r--.   1 root   root   0 Feb 28 08:25 sessionid

-r--r--r--.   1 root   root   0 Feb 28 07:52 smaps

-r--r--r--.   1 root   root   0 Feb 28 08:25 stack

-r--r--r--.   1 root   root   0 Feb 26 03:15 stat

-r--r--r--.   1 root   root   0 Feb 26 03:15 statm

-r--r--r--.   1 root   root   0 Feb 26 04:03 status

-r--r--r--.   1 root   root   0 Feb 28 08:25 syscall

dr-xr-xr-x.   3 apache apache 0 Feb 27 11:41 task

-r--r--r--.   1 root   root   0 Feb 28 08:25 wchan

Listing Seven: Inside “/proc/23022” we can see a number of pseudo files and directories for our Web server

You might want to think of this content as belonging to runtime system information. It has been said that /proc is a centralised config system for the kernel and it’s easy to see that directory is full of a mountain of information for just one process. As suggested rummaging through these directories and looking up which file does what when you have a spare moment might be described as therapeutic. It’s well worth the effort.

  Pseudo Filesystems

It’s hard to dismiss the power which /proc wields. Be wary though that there’s a lot going on inside your server when it is running, even if no-one is hitting your website. As a result wouldn’t it be sensible to separate the tricksy hardware settings aside from the kernel settings and Process Table?

Continuing onwards with our “Everything Is A File” mantra that’s exactly what Unix-type Operating Systems do. Step forward “/dev”.

When dealing with physical devices, whether connected to the machine or not, we turn to “/dev” and not “/proc”.

Here’s an abbreviated directory listing of “/dev” as shown in Listing Eight.

drwxr-xr-x.  2 root root         740 Feb 11 13:31 block

drwxr-xr-x.  2 root root          80 Feb 11 13:31 bsg

lrwxrwxrwx.  1 root root           3 Feb 11 13:31 cdrom -> sr0

lrwxrwxrwx.  1 root root           3 Feb 11 13:31 cdrw -> sr0

drwxr-xr-x.  2 root root           2.5K Feb 11 13:31 char

crw-------.  1 root root            5,1 Feb 11 13:31 console

lrwxrwxrwx.  1 root root         11 Feb 11 13:31 core -> /proc/kcore

drwxr-xr-x.  4 root root          80 Feb 11 13:31 cpu

crw-rw----.  1 root root          10,  61 Feb 11 13:31 cpu_dma_latency

crw-rw----.  1 root root          10,  62 Feb 11 13:31 crash

drwxr-xr-x.  5 root root         100 Feb 11 13:31 disk

Listing Eight: We can see an abbreviated list of some of the devices which “/dev” deals with

What about another example of what “/dev” can do for us? Lets take an example of the superb “lsof” utility. If you're not familiar with “lsof” then it's unquestionably worth a look at, I’m a big fan. The abbreviation “lsof” stands for “list open files” and its seemingly endless functionality is exceptionally useful.

Listing Nine shows us output from “lsof” when looking up information relating to the directory “/var/log“. We can display this information having run this command:

# lsof +D /var/log/


rsyslogd       1103   root    1w   REG  253,4     2743     19 /var/log/messages

rsyslogd       1103   root    2w   REG  253,4     1906     17 /var/log/cron

rsyslogd       1103   root    4w   REG  253,4      747     18 /var/log/maillog

rsyslogd       1103   root    5w   REG  253,4     1753     27 /var/log/secure

apache2       22856   root    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       22856   root    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23022 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23022 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23024 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23024 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23026 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23026 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23027 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23027 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23028 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23028 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23029 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23029 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23030 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23030 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23031 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23031 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

Listing Nine: The output from the mighty “lsof” looks much like that from the ps command

I am using this “lsof” example because it highlights how a system weaves in and out referencing data from “/proc” and “/dev”. I won’t pretend to understand the nuances.

From its manual the versatile “lsof” transparently informs us of how it gathered such information about that directory, by telling us which files it references:

“/dev/kmem” - the kernel virtual memory device

“/dev/mem” - the physical memory device

“/dev/swap” - the system paging device

From what I can gather these files change between varying Unix versions but hopefully they at least give you a taste of which file is responsible for which task.

As we can see “/dev and “/proc” are useful for all sorts of things whether they be network related, devices (real or virtual), disks (loop disks and physical drives) and much more.

  Counting Processes

Now we’ve whet our whistles with a quick look at the Process Table and pseudo filesystems, “/dev” and “/proc”, let’s explore a few (hopefully useful) unrelated command lines which may save the day at some point in the future.

You might be concerned that a single member of a cluster of identical machines is beginning to give up the ghost. You could check how many processes the system has created since its last reboot by using this command:

# grep processes /proc/stat

Shovel at the ready now look inside the file “/proc/$PID/cmdline” (replacing “$PID” for your Process ID) and there you will find some buried goodies. This file retains the entire and complete command line for a process (almost always at least, zombie processes being one of the exceptions so please don’t fret if your mileage varies).

Sitting alongside that file, in the PID’s directory, is the “cwd” symbolic link (or symlink) to the current working directory of that process.You may need to be logged in as “root”.

In order to discover the current-working-directory of that process run this command:

# cd /proc/$PID/cwd; /bin/pwd

We probably need to be “root” sometimes because the symlink is hidden to normal users. I won't claim to understand all of the intricacies of these pseudo files but if you use the commands “cat” or “less” to view some of these files usually then a little more light is shed on their raison d’ etre.

One such pseudo file (which is less mysterious thanks to its name) is the disk input/output statistics file, named “io”. By running this command and seeing its output in Listing Ten:

# cat io

rchar: 0

wchar: 0

syscr: 0

syscw: 0

read_bytes: 342

write_bytes: 11

cancelled_write_bytes: 0

Listing Ten: Here we can see what this process has been up to in relation to disk activities

Amongst many others there's another useful addition which is the "maps" file. In Listing Eleven we can see the memory regions and their access permissions for a process by using:

# cat maps

7eff839c7000-7eff839de000 r-xp 00000000 fd:01 3221                       /lib64/

7eff839de000-7eff83bde000 ---p 00017000 fd:01 3221                       /lib64/

7eff83bde000-7eff83bdf000 r--p 00017000 fd:01 3221                        /lib64/

7eff83bdf000-7eff83be0000 rw-p 00018000 fd:01 3221                       /lib64/

7eff843ac000-7eff843b3000 r-xp 00000000 fd:01 3201                       /lib64/

7eff843b3000-7eff845b3000 ---p 00007000 fd:01 3201                       /lib64/

7eff845b3000-7eff845b4000 r--p 00007000 fd:01 3201                       /lib64/

7eff845b4000-7eff845b5000 rw-p 00008000 fd:01 3201                      /lib64/

7eff82fb4000-7eff83025000 r-xp 00000000 fd:01 3478                        /lib64/

7eff83025000-7eff83224000 ---p 00071000 fd:01 3478                       /lib64/

7eff83224000-7eff83226000 r--p 00070000 fd:01 3478                       /lib64/

7eff83226000-7eff83227000 rw-p 00072000 fd:01 3478                      /lib64/

Listing Eleven: An extremely abbreviated sample of the “maps” file for a process

Apparently the legend for the permissions is as follows:

r = read

w = write

x = execute

s = shared

p = private (copy on write)

The "maps" file can be useful to see how a process is interacting with the system’s files. Or maybe you’re curious as to which libraries a process needs and you have forgotten the correct options to add to a tool like the super-duper “lsof”.

Any eagle-eyed console champions will spot that the file sizes, having used "ls" to list the

files in the directory, all appear as zero bytes. Clearly these pseudo files are different animals than we’re used to getting our fingers nipped by.

   Kernel Support

Let's now move onto other benefits of /proc and not just on a per-process basis.

For example you might look at the filesystems which were compiled into the kernel by checking out "/proc/filesystems".

In Listing Twelve you can see (an abbreviated list) of what filesystems our kernel supports and even as standard it’s a sizeable list.

nodev sysfs
nodev rootfs
nodev bdev
nodev proc
nodev cgroup
nodev cpuset
nodev tmpfs
nodev devtmpfs
nodev binfmt_misc
nodev debugfs
nodev securityfs
nodev sockfs
nodev usbfs
nodev pipefs
nodev anon_inodefs
nodev inotifyfs
nodev devpts
nodev ramfs
nodev hugetlbfs
nodev pstore
nodev mqueue

Listing Twelve: Here we can see an abbreviated list of the types of file systems supported by the kernel without having to make any tricky changes

You may have heard of an excellent lightweight utility called "vmstat" which reports back dutifully with screeds of useful memory statistics.

You may not, at this juncture, fall off your seat if I tell you that in order to retrieve this useful

information the following files are used. Note the asterisk for all PIDs etc.




Another aspect that the kernel deals with on a system is the hardware ports, you know the kind that accept a keyboard or a mouse. Have a peek at Listing Thirteen having looked into the file "/proc/ioports", again abbreviated for ease.

0000-001f : dma1

0020-0021 : pic1

0040-0043 : timer0

0050-0053 : timer1

0060-0060 : keyboard

0064-0064 : keyboard

0070-0071 : rtc0

0080-008f : dma page reg

00a0-00a1 : pic2

00c0-00df : dma2

00f0-00ff : fpu

Listing Thirteen: Displaying an abbreviated output from the file "/proc/ioports"

As expected we can also look at network information. I’ll leave you to trying some of these yourself. You might want to begin by trying this:

# ls /proc/net

Or for live ARP information (the Address Resolution Protocol) use the “cat” command in front of this:

# cat proc/net/arp

If that output looks familiar then think of how many utilities which you’ve used in the past that refer to this file.

Why stop with ARP? SNMP statistics are also readily available:

# cat /proc/net/snmp

There's a whole heap (no memory puns intended) of other options which we could explore but instead what about a look at changing some of the "sysctl" settings I mentioned earlier too.

Sadly the majority of the powerful changes you can make to a running system's kernel are for another day (there's simply too many to cover) but let's take the popular "/proc/sys" directory for a quick look before we finish.

What about if the system-wide limit of the number of open files for every process on the system is holding you up when you’re trying to launch your latest ground-breaking application?

This setting can cause all sorts of system headaches if it’s abused (and it appears that not only the “root” user can change it on occasion). Warnings aside it’s very useful to arm yourself with this quick fix if it's ever necessary.

Try this command yourself on a test box:

# echo 999999 > /proc/sys/fs/file-max

Once you've conquered that particular outage-inducing issue, taking all the glory as you go, how about checking that the new setting has stuck? Treat it like any other file:

# cat /proc/sys/fs/file-max

I should have of course mentioned that you should check the original value before making any changes (so you can revert the value if need be). My computer used "6815744".

What about another setting? This usually uses a start/stop daemon to make the change, on some systems, but let's do it on the fly:

# echo "ChrisLinux" > /proc/sys/kernel/hostname

In case that you’re wondering you will probably struggle to edit these files with an editor (you may get “file has changed” errors) hence using “echo” as above.

On other distributions you can affect the same change with:

# hostname "ChrisLinux"

I'm sure you get the idea; it’s hardly Computer Science after all.

   The End

We have looked in some detail (but not nearly enough) at the very clever Unix-like process table, procfs and “/dev” briefly. We covered how to read and sometimes manipulate tunable settings and how they integrate with components running on Unix-like systems.

Hopefully this starter-for-ten chapter will give you enough of an insight to know what to look if you encounter issues in the future. Above everything else however it’s important for you to find the time to fire up a test machine, dig into otherwise buried file paths and tweak the available settings to further increase your knowledge.

Incidentally should you ever find yourself at a loss on a rainy afternoon then try running this command:

# /sbin/sysctl -a

That should definitely offer you some food for thought. It helpfully lists all of the tunable kernel parameters on a system, each of which can be altered with "echo", simply read the “dots” as “slashes” to find them inside /proc. Just don’t make ill-advised, arbitrary kernel changes on a production machine or you will likely rue the day!

If you weren’t aware of some of the settings that we’ve covered then I hope you will enjoy your newly gained Sysadmin knowledge and use it creatively in the future.

   Linux Books

If you've enjoyed reading the technical content on this site then please have a look at my Linux books which were both published in 2016 and some of my articles in Linux Magazine and Admin Magazine are available on their respective websites.

Linux Server Security: Hack and Defend by Chris Binnie           Practical Linux Topics by Chris Binnie

Postfix Howtos

I've written three articles on the admin and performance of the powerful Postfix MTA.

Docker Security

I wrote about the heavyweight champion of containers here: Docker Security.

Monitoring Howtos

There's comprehensive articles about the excellent Monit and the flexible nload.