
Killing a process and all of its descendants - shiroyasha23
http://morningcoffee.io/killing-a-process-and-all-of-its-descendants.html
======
rlpb
This is why cgroups were invented. They solve this problem. Start a process in
its own cgroup, and you can later confidently kill the process and all of its
descendants. Container "technologies" use cgroups extensively, as does systemd
service management.

~~~
menage
[CGroups original developer]

Yes, for tracking processes and reliable resource control. Prior to cgroups,
in Google's Borg cluster management daemon the best strategy I was able to
come up with for reliably and efficiently tracking all the processes in a job
was:

\- assign each job a supplementary group id from a range reserved for Borg,
and tag any processes that were forked into that job with that group id

\- use a kernel netlink connector socket to follow PROC_EVENT_FORK events to
find new processes/threads, and assign them to a job based on the parent
process; if the parent process wasn't found for some reason then query the
process' groups in /proc to find the Borg-added group id to determine which
job it's a part of.

\- if the state gets out of sync (due to a netlink queue overflow, or a daemon
restart) do a full scan of /proc (generally avoided since the overhead for
continually scanning /proc got really high on a busy machine).

That way we always have the full list of pids for a given group. To kill a
job, nuke all the known processes and mark the group id as invalid, so any
racy forks will cause the new processes to show up with a stale Borg group id,
which will cause them to be killed immediately.

This approach might would have had trouble keeping up with a really energetic
fork bomb, but fortunately Borg didn't generally have to deal with actively
malicious jobs, just greedy/misconfigured ones.

Once we'd developed cgroups this got a lot simpler.

~~~
the8472
Was giving each job its own UID not an option? users are the original
privilege separation after all and kill -1 respects that.

~~~
menage
No, because multiple jobs being run by the same end-user could share data
files on the machine, in which case they needed to share the same uid. (Or
alternatively we could have used the extra-gid trick to give shared group
access to files, but that would have involved more on-disk state and hence be
harder to change, versus the job tracking which was more ephemeral.) It's been
a while now, but I have a hazy memory that in the case where a job was the
only one with that uid running on a particular machine, we could make use of
that and avoid needing to check the extra groups.

------
pixelbeat__
The GNU coreutils timeout command encapsulates a lot of these issues. It's
surprisingly difficult to handle all the edge cases and races.

[https://www.maizure.org/projects/decoded-gnu-
coreutils/timeo...](https://www.maizure.org/projects/decoded-gnu-
coreutils/timeout.html)

~~~
castratikron
Cool. Anyone know how they make diagrams like those?

~~~
kevinoid
I was curious as well. According to the author, "All the diagrams were hand-
crafted in PowerPoint."

[https://github.com/MaiZure/coreutils-8.3/issues/1#issuecomme...](https://github.com/MaiZure/coreutils-8.3/issues/1#issuecomment-455773553)

------
srathi
Sometimes I wonder if I'm on a list somewhere for frequent Google searches
such as this one - "How to kill a parent with all the children"!

~~~
stephen82
Personally I use `killall` command and works as expected, at least on Debian.

For instance, when I want to stop conky, all I do is run `killall conky` and
kills all of its processes at once.

Another longer way to do such thing is to run `kill -9 $(pidof conky)` which
kills all returned processes.

~~~
Dylan16807
Just be careful around Solaris!

------
jchw
Bazel-watcher tries to accomplish this on Linux by using process group IDs. It
works, if imperfectly sometimes. I ported this to Windows[1] using Job
Objects. As usual, the Windows API was hell, and I made use of undocumented
syscalls in order to make it work (though that part is partly Go’s fault: when
you start a process, it immediately drops the thread handle on the floor. If
you start a process suspended, that means it’s impossible to resume _using
documented APIs._ )

Thankfully Raymond Chen wrote an article[2] about Job Objects which helped me
figure out the last bits. I genuinely am not sure I could have gotten it right
without that article. There’s so many subtle ways to fail!

[1] [https://github.com/bazelbuild/bazel-
watcher/pull/144/files](https://github.com/bazelbuild/bazel-
watcher/pull/144/files)

[2]
[https://devblogs.microsoft.com/oldnewthing/20130405-00/?p=47...](https://devblogs.microsoft.com/oldnewthing/20130405-00/?p=4743)

~~~
neerajsi
I'm confused by this post, but I'm an ex-dev on the NT kernel.

ResumeThread is a well documented API. As is TerminateJobObject. The one thing
I find a bit baroque is the recently added way to make sure a process is in a
job on creation, which I believe is part is the ProcThreadAttributes
mechanism.

~~~
jchw
Yes, ResumeThread is. NtResumeProcess is what I had to use, because Go
immediately closes the thread handle.

I’m sure it’s safe to rely on NtResumeProcess. I have used it since XP without
issue. But I definitely wish there was a better way to go back from a process
to a thread. The best I could find is using Toolhelp32 to iterate all the
threads on the system, which I believe is just wrapping
NtQuerySystemInformation. Would’ve worked but definitely wasn’t fast.

------
eikenberry
On Linux you can also use prctl with PR_SET_PDEATHSIG to set a signal that
will be sent to all child processes when the parent dies. This is a syscall
you'd need to make from in the program. See 'man 2 prctl'.

------
PopeDotNinja
One low hanging fruit I've thrown into my ssh-ing aliases/functions is is just
throwing 'timeout' on the front. I usually exit cleanly and/or don't spin up
zombie-prone processes, but sometimes I do dumb things, so to exit after a
day...

timeout 86400 ssh me@example.com

I was reading the man page for timeout, and it looks like you can throw some
kill options, but I never needed them, so I never looked for them.

------
adrianmonk
Really nice writing style on this article.

It covers everything that needs to be covered, but it also gets right to the
point. Yet without being overly terse or dry.

And it explains everything clearly. So often the writer is good at
understanding an idea but not conveying it. This lays it out where it's easy
to pick up.

------
k_sze
Can somebody re-explain the last part about nohup propagation to descendant
processes?

I don’t quite get what the implications are. The author also doesn’t seem to
talk about their solution for it in the context.

------
nanaya
>but on BSD and its variants like MacOS, the session ID isn’t present or
always zero

Don't know about macOS, but session ID/pointer does present on FreeBSD and
OpenBSD.

------
psef
Related post with interesting comments:

UNIX one-liner to kill a hanging Firefox process:

[https://jugad2.blogspot.com/2008/09/unix-one-liner-to-
kill-h...](https://jugad2.blogspot.com/2008/09/unix-one-liner-to-kill-hanging-
firefox.html?m=1)

------
bandrami
Isn't this exactly why you can freeze cgroups?

------
mehrdadn
Even _waiting for a process to exit_ is surprisingly hard (impossible?) in
Linux, unless it's your child.

~~~
AnssiH
That is getting considerably easier with the addition of pidfds, though:
[https://lwn.net/SubscriberLink/794707/93ffb35438fd3710/](https://lwn.net/SubscriberLink/794707/93ffb35438fd3710/)

~~~
twic
> Beyond the ability to unambiguously specify which process should be waited
> for, this change will eventually enable another interesting feature: it will
> make it possible to wait for a process that is not a child — something that
> waitid() cannot do now. Since a pidfd is a file descriptor, it can be passed
> to another process via an SCM_RIGHTS datagram in the usual manner. The
> recipient of a pidfd will, once this functionality is completed, be able to
> use it in most of the ways that the parent can to operate on (or wait for)
> the associated process.

So, to wait for a process that is not your child, do you have to get the
relevant pidfd from its parent? In which case, this doesn't help all that
much. Or is there some other way to get pidfds for arbitrary processes?

~~~
mehrdadn
Perhaps pidfd_open(pid, ...)?
[https://lwn.net/Articles/789023/](https://lwn.net/Articles/789023/)

I find it bizarre they called it "pidfd_" rather than just "process_" or
"proc_"... almost seems like they deliberately avoided the obvious?

~~~
cyphar
It's because the object you get is a file descriptor.

In fact it's exactly equivalent to getting a file descriptor for /proc/$pid --
Christian (the person who developed the patchsets) quite cleverly solidified a
trick that some folks knew about for several years (that you could use
/proc/$pid as a race-free way of checking if a process has died if you grabbed
a handle while it was still alive). Before pidfd_send_signal(2) there wasn't a
way to use that "interface" nicely. But now it's a first-class citizen (and
Christian had to fight a lot of battles to get this in over several releases).

It's really cool work and I have high hopes it will be used far and wide
because it solves so many individual problems in one fell swoop.

~~~
mehrdadn
> It's because the object you get is a file descriptor.

It is, but you're opening the object, not the descriptor (which doesn't even
exist yet). When you open a kernel object, you always get back a (new) file
descriptor representing that object. It's against previous naming conventions
too. It's not like mkfifo() was called mkfifofd() or socket() was called
socketfd() or perf_event_open() was called perf_event_fd_open()...

I'm really amused that you're so excited about it and find it so cool. I mean,
I'm not suggesting it isn't awesome that they added it, but to me, it's such a
glaring obvious deficiency that I'm completely flabbergasting that a lot of
battles had to be fought to include it. It should've been added and embraced
with open arms over two decades ago...

~~~
cyphar
> It is, but you're opening the object, not the descriptor (which doesn't even
> exist yet).

pidfd_open(2) is still a proposed interface, and isn't in mainline yet (and if
I'm remembering the ML discussions correctly, it might not even go in any time
soon). The currently-available interfaces are pidfd_send_signal(2),
CLONE_PIDFD, and the new pidfd_poll(2) support for pidfds. In that context,
calling it "pidfd_" makes more sense (since you're operating on existing
handles) and thus pidfd_open(2) also makes sense because otherwise the naming
would be needlessly inconsistent.

There are also several pre-existing APIs that are called process_ (such as
process_vm_{read,write}v(2)) which use pids and not pidfds -- so calling the
new APIs process_ (or even proc_) could lead to confusion. From memory the
first couple of iterations of the patchset changed the name several times
until we landed on pidfd_ and nobody really complained much afterwards.

Also (and now I'm just nitpicking), mkfifo(3) doesn't give you an fd -- it's
just a wrapper around mknod(2). But I do get your point.

> I'm really amused that you're so excited about it and find it so cool.

I might be a little bit more biased towards thinking it's cool, since the
developer is a good friend of mine and we went back and forth on the design
quite a lot (the fact he managed to get /proc/$pid fds to have these features
is pretty remarkable and it's unbelievably cool that it didn't require having
multiple classes of fds -- if you'd have asked me a year ago I would've said
it'd be very hard to get right and would never be merged because it'd be so
invasive). But thinking that it's neat isn't mutually exclusive with thinking
that (something like) it should've been implemented a long time ago.

------
wilsocr88
Off topic, but this headline makes me think "...unto the tenth generation upon
the Earth..."

------
sabujp
advanced programming in the unix environment is an excellent book that dives
more into this

------
sadris
I just use rkill

