Thanks for not dismissing my rant outright :p

>> There are not really many daemon-specific features in Unix that people
>> should actually use :) Programmers want to be l33t by using daemon(),
>> nohup, syslog, setuid() and stuff like that for their programs but it
>> usually doesn't make any sense....
>
> Syslog as in sending log messages to the system?  I use that every day
> in my backup scripts, which logwatch then displays to assure me that
> my local to disk, local to tape, and offsite ones all ran.

I'm not intimately familiar with how syslog works; it is probably
convenient for some scenarios. The point is that daemons should not be
written against the One True Logging Framework - whether that be syslog,
rotated log files, a binary journal ala systemd, or something else. They
should just write all their logs to stdout/stderr like normal programs,
and the sysadmin can then set up a logger to collect those logs and
route them to the appropriate place (disk files, logs-as-a-service
sites, filter out critical messages and email the sysadmin about them,
etc. etc.)

Daemontools includes a logger that writes rotated text files, but that's
just one of many possibilities. The daemontools remake "runit" has a
logger that can also route the logs to a UDP socket. Since the
supervisor just collects logs from stdout, it's easy to write custom
routers as needed without reprogramming any daemons.

For an idea of how far this architecture can be taken, see
<https://12factor.net/logs>. The 12factor web app architecture is the
backbone of Heroku's awesomeness. It's essentially just "web-scale
daemontools".

> nohup, I've read that systemd has done a number on it.  But isn't it
> in theory useful for starting a batch process and exiting that shell?

It can be useful for one-off hacks, but daemons should not be started
from a user shell at all. A random user login and shell session carries
too much implicit environment that can cause heisenbugs for the daemon.
At worst those bugs can be security holes. Modern supervisors such as
daemontools and systemd (whose core tenets are also more or less copied
from there) start a supervisor at system boot time, and all daemons are
child processes of that supervisor. Since the supervisor process has a
simple, predictable lineage starting from init (PID 1) - indeed, on some
systems the supervisor process _is_ PID 1 - it lives in a predictable
environment which it can pass on to the daemons.

Under this system, daemons are started by sending a command to the
supervisor to "please start daemon foo". So the sysadmin never spawns
daemon processes directly.

Again, Heroku is just an auto-scaled version of this system with a
glitzy web GUI on top. It's simply great stuff.