Pidfile generated by /usr/sbin/daemon not usable by rc.d script

[Adam Lindberg]

Hi!

I’m trying to create a minimal rc.d script for a service, and discovered that using /usr/sbin/daemon with the -p flag creates a pidfile which is not readable by /etc/rc.subr. The pidfile is created without a newline, in which case all the service commands stop working. That means, running “stop” or “status” prints nothing. If I add a newline to the file after the fact, they all start working again. Running the service script with debug output, shows the ‘read’ builtin halting the execution of the script when trying to read the pidfile.

Attached is the service script (“foo.rcscript”) and the output when it is broken, i.e. no newline (“service-foo-status-without-newline.output”) and when it works after manually adding the newline to the pidfile (“service-foo-status-with-newline.output”).

The service script as a ‘sed’ line commented out, which if enabled will make the service script work. However, it feels like a bit of a hack to modify the pidfile created by ‘daemon’ manually afterwards.

Is there a better way to make this work? Is this a bug somehow, either in the service script I wrote or in the rc.subr functions? Any insight would be helpful.

Thanks in advance!
Adam

--
Adam Lindberg | Backend Engineer
Wooga GmbH | SaarbrÃŒcker Str. 38 | D-10405 Berlin

Place of business: Berlin
Registered at the local court Berlin-Charlottenburg, HRB 117846 B
Managing Directors: Jens Begemann, Philipp Möser, Jan Miczaika

[RW via freebsd-questions]

On Wed, 25 May 2016 11:51:31 +0200

Hi!
I’m trying to create a minimal rc.d script for a service, and
discovered that using /usr/sbin/daemon with the -p flag creates a
pidfile which is not readable by /etc/rc.subr. The pidfile is created
without a newline, in which case all the service commands stop
working. That means, running “stop” or “status” prints nothing. If I
add a newline to the file after the fact, they all start working
again. Running the service script with debug output, shows the ‘read’
builtin halting the execution of the script when trying to read the
pidfile.

This is strange because powerd.pid works without a newline.

I wouldn't have expected it to work at all because you've set

command=yes

I don't know if there's a reason for that, but AFAIK either command or
procname is needed in addition to the pid to verify it's the correct
process - they're matched against the output of ps.

[RW via freebsd-questions]

On Wed, 25 May 2016 14:13:47 +0100

Post by RW via freebsd-questions
On Wed, 25 May 2016 11:51:31 +0200

Hi!
I’m trying to create a minimal rc.d script for a service, and
discovered that using /usr/sbin/daemon with the -p flag creates a
pidfile which is not readable by /etc/rc.subr. The pidfile is
created without a newline, in which case all the service commands
stop working. That means, running “stop” or “status” prints
nothing. If I add a newline to the file after the fact, they all
start working again. Running the service script with debug output,
shows the ‘read’ builtin halting the execution of the script when
trying to read the pidfile.

This is strange because powerd.pid works without a newline.

Actually on closer inspection it appears that rc.d/powerd doesn't
define a pidfile - it's getting shutdown based on the command name. I
think you should submit a PR.

[Adam Lindberg]

Hi,

I don’t know what powerd is and how it factors into this. :-)

We’re trying to create a service script for our own program (an Erlang VM) and hit this “weird behavior”. We were using command=yes as a way to get some program running in the background with daemon wrapped around it. We were able to reproduce the problem with just this minor rc.d script and running ‘yes’ (the executable) as our service. This problem happens no matter what the command is, as far as we can tell.

Cheers,
Adam

--
Adam Lindberg | Backend Engineer
Wooga GmbH | Saarbrücker Str. 38 | D-10405 Berlin

Place of business: Berlin
Registered at the local court Berlin-Charlottenburg, HRB 117846 B
Managing Directors: Jens Begemann, Philipp Möser, Jan Miczaika

Post by RW via freebsd-questions
On Wed, 25 May 2016 14:13:47 +0100

Post by RW via freebsd-questions
On Wed, 25 May 2016 11:51:31 +0200

Hi!
I’m trying to create a minimal rc.d script for a service, and
discovered that using /usr/sbin/daemon with the -p flag creates a
pidfile which is not readable by /etc/rc.subr. The pidfile is
created without a newline, in which case all the service commands
stop working. That means, running “stop” or “status” prints
nothing. If I add a newline to the file after the fact, they all
start working again. Running the service script with debug output,
shows the ‘read’ builtin halting the execution of the script when
trying to read the pidfile.

This is strange because powerd.pid works without a newline.

Actually on closer inspection it appears that rc.d/powerd doesn't
define a pidfile - it's getting shutdown based on the command name. I
think you should submit a PR.
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions

[RW via freebsd-questions]

On Wed, 25 May 2016 17:12:36 +0200

Post by Adam Lindberg
Hi,
I don’t know what powerd is and how it factors into this. :-)

It's an example of a daemon that generates a pid file without a
newline, and yet "stop" and "status" work.

It turns out that /etc/rc.d/powerd doesn't define pidfile, in which
case rc.subr finds the pid from the output of ps. So it's not a
counter-example and it does appear to be a bug that rc.subr can't cope
with a pid file without a newline.

You could use the same method, but using sed seems safer.

[Adam Lindberg]

Okay, thanks, that makes sense.

1. Is the sed “hack” okay for now? Are there any risks with modifying the pid file after it’s been created (too early, too late etc.)?
2. If this is a bug in /etc/rc.subr, how do I report it? I’m new to FreeBSD. :-)

Cheers,
Adam

--
Adam Lindberg | Backend Engineer
Wooga GmbH | Saarbrücker Str. 38 | D-10405 Berlin

Place of business: Berlin
Registered at the local court Berlin-Charlottenburg, HRB 117846 B
Managing Directors: Jens Begemann, Philipp Möser, Jan Miczaika

Post by RW via freebsd-questions
On Wed, 25 May 2016 17:12:36 +0200

Post by Adam Lindberg
Hi,
I don’t know what powerd is and how it factors into this. :-)

It's an example of a daemon that generates a pid file without a
newline, and yet "stop" and "status" work.
It turns out that /etc/rc.d/powerd doesn't define pidfile, in which
case rc.subr finds the pid from the output of ps. So it's not a
counter-example and it does appear to be a bug that rc.subr can't cope
with a pid file without a newline.
You could use the same method, but using sed seems safer.
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions

[Ian Smith]

In freebsd-questions Digest, Vol 625, Issue 4, Message: 19

Post by RW via freebsd-questions
On Wed, 25 May 2016 17:12:36 +0200

Hi,
I don?t know what powerd is and how it factors into this. :-)

It's an example of a daemon that generates a pid file without a
newline, and yet "stop" and "status" work.
It turns out that /etc/rc.d/powerd doesn't define pidfile, in which
case rc.subr finds the pid from the output of ps. So it's not a
counter-example and it does appear to be a bug that rc.subr can't cope
with a pid file without a newline.
You could use the same method, but using sed seems safer.

Just as another data point, from a 9.3 system (if relevant), where the
majority of pidfiles have no trailing newline; no obvious consistency:

***@x200:~ # sh -c 'for i in /var/run/*.pid ; do echo $i; hd $i; done'
/var/run/consolekit.pid
00000000 39 31 31 0a |911.|
00000004
/var/run/cron.pid
00000000 38 35 31 |851|
00000003
/var/run/devd.pid
00000000 34 32 37 |427|
00000003
/var/run/dhclient.em0.pid
00000000 33 34 38 |348|
00000003
/var/run/moused.pid
00000000 37 31 39 |719|
00000003
/var/run/ntpd.pid
00000000 36 39 32 |692|
00000003
/var/run/powerd.pid
00000000 36 39 35 |695|
00000003
/var/run/sendmail.pid
00000000 38 32 34 0a 2f 75 73 72 2f 73 62 69 6e 2f 73 65 |824./usr/sbin/se|
00000010 6e 64 6d 61 69 6c 20 2d 4c 20 73 6d 2d 6d 74 61 |ndmail -L sm-mta|
00000020 20 2d 62 64 20 2d 71 33 30 6d 20 2d 4f 44 61 65 | -bd -q30m -ODae|
00000030 6d 6f 6e 50 6f 72 74 4f 70 74 69 6f 6e 73 3d 41 |monPortOptions=A|
00000040 64 64 72 3d 6c 6f 63 61 6c 68 6f 73 74 0a |ddr=localhost.|
0000004e
/var/run/sshd.pid
00000000 38 30 31 0a |801.|
00000004
/var/run/syslog.pid
00000000 35 39 30 |590|
00000003

check_pidfile in /etc/rc.subr (at 9.3 and in head I see) uses:
read _pid _junk < $_pidfile
which extracts pids ok with or without the newline. Don't suppose the
behaviour of read in sh(1) might have changed in terms of separating
words or in whitespace reckoning? Otherwise this is rather mysterious.

cheers, Ian

[Adam Lindberg]

Sorry for the late reply.

What we observed was that the `read _pid _junk < $_pidfile` line did indeed work on the command line, after sourcing /etc/rc.subr. For some strange reason it seems not to work from inside the service script for us.

Cheers,
Adam

--
Adam Lindberg | Backend Engineer
Wooga GmbH | Saarbrücker Str. 38 | D-10405 Berlin

Place of business: Berlin
Registered at the local court Berlin-Charlottenburg, HRB 117846 B
Managing Directors: Jens Begemann, Philipp Möser, Jan Miczaika

Post by Ian Smith
In freebsd-questions Digest, Vol 625, Issue 4, Message: 19

Post by RW via freebsd-questions
On Wed, 25 May 2016 17:12:36 +0200

Hi,
I don?t know what powerd is and how it factors into this. :-)

It's an example of a daemon that generates a pid file without a
newline, and yet "stop" and "status" work.
It turns out that /etc/rc.d/powerd doesn't define pidfile, in which
case rc.subr finds the pid from the output of ps. So it's not a
counter-example and it does appear to be a bug that rc.subr can't cope
with a pid file without a newline.
You could use the same method, but using sed seems safer.

Just as another data point, from a 9.3 system (if relevant), where the
/var/run/consolekit.pid
00000000 39 31 31 0a |911.|
00000004
/var/run/cron.pid
00000000 38 35 31 |851|
00000003
/var/run/devd.pid
00000000 34 32 37 |427|
00000003
/var/run/dhclient.em0.pid
00000000 33 34 38 |348|
00000003
/var/run/moused.pid
00000000 37 31 39 |719|
00000003
/var/run/ntpd.pid
00000000 36 39 32 |692|
00000003
/var/run/powerd.pid
00000000 36 39 35 |695|
00000003
/var/run/sendmail.pid
00000000 38 32 34 0a 2f 75 73 72 2f 73 62 69 6e 2f 73 65 |824./usr/sbin/se|
00000010 6e 64 6d 61 69 6c 20 2d 4c 20 73 6d 2d 6d 74 61 |ndmail -L sm-mta|
00000020 20 2d 62 64 20 2d 71 33 30 6d 20 2d 4f 44 61 65 | -bd -q30m -ODae|
00000030 6d 6f 6e 50 6f 72 74 4f 70 74 69 6f 6e 73 3d 41 |monPortOptions=A|
00000040 64 64 72 3d 6c 6f 63 61 6c 68 6f 73 74 0a |ddr=localhost.|
0000004e
/var/run/sshd.pid
00000000 38 30 31 0a |801.|
00000004
/var/run/syslog.pid
00000000 35 39 30 |590|
00000003
read _pid _junk < $_pidfile
which extracts pids ok with or without the newline. Don't suppose the
behaviour of read in sh(1) might have changed in terms of separating
words or in whitespace reckoning? Otherwise this is rather mysterious.
cheers, Ian

[Ian Smith]

Post by Adam Lindberg
Sorry for the late reply.
What we observed was that the `read _pid _junk < $_pidfile` line did
indeed work on the command line, after sourcing /etc/rc.subr. For
some strange reason it seems not to work from inside the service
script for us.

I just had another look at your foo.rcscript attachment, and bounced
through all in {/usr/local,}/etc/rc.d for examples. As RW said earlier,
'command=yes' appears unlike all the others, in that it does not provide
the full pathname of the executable. I don't know if that matters here.

Also, none of the others (here) need daemon(8) to run, in background or
otherwise .. are you sure that you require its functionality for 'foo'?

For one thing, it seems that daemon keeps the -p pidfile locked during
execution of the process; might that affect service status, stop, etc?

Otherwise I have no idea; a PR may indeed be worthwhile pursuing.

cheers, Ian

[RW via freebsd-questions]

On Wed, 1 Jun 2016 22:58:28 +1000 (EST)

Post by Ian Smith

Post by Adam Lindberg
Sorry for the late reply.
What we observed was that the `read _pid _junk < $_pidfile` line
did indeed work on the command line, after sourcing /etc/rc.subr.
For some strange reason it seems not to work from inside the
service script for us.

I just had another look at your foo.rcscript attachment, and bounced
through all in {/usr/local,}/etc/rc.d for examples. As RW said
earlier, 'command=yes' appears unlike all the others, in that it does
not provide the full pathname of the executable. I don't know if
that matters here.

I did misunderstand that. When I saw 'command=yes' it looked like the
OP was trying to treat command as a flag.

Post by Ian Smith
Also, none of the others (here) need daemon(8) to run, in background
or otherwise .. are you sure that you require its functionality for
'foo'?

Most daemons were written as such. daemon(8) is there for those that
weren't or were written in a scripting language that doesn't support
the double fork.

Post by Ian Smith
For one thing, it seems that daemon keeps the -p pidfile locked
during execution of the process; might that affect service status,
stop, etc?

I think the problem is pretty straightforward. If you run this

read _pid _junk < $_pidfile

and $_pidfile doesn't end in a newline, read will wait for one, just as
it would if you typed in a line and didn't hit return.

[Ian Smith]

In freebsd-questions Digest, Vol 626, Issue 4, Message: 3

Post by RW via freebsd-questions
On Wed, 1 Jun 2016 22:58:28 +1000 (EST)

[..]

Post by RW via freebsd-questions

Post by Ian Smith
Also, none of the others (here) need daemon(8) to run, in background
or otherwise .. are you sure that you require its functionality for
'foo'?

Most daemons were written as such. daemon(8) is there for those that
weren't or were written in a scripting language that doesn't support
the double fork.

Ah right, fair enough and thanks.

Post by RW via freebsd-questions

Post by Ian Smith
For one thing, it seems that daemon keeps the -p pidfile locked
during execution of the process; might that affect service status,
stop, etc?

I think the problem is pretty straightforward. If you run this
read _pid _junk < $_pidfile
and $_pidfile doesn't end in a newline, read will wait for one, just as
it would if you typed in a line and didn't hit return.

But it doesn't wait on 9.3 stable of last October at least with this
small set of pidfiles, some with & some without trailing newline, thus:

% cat ~/bin/pidtest.sh
#!/bin/sh
# 2/6/16 re RW's msg in questions@
for _pidfile in /var/run/*.pid; do
echo $_pidfile
hd $_pidfile | head -1
read _pid _junk < $_pidfile
echo "_pid: '$_pid' _junk: '$_junk'"
done
echo done

***@x200:~ # pidtest.sh
/var/run/consolekit.pid
00000000 39 31 31 0a |911.|
_pid: '911' _junk: ''
/var/run/cron.pid
00000000 38 35 31 |851|
_pid: '851' _junk: ''
/var/run/devd.pid
00000000 34 32 37 |427|
_pid: '427' _junk: ''
/var/run/dhclient.em0.pid
00000000 33 34 38 |348|
_pid: '348' _junk: ''
/var/run/moused.pid
00000000 37 31 39 |719|
_pid: '719' _junk: ''
/var/run/ntpd.pid
00000000 36 39 32 |692|
_pid: '692' _junk: ''
/var/run/powerd.pid
00000000 36 39 35 |695|
_pid: '695' _junk: ''
/var/run/sendmail.pid
00000000 38 32 34 0a 2f 75 73 72 2f 73 62 69 6e 2f 73 65 |824./usr/sbin/se|
_pid: '824' _junk: ''
/var/run/sshd.pid
00000000 38 30 31 0a |801.|
_pid: '801' _junk: ''
/var/run/syslog.pid
00000000 35 39 30 |590|
_pid: '590' _junk: ''
done

No ptoblem interactively either; one of each (that non-root can read):

$ read _pid _junk < /var/run/ntpd.pid
$ echo $_pid
692
$ echo $_junk

$ read _pid _junk < /var/run/sshd.pid
$ echo $_pid
801
$ echo $_junk

So maybe this issue is something new(er) in FreeBSD somewhere?

cheers, Ian (please cc me, digests can take a day)

[Adam Lindberg]

We are using FreeBSD 10.1 STABLE. We are trying to demonize an Erlang program. Erlang itself has support for detaching but cannot write pidfiles. That’s why we’re trying to use daemon to wrap it.

My experience is also that read works in all cases without a newline, except from inside the rc script. That I cannot explain.

Cheers,
Adam

--
Adam Lindberg | Backend Engineer
Wooga GmbH | Saarbrücker Str. 38 | D-10405 Berlin

Place of business: Berlin
Registered at the local court Berlin-Charlottenburg, HRB 117846 B
Managing Directors: Jens Begemann, Philipp Möser, Jan Miczaika

Post by Ian Smith
In freebsd-questions Digest, Vol 626, Issue 4, Message: 3

Post by RW via freebsd-questions
On Wed, 1 Jun 2016 22:58:28 +1000 (EST)

[..]

Post by RW via freebsd-questions

Post by Ian Smith
Also, none of the others (here) need daemon(8) to run, in background
or otherwise .. are you sure that you require its functionality for
'foo'?

Most daemons were written as such. daemon(8) is there for those that
weren't or were written in a scripting language that doesn't support
the double fork.

Ah right, fair enough and thanks.

Post by RW via freebsd-questions

Post by Ian Smith
For one thing, it seems that daemon keeps the -p pidfile locked
during execution of the process; might that affect service status,
stop, etc?

I think the problem is pretty straightforward. If you run this
read _pid _junk < $_pidfile
and $_pidfile doesn't end in a newline, read will wait for one, just as
it would if you typed in a line and didn't hit return.

But it doesn't wait on 9.3 stable of last October at least with this
% cat ~/bin/pidtest.sh
#!/bin/sh
for _pidfile in /var/run/*.pid; do
echo $_pidfile
hd $_pidfile | head -1
read _pid _junk < $_pidfile
echo "_pid: '$_pid' _junk: '$_junk'"
done
echo done
/var/run/consolekit.pid
00000000 39 31 31 0a |911.|
_pid: '911' _junk: ''
/var/run/cron.pid
00000000 38 35 31 |851|
_pid: '851' _junk: ''
/var/run/devd.pid
00000000 34 32 37 |427|
_pid: '427' _junk: ''
/var/run/dhclient.em0.pid
00000000 33 34 38 |348|
_pid: '348' _junk: ''
/var/run/moused.pid
00000000 37 31 39 |719|
_pid: '719' _junk: ''
/var/run/ntpd.pid
00000000 36 39 32 |692|
_pid: '692' _junk: ''
/var/run/powerd.pid
00000000 36 39 35 |695|
_pid: '695' _junk: ''
/var/run/sendmail.pid
00000000 38 32 34 0a 2f 75 73 72 2f 73 62 69 6e 2f 73 65 |824./usr/sbin/se|
_pid: '824' _junk: ''
/var/run/sshd.pid
00000000 38 30 31 0a |801.|
_pid: '801' _junk: ''
/var/run/syslog.pid
00000000 35 39 30 |590|
_pid: '590' _junk: ''
done
$ read _pid _junk < /var/run/ntpd.pid
$ echo $_pid
692
$ echo $_junk
$ read _pid _junk < /var/run/sshd.pid
$ echo $_pid
801
$ echo $_junk
So maybe this issue is something new(er) in FreeBSD somewhere?
cheers, Ian (please cc me, digests can take a day)
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions

[Ian Smith]

Post by Adam Lindberg
We are using FreeBSD 10.1 STABLE. We are trying to demonize an Erlang
program. Erlang itself has support for detaching but cannot write
pidfiles. Thatÿÿs why weÿÿre trying to use daemon to wrap it.
My experience is also that read works in all cases without a newline,
except from inside the rc script. That I cannot explain.

I'm mystified too. Sounds like raising a bug might be your best bet; it
may be come across by someone who knows more or recognises something.

Looking at your sh -ex runs, good and fail, the only thing I wonder
about is whether running it without -e might make any difference ..
although the sourced /etc/rc.subr certainly should be -e safe.

cheers, Ian