root.hints - apparmor access error with Bind from PPA

Discussion:

3coma3

2021-06-04 02:45:42 UTC

Dear list:

I've used the PPA at https://launchpad.net/~isc/+archive/ubuntu/bind to
upgrade
bind from 9.11.3+dfsg-1ubuntu1.15 (current version for
bionic-{updates,security}) to 9.16.16-2+ubuntu18.04.1+isc+1

(I was needing to use the validate-except clause and this new version
supports it)

After the upgrade, attempting to start the named service failed with
this error:
Jun 3 22:03:53 top named[19946]: could not configure root hints from
'/usr/share/dns/root.hints': permission denied

Right below that apparmor logs this:

Jun 3 22:03:53 top kernel: [17981.067014] audit: type=1400
audit(1622768633.158:559): apparmor="DENIED" operation="open"
profile="/usr/sbin/named" name="/usr/share/dns/root.hints" pid=19946
comm="isc-worker0000" requested_mask="r" denied_mask="r" fsuid=129 ouid=0

What's puzzling is that the apparmor profile apparently allows the read
@ line 36:

find /etc/apparmor.d -type f | xargs grep -n '/usr/share/dns'
/etc/apparmor.d/usr.sbin.named:36: /usr/share/dns/root.* r,

dpkg -S /etc/apparmor.d/usr.sbin.named
bind9: /etc/apparmor.d/usr.sbin.named

apt-cache policy bind9
bind9:
Installed: 1:9.16.16-2+ubuntu18.04.1+isc+1
Candidate: 1:9.16.16-2+ubuntu18.04.1+isc+1
Version table:
*** 1:9.16.16-2+ubuntu18.04.1+isc+1 500
        500 http://ppa.launchpad.net/isc/bind/ubuntu bionic/main amd64
Packages
        100 /var/lib/dpkg/status
     1:9.11.3+dfsg-1ubuntu1.15 500
        500 http://mirrors.us.kernel.org/ubuntu bionic-updates/main
amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64
Packages
     1:9.11.3+dfsg-1ubuntu1 500
        500 http://mirrors.us.kernel.org/ubuntu bionic/main amd64 Packages

Although the error appears to not be related to file perms, here's for
completeness:

ls -la /usr/share/dns
total 28
drwxr-xr-x   2 root root    55 dic 13 2019 .
drwxr-xr-x 457 root root 12288 jun 3 21:44 ..
-rw-r--r--   1 root root   166 feb 1 2018 root.ds
-rw-r--r--   1 root root 3315 feb 1 2018 root.hints
-rw-r--r--   1 root root   864 feb 1 2018 root.key

It helped me to find a previous report at
https://lists.isc.org/pipermail/bind-users/2020-July/103454.html

And then I ended up solving the problem as Brett did there, by copying
/usr/share/dns to /etc/bind/dns and changing the zone definition.

Still I am reporting this in case it's affecting someone else, and
because maybe you guys have an idea as to what's going on with apparmor
here? I'm not very knowledgeable in it and would appreciate any info /
help to solve the root cause (and maybe learn something).

Thanks in advance

full log:

Jun 3 22:03:53 top systemd[1]: Started BIND Domain Name Server.
Jun 3 22:03:53 top named[19946]: starting BIND 9.16.16-Ubuntu (Stable
Release) <id:0c314d8>
Jun 3 22:03:53 top named[19946]: running on Linux x86_64
5.6.7-050607-generic #202004230933 SMP Thu Apr 23 09:35:28 UTC 2020
Jun 3 22:03:53 top named[19946]: built with '--build=x86_64-linux-gnu'
'--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
'--disable-silent-rules' '
--libdir=/usr/lib/x86_64-linux-gnu'
'--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode'
'--disable-dependency-tracking' '--libdir=/usr/lib/x86_64-linux-gnu'
'--sysconfdir=/etc/bind' '--with-python=python3' '--localstatedir
=/' '--enable-threads' '--enable-largefile' '--with-libtool'
'--enable-shared' '--enable-static' '--with-gost=no'
'--with-openssl=/usr' '--with-gssapi=/usr' '--with-libidn2'
'--with-json-c' '--with-lmdb=/usr' '--with-gnu-ld' '--with-maxmin
ddb' '--with-atf=no' '--enable-ipv6' '--enable-rrl'
'--enable-filter-aaaa' '--disable-native-pkcs11' '--enable-dnstap'
'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2
-fdebug-prefix-map=/build/bind9-suAN9q/bind9-9.16.16=. -fstack-protector-s
trong -Wformat -Werror=format-security -fno-strict-aliasing
-fno-delete-null-pointer-checks -DNO_VERSION_DATE -DDIG_SIGCHASE'
'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now'
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
Jun 3 22:03:53 top named[19946]: running as: named -f -u bind
Jun 3 22:03:53 top named[19946]: compiled by GCC 7.5.0
Jun 3 22:03:53 top named[19946]: compiled with OpenSSL version: OpenSSL
1.1.1 11 Sep 2018
Jun 3 22:03:53 top named[19946]: linked to OpenSSL version: OpenSSL
1.1.1 11 Sep 2018
Jun 3 22:03:53 top named[19946]: compiled with libxml2 version: 2.9.4
Jun 3 22:03:53 top named[19946]: linked to libxml2 version: 20904
Jun 3 22:03:53 top named[19946]: compiled with json-c version: 0.12.1
Jun 3 22:03:53 top named[19946]: linked to json-c version: 0.12.1
Jun 3 22:03:53 top named[19946]: compiled with zlib version: 1.2.11
Jun 3 22:03:53 top named[19946]: linked to zlib version: 1.2.11
Jun 3 22:03:53 top named[19946]:
----------------------------------------------------
Jun 3 22:03:53 top named[19946]: BIND 9 is maintained by Internet
Systems Consortium,
Jun 3 22:03:53 top named[19946]: Inc. (ISC), a non-profit 501(c)(3)
public-benefit
Jun 3 22:03:53 top named[19946]: corporation. Support and training for
BIND 9 are
Jun 3 22:03:53 top named[19946]: available at https://www.isc.org/support
Jun 3 22:03:53 top named[19946]:
----------------------------------------------------
Jun 3 22:03:53 top named[19946]: adjusted limit on open files from 4096
to 1048576
Jun 3 22:03:53 top named[19946]: found 12 CPUs, using 12 worker threads
Jun 3 22:03:53 top named[19946]: using 12 UDP listeners per interface
Jun 3 22:03:53 top named[19946]: using up to 21000 sockets
Jun 3 22:03:53 top named[19946]: loading configuration from
'/etc/bind/named.conf'
Jun 3 22:03:53 top named[19946]: reading built-in trust anchors from
file '/etc/bind/bind.keys'
Jun 3 22:03:53 top named[19946]: looking for GeoIP2 databases in
'/usr/share/GeoIP'
Jun 3 22:03:53 top named[19946]: using default UDP/IPv4 port range:
[32768, 60999]
Jun 3 22:03:53 top named[19946]: using default UDP/IPv6 port range:
[32768, 60999]
Jun 3 22:03:53 top named[19946]: listening on IPv4 interface lo,
127.0.0.1#53
Jun 3 22:03:53 top named[19946]: generating session key for dynamic DNS
Jun 3 22:03:53 top named[19946]: sizing zone task pool based on 25 zones
Jun 3 22:03:53 top named[19946]: could not configure root hints from
'/usr/share/dns/root.hints': permission denied
Jun 3 22:03:53 top named[19946]: loading configuration: permission denied
Jun 3 22:03:53 top named[19946]: exiting (due to fatal error)
Jun 3 22:03:53 top kernel: [17981.067013] kauditd_printk_skb: 24
callbacks suppressed
Jun 3 22:03:53 top kernel: [17981.067014] audit: type=1400
audit(1622768633.158:559): apparmor="DENIED" operation="open"
profile="/usr/sbin/named" name="/usr/share/dns/root.hints" pid=19946
comm="isc-worker0000" requested_mask="r" denied_mask="r" fsuid=129 ouid=0
Jun 3 22:03:53 top systemd[1]: named.service: Main process exited,
code=exited, status=1/FAILURE
Jun 3 22:03:53 top systemd[1]: named.service: Failed with result
'exit-code'.
Jun 3 22:03:53 top systemd[1]: named.service: Service hold-off time
over, scheduling restart.
Jun 3 22:03:53 top systemd[1]: named.service: Scheduled restart job,
restart counter is at 1.
Jun 3 22:03:53 top systemd[1]: Stopped BIND Domain Name Server.

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mail

Timothe Litt

2021-06-04 13:13:28 UTC

Permalink

I'm not an apparmor user - but have you looked at the parent directory
permissions?Â From what you posted, that would be the logical culprit.

In any case, unless you are using a private root zone, since named has
the root nameserver addresses built-in, the use of root.hint is
unnecessary.Â (Even if one or two change addresses before the next
release, as does happen infrequently, once named starts it will ask the
network for the full set.Â It only needs one - of the 13 - to bootstrap
itself.)

There is an argument for running your own root server with a copy of the
root zone - but most small operators don't.Â Simplifying, it makes sense
if you are "far" from the global root servers, have regular outages that
leave a local region intact, or are very concerned about privacy.Â (In
the latter case, qname minimization is likely a better choice.)

It seems that a lot of distributions configure a root.hint out of
habit.Â It's actually a step backwards, since unless you have a process
to update root.hint, your copy is likely to end up being older than
named's built-ins...

It's been a while since I looked, but at that time, a 20ish year old
root.hint had only a couple of IPv4 addresses wrong.Â (Didn't have many
IPv6.)Â root.hint really IS stable - and so, therefore, are the named
built-ins.

Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

Post by 3coma3
I've used the PPA at https://launchpad.net/~isc/+archive/ubuntu/bind to
upgrade
bind from 9.11.3+dfsg-1ubuntu1.15 (current version for
bionic-{updates,security}) to 9.16.16-2+ubuntu18.04.1+isc+1
(I was needing to use the validate-except clause and this new version
supports it)
After the upgrade, attempting to start the named service failed with
JunÂ 3 22:03:53 top named[19946]: could not configure root hints from
'/usr/share/dns/root.hints': permission denied
JunÂ 3 22:03:53 top kernel: [17981.067014] audit: type=1400
audit(1622768633.158:559): apparmor="DENIED" operation="open"
profile="/usr/sbin/named" name="/usr/share/dns/root.hints" pid=19946
comm="isc-worker0000" requested_mask="r" denied_mask="r" fsuid=129 ouid=0
What's puzzling is that the apparmor profile apparently allows the read
find /etc/apparmor.d -type f | xargs grep -n '/usr/share/dns'
/etc/apparmor.d/usr.sbin.named:36:Â /usr/share/dns/root.* r,
dpkg -S /etc/apparmor.d/usr.sbin.named
bind9: /etc/apparmor.d/usr.sbin.named
apt-cache policy bind9
Â Installed: 1:9.16.16-2+ubuntu18.04.1+isc+1
Â Candidate: 1:9.16.16-2+ubuntu18.04.1+isc+1
Â *** 1:9.16.16-2+ubuntu18.04.1+isc+1 500
Â Â Â Â Â Â Â 500 http://ppa.launchpad.net/isc/bind/ubuntu bionic/main amd64
Packages
Â Â Â Â Â Â Â 100 /var/lib/dpkg/status
Â Â Â Â 1:9.11.3+dfsg-1ubuntu1.15 500
Â Â Â Â Â Â Â 500 http://mirrors.us.kernel.org/ubuntu bionic-updates/main
amd64 Packages
Â Â Â Â Â Â Â 500 http://security.ubuntu.com/ubuntu bionic-security/main amd64
Packages
Â Â Â Â 1:9.11.3+dfsg-1ubuntu1 500
Â Â Â Â Â Â Â 500 http://mirrors.us.kernel.org/ubuntu bionic/main amd64 Packages
Although the error appears to not be related to file perms, here's for
ls -la /usr/share/dns
total 28
drwxr-xr-xÂ Â 2 root rootÂ Â Â 55 dic 13Â 2019 .
drwxr-xr-x 457 root root 12288 junÂ 3 21:44 ..
-rw-r--r--Â Â 1 root rootÂ Â 166 febÂ 1Â 2018 root.ds
-rw-r--r--Â Â 1 root rootÂ 3315 febÂ 1Â 2018 root.hints
-rw-r--r--Â Â 1 root rootÂ Â 864 febÂ 1Â 2018 root.key
It helped me to find a previous report at
https://lists.isc.org/pipermail/bind-users/2020-July/103454.html
And then I ended up solving the problem as Brett did there, by copying
/usr/share/dns to /etc/bind/dns and changing the zone definition.
Still I am reporting this in case it's affecting someone else, and
because maybe you guys have an idea as to what's going on with apparmor
here? I'm not very knowledgeable in it and would appreciate any info /
help to solve the root cause (and maybe learn something).
Thanks in advance
JunÂ 3 22:03:53 top systemd[1]: Started BIND Domain Name Server.
JunÂ 3 22:03:53 top named[19946]: starting BIND 9.16.16-Ubuntu (Stable
Release) <id:0c314d8>
JunÂ 3 22:03:53 top named[19946]: running on Linux x86_64
5.6.7-050607-generic #202004230933 SMP Thu Apr 23 09:35:28 UTC 2020
JunÂ 3 22:03:53 top named[19946]: built with '--build=x86_64-linux-gnu'
'--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
'--disable-silent-rules' '
--libdir=/usr/lib/x86_64-linux-gnu'
'--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode'
'--disable-dependency-tracking' '--libdir=/usr/lib/x86_64-linux-gnu'
'--sysconfdir=/etc/bind' '--with-python=python3' '--localstatedir
=/' '--enable-threads' '--enable-largefile' '--with-libtool'
'--enable-shared' '--enable-static' '--with-gost=no'
'--with-openssl=/usr' '--with-gssapi=/usr' '--with-libidn2'
'--with-json-c' '--with-lmdb=/usr' '--with-gnu-ld' '--with-maxmin
ddb' '--with-atf=no' '--enable-ipv6' '--enable-rrl'
'--enable-filter-aaaa' '--disable-native-pkcs11' '--enable-dnstap'
'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2
-fdebug-prefix-map=/build/bind9-suAN9q/bind9-9.16.16=. -fstack-protector-s
trong -Wformat -Werror=format-security -fno-strict-aliasing
-fno-delete-null-pointer-checks -DNO_VERSION_DATE -DDIG_SIGCHASE'
'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now'
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
JunÂ 3 22:03:53 top named[19946]: running as: named -f -u bind
JunÂ 3 22:03:53 top named[19946]: compiled by GCC 7.5.0
JunÂ 3 22:03:53 top named[19946]: compiled with OpenSSL version: OpenSSL
1.1.1Â 11 Sep 2018
JunÂ 3 22:03:53 top named[19946]: linked to OpenSSL version: OpenSSL
1.1.1Â 11 Sep 2018
JunÂ 3 22:03:53 top named[19946]: compiled with libxml2 version: 2.9.4
JunÂ 3 22:03:53 top named[19946]: linked to libxml2 version: 20904
JunÂ 3 22:03:53 top named[19946]: compiled with json-c version: 0.12.1
JunÂ 3 22:03:53 top named[19946]: linked to json-c version: 0.12.1
JunÂ 3 22:03:53 top named[19946]: compiled with zlib version: 1.2.11
JunÂ 3 22:03:53 top named[19946]: linked to zlib version: 1.2.11
----------------------------------------------------
JunÂ 3 22:03:53 top named[19946]: BIND 9 is maintained by Internet
Systems Consortium,
JunÂ 3 22:03:53 top named[19946]: Inc. (ISC), a non-profit 501(c)(3)
public-benefit
JunÂ 3 22:03:53 top named[19946]: corporation.Â Support and training for
BIND 9 are
JunÂ 3 22:03:53 top named[19946]: available at https://www.isc.org/support
----------------------------------------------------
JunÂ 3 22:03:53 top named[19946]: adjusted limit on open files from 4096
to 1048576
JunÂ 3 22:03:53 top named[19946]: found 12 CPUs, using 12 worker threads
JunÂ 3 22:03:53 top named[19946]: using 12 UDP listeners per interface
JunÂ 3 22:03:53 top named[19946]: using up to 21000 sockets
JunÂ 3 22:03:53 top named[19946]: loading configuration from
'/etc/bind/named.conf'
JunÂ 3 22:03:53 top named[19946]: reading built-in trust anchors from
file '/etc/bind/bind.keys'
JunÂ 3 22:03:53 top named[19946]: looking for GeoIP2 databases in
'/usr/share/GeoIP'
[32768, 60999]
[32768, 60999]
JunÂ 3 22:03:53 top named[19946]: listening on IPv4 interface lo,
127.0.0.1#53
JunÂ 3 22:03:53 top named[19946]: generating session key for dynamic DNS
JunÂ 3 22:03:53 top named[19946]: sizing zone task pool based on 25 zones
JunÂ 3 22:03:53 top named[19946]: could not configure root hints from
'/usr/share/dns/root.hints': permission denied
JunÂ 3 22:03:53 top named[19946]: loading configuration: permission denied
JunÂ 3 22:03:53 top named[19946]: exiting (due to fatal error)
JunÂ 3 22:03:53 top kernel: [17981.067013] kauditd_printk_skb: 24
callbacks suppressed
JunÂ 3 22:03:53 top kernel: [17981.067014] audit: type=1400
audit(1622768633.158:559): apparmor="DENIED" operation="open"
profile="/usr/sbin/named" name="/usr/share/dns/root.hints" pid=19946
comm="isc-worker0000" requested_mask="r" denied_mask="r" fsuid=129 ouid=0
JunÂ 3 22:03:53 top systemd[1]: named.service: Main process exited,
code=exited, status=1/FAILURE
JunÂ 3 22:03:53 top systemd[1]: named.service: Failed with result
'exit-code'.
JunÂ 3 22:03:53 top systemd[1]: named.service: Service hold-off time
over, scheduling restart.
JunÂ 3 22:03:53 top systemd[1]: named.service: Scheduled restart job,
restart counter is at 1.
JunÂ 3 22:03:53 top systemd[1]: Stopped BIND Domain Name Server.

3coma3

2021-06-04 15:59:26 UTC

Permalink

Hi Timothe,

Post by Timothe Litt
I'm not an apparmor user - but have you looked at the parent directory
permissions?Â From what you posted, that would be the logical culprit.

Your suggestion helped me indirectly to pinpoint the problem.

I added above line 36 the following (redundant) permissions:

/ r,
/usr r,
/usr/share r,
/usr/share/dns r,

Then reloaded the apparmor profiles, changed back the zone definition,
restarted bind and voila, it started correctly.

What's interesting is that after *undoing* the above permissions and
going back to the original apparmor profile, the permission problem
didn't return and things continued to work instead of failing again (big
question mark here).

I confirmed that the original permission is now in effect by removing it
and reloading the profile, and also by moving /usr/local/share/bind to
somewhere else - both changes caused Bind to fail.

My vague assumption here it's likely that some semantics (or bug) in the
apparmor profile parsing / addition into the kernel was causing the
specific permission to not be effective, until after I made changes that
caused new evaluations of the rules to take place.

So everything looks good now thanks to this simple experiment. There's
some optional research ahead regarding apparmor, if you ask me this is
very counter-intuitive behaviour to say the least.

Post by Timothe Litt
In any case, unless you are using a private root zone, since named has
the root nameserver addresses built-in, the use of root.hint is
unnecessary.Â (Even if one or two change addresses before the next
release, as does happen infrequently, once named starts it will ask
the network for the full set.Â It only needs one - of the 13 - to
bootstrap itself.)
There is an argument for running your own root server with a copy of
the root zone - but most small operators don't.Â Simplifying, it makes
sense if you are "far" from the global root servers, have regular
outages that leave a local region intact, or are very concerned about
privacy.Â (In the latter case, qname minimization is likely a better
choice.)
It seems that a lot of distributions configure a root.hint out of
habit.Â It's actually a step backwards, since unless you have a
process to update root.hint, your copy is likely to end up being older
than named's built-ins...
It's been a while since I looked, but at that time, a 20ish year old
root.hint had only a couple of IPv4 addresses wrong.Â (Didn't have
many IPv6.)Â root.hint really IS stable - and so, therefore, are the
named built-ins.

Thanks for the additional information on root hinting, makes much sense.
Funnily enough I'm now considering disabling the root hint - right after
having solved the original problem :-) .. a classic.

I wasn't aware either about qname minimization, I went on to read about
it and also found it very valuable information. I am indeed concerned
about privacy.

In all, you helped me to solve the issue AND I also learned about Bind,
so I'm very grateful. Brilliant indeed!

Kind regards

Post by Timothe Litt
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users

G.W. Haywood via bind-users

2021-06-04 15:33:16 UTC

Permalink

Hi there,

Jun 3 22:03:53 ... apparmor="DENIED" ... "/usr/share/dns/root.hints" ...

This isn't exactly an answer to your question but I don't think you
need root.hints any more - you can just delete it.

I'm currently using 9.11.26, and I haven't used root.hints for years.
The hints section (zone ".") in my named.conf is just commented out.

https://kb.isc.org/docs/aa-01309

HTH
--
73,
Ged.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

3coma3

2021-06-04 16:18:02 UTC

Permalink

Hi G.W.,

Post by G.W. Haywood via bind-users
Hi there,

Jun 3 22:03:53 ... apparmor="DENIED" ... "/usr/share/dns/root.hints" ...

Your suggestion is in line with what was pointed out by Timothe, also
great explanation from the KB.

It seems this is an extra precaution on the side of Debian, perhaps to
cover some obscure corner case of unreachable root servers? Otherwise I
cannot think of a good reason they include this. I've turned off the
root hint now.

Thanks for the help and info

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users