BIND 9.11.4 dnstap not capturing updates

[g***@bt.com]

Hello BIND users,
I am running BIND 9.11.4 on CentOS 7, built with support for dnstap. I am testing capturing of all DNS packets, including DNS update packets, but they don't seem to be captured. Here are my named.conf options:

dnstap-output file "/tmp/dnstap.output" ;
dnstap { all ; };

I use nsupdate to send a DDNS update to my zone, which is added successfully. However, the dnstap.output does not record the DNS update. I see only the following three packets captured:

02-Aug-2018 16:24:37.365 AQ ::1:8145 -> ::1:0 UDP 38b test1.dnstaptest.com/IN/SOA
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18817
;; flags:; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;test1.dnstaptest.com. IN SOA

02-Aug-2018 16:24:37.365 AR ::1:8145 <- ::1:0 UDP 104b test1.dnstaptest.com/IN/SOA
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18817
;; flags: qr aa ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;test1.dnstaptest.com. IN SOA

;; AUTHORITY SECTION:
dnstaptest.com. 0 IN SOA centos7-dns-test1. dnsadmin.dnstaptest.com. 6 10800 3600 604800 86400

02-Aug-2018 16:24:37.367 AR ::1:8145 <- ::1:0 UDP 32b dnstaptest.com/IN/SOA
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 22809
;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; ZONE SECTION:
;dnstaptest.com. IN SOA


What is going on here? Why is the DNS Update packet not captured/logged?

Thanks,
Greg Rabil

[g***@bt.com]

Hello BIND users,
(my apologies if this gets posted twice, I first sent to bind-***@isc.org<mailto:bind-***@isc.org> instead of bind-***@lists.isc.org<mailto:bind-***@lists.isc.org>)

I am running BIND 9.11.4 on CentOS 7, built with support for dnstap. I am testing capturing of all DNS packets, including DNS update packets, but they don't seem to be captured. Here are my named.conf options:

dnstap-output file "/tmp/dnstap.output" ;
dnstap { all ; };

I use nsupdate to send a DDNS update to my zone, which is added successfully. However, the dnstap.output does not record the DNS update. I see only the following three packets captured:

02-Aug-2018 16:24:37.365 AQ ::1:8145 -> ::1:0 UDP 38b test1.dnstaptest.com/IN/SOA
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18817
;; flags:; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;test1.dnstaptest.com. IN SOA

02-Aug-2018 16:24:37.365 AR ::1:8145 <- ::1:0 UDP 104b test1.dnstaptest.com/IN/SOA
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18817
;; flags: qr aa ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;test1.dnstaptest.com. IN SOA

;; AUTHORITY SECTION:
dnstaptest.com. 0 IN SOA centos7-dns-test1. dnsadmin.dnstaptest.com. 6 10800 3600 604800 86400

02-Aug-2018 16:24:37.367 AR ::1:8145 <- ::1:0 UDP 32b dnstaptest.com/IN/SOA
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 22809
;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; ZONE SECTION:
;dnstaptest.com. IN SOA


Why is the DNS Update packet not captured/logged?

Thanks,
Greg Rabil


A. Gregory Rabil | Principal Software Architect| BT Diamond IP |
Tel: [phone] | Fax: [phone] |
***@bt.com<mailto:***@bt.com> | http://www.btdiamondip.com
This email contains BT information which may be privileged or confidential. It is meant only for the individual(s) or entity named above. If you are not the intended recipient, note that disclosing, copying, distributing or using this information is prohibited. If you have received this email in error, please let me know immediately on the email above. Thank you. We monitor our email system and may record your emails.
BT Americas Inc. 415 Eagleview Blvd., Suite 112, Exton, PA 19341
BT Americas Inc. is a wholly owned subsidiary of British Telecommunications plc.

[Tony Finch]

***@bt.com <***@bt.com> wrote:

> I use nsupdate to send a DDNS update to my zone, which is added
> successfully. However, the dnstap.output does not record the DNS
> update.

I think (arguably) this is a limitation of the dnstap specification. It's
defined in a Protocol Buffers declaration file (see the link below) and it
only specifies message types for normal queries and responses. The types
correspond roughly to tap points in the code - it isn't as low-level as
you might expect, if you are imagining something that hooks into the
network IO layer.

If you want to record other kinds of messages (UPDATE, NOTIFY, etc.) it
would probably be best to extend the dnstap `Type` enum, and add
corresponding dns_dt_send() calls to BIND's code. But you should check
with Robert Edmonds first :-)

https://gitlab.isc.org/isc-projects/bind9/blob/master/lib/dns/dnstap.proto#L69

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
Sole, Lundy, Fastnet, Irish Sea: Variable 3 or 4. Smooth or slight. Fog
patches. Moderate, occasionally very poor, becoming good for a time.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Darcy, Kevin]

I'm no expert in DNSTAP, but I see this in the output:

opcode: UPDATE

along with proper reinterpretations of the sections:

ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0

How is that "not record[ing} the DNS update"? Are you looking for something
prettier? More detailed?


- Kevin

On Fri, Aug 3, 2018 at 7:36 AM, Tony Finch <***@dotat.at> wrote:

> ***@bt.com <***@bt.com> wrote:
>
> > I use nsupdate to send a DDNS update to my zone, which is added
> > successfully. However, the dnstap.output does not record the DNS
> > update.
>
> I think (arguably) this is a limitation of the dnstap specification. It's
> defined in a Protocol Buffers declaration file (see the link below) and it
> only specifies message types for normal queries and responses. The types
> correspond roughly to tap points in the code - it isn't as low-level as
> you might expect, if you are imagining something that hooks into the
> network IO layer.
>
> If you want to record other kinds of messages (UPDATE, NOTIFY, etc.) it
> would probably be best to extend the dnstap `Type` enum, and add
> corresponding dns_dt_send() calls to BIND's code. But you should check
> with Robert Edmonds first :-)
>
> https://gitlab.isc.org/isc-projects/bind9/blob/master/
> lib/dns/dnstap.proto#L69
>
> Tony.
> --
> f.anthony.n.finch <***@dotat.at> http://dotat.at/
> Sole, Lundy, Fastnet, Irish Sea: Variable 3 or 4. Smooth or slight. Fog
> patches. Moderate, occasionally very poor, becoming good for a time.
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> unsubscribe from this list
>
> bind-users mailing list
> bind-***@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>

[g***@bt.com]

That would be the update response, but not the update request.

Regards,
Greg

From: bind-users [mailto:bind-users-***@lists.isc.org] On Behalf Of Darcy, Kevin
Sent: Friday, August 3, 2018 12:56 PM
To: bind-***@isc.org
Subject: Re: BIND 9.11.4 dnstap not capturing updates

I'm no expert in DNSTAP, but I see this in the output:

opcode: UPDATE

along with proper reinterpretations of the sections:

ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0

How is that "not record[ing} the DNS update"? Are you looking for something prettier? More detailed?

- Kevin

On Fri, Aug 3, 2018 at 7:36 AM, Tony Finch <***@dotat.at<mailto:***@dotat.at>> wrote:
***@bt.com<mailto:***@bt.com> <***@bt.com<mailto:***@bt.com>> wrote:

> I use nsupdate to send a DDNS update to my zone, which is added
> successfully. However, the dnstap.output does not record the DNS
> update.

I think (arguably) this is a limitation of the dnstap specification. It's
defined in a Protocol Buffers declaration file (see the link below) and it
only specifies message types for normal queries and responses. The types
correspond roughly to tap points in the code - it isn't as low-level as
you might expect, if you are imagining something that hooks into the
network IO layer.

If you want to record other kinds of messages (UPDATE, NOTIFY, etc.) it
would probably be best to extend the dnstap `Type` enum, and add
corresponding dns_dt_send() calls to BIND's code. But you should check
with Robert Edmonds first :-)

https://gitlab.isc.org/isc-projects/bind9/blob/master/lib/dns/dnstap.proto#L69

Tony.
--
f.anthony.n.finch <***@dotat.at<mailto:***@dotat.at>> http://dotat.at/
Sole, Lundy, Fastnet, Irish Sea: Variable 3 or 4. Smooth or slight. Fog
patches. Moderate, occasionally very poor, becoming good for a time.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org<mailto:bind-***@lists.isc.org>
https://lists.isc.org/mailman/listinfo/bind-users

[g***@bt.com]

Thanks Tony, I see in the code now where it is not calling dns_dt_send() for the update request.

Regards,
Greg

> I use nsupdate to send a DDNS update to my zone, which is added
> successfully. However, the dnstap.output does not record the DNS
> update.

I think (arguably) this is a limitation of the dnstap specification. It's defined in a Protocol Buffers declaration file (see the link below) and it only specifies message types for normal queries and responses. The types correspond roughly to tap points in the code - it isn't as low-level as you might expect, if you are imagining something that hooks into the network IO layer.

If you want to record other kinds of messages (UPDATE, NOTIFY, etc.) it would probably be best to extend the dnstap `Type` enum, and add corresponding dns_dt_send() calls to BIND's code. But you should check with Robert Edmonds first :-)

https://gitlab.isc.org/isc-projects/bind9/blob/master/lib/dns/dnstap.proto#L69

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/ Sole, Lundy, Fastnet, Irish Sea: Variable 3 or 4. Smooth or slight. Fog patches. Moderate, occasionally very poor, becoming good for a time.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Robert Edmonds]

Tony Finch wrote:
> ***@bt.com <***@bt.com> wrote:
>
> > I use nsupdate to send a DDNS update to my zone, which is added
> > successfully. However, the dnstap.output does not record the DNS
> > update.
>
> I think (arguably) this is a limitation of the dnstap specification. It's
> defined in a Protocol Buffers declaration file (see the link below) and it
> only specifies message types for normal queries and responses. The types
> correspond roughly to tap points in the code - it isn't as low-level as
> you might expect, if you are imagining something that hooks into the
> network IO layer.
>
> If you want to record other kinds of messages (UPDATE, NOTIFY, etc.) it
> would probably be best to extend the dnstap `Type` enum, and add
> corresponding dns_dt_send() calls to BIND's code. But you should check
> with Robert Edmonds first :-)
>
> https://gitlab.isc.org/isc-projects/bind9/blob/master/lib/dns/dnstap.proto#L69

The dnstap `Type` enum values specify where the message is being
observed and whether it's an inbound or outbound message. The _QUERY and
_RESPONSE variants are there as an optimization to avoid having to read
the QR bit from the header, which I now think may have been a premature
optimization [0]. (IIRC, in theory the definition of the flag bits are
opcode-dependent, but I believe in practice every known opcode defines a
QR flag bit.) That is, the *_QUERY `Type` values mean QR=1, not
OPCODE=QUERY.

For UPDATE, I don't think you need to add any new `Type` values. The
responder (an authoritative nameserver?) can record the inbound queries
as AUTH_QUERY and the outbound responses as AUTH_RESPONSE. The initiator
(usually a dedicated tool like nsupdate?) can record the outbound
queries as TOOL_QUERY and the inbound responses as TOOL_RESPONSE.

dnstap doesn't have any `Type` values for an authoritative nameserver
that is an initiator. For NOTIFY, we might need to add AUTH_CLIENT_QUERY
and AUTH_CLIENT_RESPONSE in order to distinguish the initiator and
responder in a NOTIFY transaction between two authoritative nameservers.


[0] Probably 'query_message' and 'response_message' didn't need to be
separate fields either, since no more than one should be set in any
given payload.

--
Robert Edmonds
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[g***@bt.com]

Thanks Robert. I've added a few lines of code to BIND's client.c source module to call dns_dt_send for updates with a type of AUTH_QUERY, and it works as expected.

Is there any reason that you can think that it should not be part of the standard BIND dnstap support? If not, I will gladly contribute my change to the ISC.

Regards,
Greg

-----Original Message-----
From: Robert Edmonds [mailto:***@mycre.ws]
Sent: Friday, August 3, 2018 3:08 PM
To: Tony Finch <***@dotat.at>
Cc: Rabil,AG,A Gregory,JTK2 R <***@bt.com>; bind-***@isc.org
Subject: Re: BIND 9.11.4 dnstap not capturing updates

Tony Finch wrote:
> ***@bt.com <***@bt.com> wrote:
>
> > I use nsupdate to send a DDNS update to my zone, which is added
> > successfully. However, the dnstap.output does not record the DNS
> > update.
>
> I think (arguably) this is a limitation of the dnstap specification.
> It's defined in a Protocol Buffers declaration file (see the link
> below) and it only specifies message types for normal queries and
> responses. The types correspond roughly to tap points in the code - it
> isn't as low-level as you might expect, if you are imagining something
> that hooks into the network IO layer.
>
> If you want to record other kinds of messages (UPDATE, NOTIFY, etc.)
> it would probably be best to extend the dnstap `Type` enum, and add
> corresponding dns_dt_send() calls to BIND's code. But you should check
> with Robert Edmonds first :-)
>
> https://gitlab.isc.org/isc-projects/bind9/blob/master/lib/dns/dnstap.p
> roto#L69

The dnstap `Type` enum values specify where the message is being observed and whether it's an inbound or outbound message. The _QUERY and _RESPONSE variants are there as an optimization to avoid having to read the QR bit from the header, which I now think may have been a premature optimization [0]. (IIRC, in theory the definition of the flag bits are opcode-dependent, but I believe in practice every known opcode defines a QR flag bit.) That is, the *_QUERY `Type` values mean QR=1, not OPCODE=QUERY.

For UPDATE, I don't think you need to add any new `Type` values. The responder (an authoritative nameserver?) can record the inbound queries as AUTH_QUERY and the outbound responses as AUTH_RESPONSE. The initiator (usually a dedicated tool like nsupdate?) can record the outbound queries as TOOL_QUERY and the inbound responses as TOOL_RESPONSE.

dnstap doesn't have any `Type` values for an authoritative nameserver that is an initiator. For NOTIFY, we might need to add AUTH_CLIENT_QUERY and AUTH_CLIENT_RESPONSE in order to distinguish the initiator and responder in a NOTIFY transaction between two authoritative nameservers.


[0] Probably 'query_message' and 'response_message' didn't need to be separate fields either, since no more than one should be set in any given payload.

--
Robert Edmonds
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Robert Edmonds]

***@bt.com wrote:
> Thanks Robert. I've added a few lines of code to BIND's client.c source module to call dns_dt_send for updates with a type of AUTH_QUERY, and it works as expected.
>
> Is there any reason that you can think that it should not be part of the standard BIND dnstap support? If not, I will gladly contribute my change to the ISC.
>
> Regards,
> Greg

I can't think of any reason not to have support for dnstap logging of
UPDATEs on the server side in BIND. It just wasn't a focus for the
original dnstap design work, which was very STD13 focused.

--
Robert Edmonds
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Evan Hunt]

On Fri, Aug 03, 2018 at 04:18:45PM -0400, Robert Edmonds wrote:
> ***@bt.com wrote:
> > Thanks Robert. I've added a few lines of code to BIND's client.c
> > source module to call dns_dt_send for updates with a type of
> > AUTH_QUERY, and it works as expected.
>
> > Is there any reason that you can think that it should not be part of
> > the standard BIND dnstap support? If not, I will gladly contribute my
> > change to the ISC.
>
> I can't think of any reason not to have support for dnstap logging of
> UPDATEs on the server side in BIND. It just wasn't a focus for the
> original dnstap design work, which was very STD13 focused.

The terminology's a little misleading since the QUERY and UPDATE opcodes
are two different things. But I guess the implication here is that for
dnstap purposes, we don't care about opcodes, and "query" is the same
as "request".

I can't think of any reason not to tap update requests, but I do
wonder whether an extension to the type enum would reduce confusion.

--
Evan Hunt -- ***@isc.org
Internet Systems Consortium, Inc.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[g***@bt.com]

I had started down the path of adding a new enum. However, Robert's comment made sense that the granularity of the enums is really for the QR bit and the various points along the query resolution path, so I just used AUTH_QUERY for prototyping purposes.

I could see it being useful to extend the enum to AUTH_QUERY_UPDATE/AUTH_QUERY_RESPONSE, for example, and then extend the filters accordingly. E.g. dnstap { auth query update; };

Regards,
Greg

-----Original Message-----
From: Evan Hunt [mailto:***@isc.org]
Sent: Friday, August 3, 2018 4:34 PM
To: Robert Edmonds <***@mycre.ws>
Cc: Rabil,AG,A Gregory,JTK2 R <***@bt.com>; ***@dotat.at; bind-***@isc.org
Subject: Re: BIND 9.11.4 dnstap not capturing updates

On Fri, Aug 03, 2018 at 04:18:45PM -0400, Robert Edmonds wrote:
> ***@bt.com wrote:
> > Thanks Robert. I've added a few lines of code to BIND's client.c
> > source module to call dns_dt_send for updates with a type of
> > AUTH_QUERY, and it works as expected.
>
> > Is there any reason that you can think that it should not be part of
> > the standard BIND dnstap support? If not, I will gladly contribute
> > my change to the ISC.
>
> I can't think of any reason not to have support for dnstap logging of
> UPDATEs on the server side in BIND. It just wasn't a focus for the
> original dnstap design work, which was very STD13 focused.

The terminology's a little misleading since the QUERY and UPDATE opcodes are two different things. But I guess the implication here is that for dnstap purposes, we don't care about opcodes, and "query" is the same as "request".

I can't think of any reason not to tap update requests, but I do wonder whether an extension to the type enum would reduce confusion.

--
Evan Hunt -- ***@isc.org
Internet Systems Consortium, Inc.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[g***@bt.com]

> I could see it being useful to extend the enum to AUTH_QUERY_UPDATE/AUTH_QUERY_RESPONSE

That should read:
I could see it being useful to extend the enum to AUTH_QUERY_UPDATE/AUTH_RESPONSE_UPDATE

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Tony Finch]

> On 3 Aug 2018, at 20:08, Robert Edmonds <***@mycre.ws> wrote:
>
> dnstap doesn't have any `Type` values for an authoritative nameserver
> that is an initiator. For NOTIFY, we might need to add AUTH_CLIENT_QUERY
> and AUTH_CLIENT_RESPONSE in order to distinguish the initiator and
> responder in a NOTIFY transaction between two authoritative nameservers.

The other queries that auth servers make are to resolve the addresses of name servers that should receive NOTIFYs. I guess they go through the usual resolver machinery, so they are covered by existing dnstap probes, but I haven’t checked...

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at


_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.i