How to Stop the "unexpected RCODE (SERVFAIL)" Messages?

Discussion:

imacat

2006-02-04 18:13:47 UTC

Dear all,
Hi. I'm new here. I'm running BIND 9.3.2, with sendmail on a same
box. The "unexpected RCODE (SERVFAIL)" message is blowing up my syslog.
Just tonight I heard my harddisk scratching hardly again. I checked the
syslog. There are 1277 "unexpected RCODE (SERVFAIL)" in one hour:

imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:'
Feb 4 21:00:32 rinse named[2037]: unexpected RCODE (SERVFAIL) resolving
'194.183.255.216.in-addr.arpa/PTR/IN': 127.0.0.1#50053
Feb 4 21:00:34 leaf named[2008]: unexpected RCODE (SERVFAIL) resolving
'194.183.255.216.in-addr.arpa/PTR/IN': 168.95.1.1#53
...
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:' | wc -l
1277
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:00' | wc -l
24
imacat at rinse ~ %

I searched the archive. On Sep 2, 2004 Kevin Darcy said that these
are problems of others' name servers. There is nothing I can do here.
But, hey, this is my harddisk! My syslog file is growing up enormously,
and I feel the life-span of my root harddisk shortened on every scratch.
This is sort of DoS. My heart aches on every such scratch~ >_<

Is there any way I can suppress these messages? These messages are
irrevelent to my system at all. I would rather like to keep more
relevant information in my syslog, for ex, other type of failurs than
SERVFAIL, or even supress the whole 'unexpected RCODE'. Or is there
something like the "limit" target in Linux iptables/netfilter that can
prevent my harddisk from blowing up? Can anyone teach me how I can do
on this? Thank you.

--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt

<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

Stephane Bortzmeyer

2006-02-04 21:12:12 UTC

Permalink

On Sun, Feb 05, 2006 at 02:13:47AM +0800,

...

Post by imacat
My syslog file is growing up enormously,
and I feel the life-span of my root harddisk shortened on every scratch.

Reading the source code of BIND (because I cannot find it in the
documentation), it seems that this message is logged in the category
"lame-servers". So, adding:

logging {
...
category lame-servers { null; };

should be enough (I never understood that category, anyway, since I
don't care in errors in other domains, there are much too common).

You can read the BIND ARM to learn more about categories and
logging. See chapter "logging Statement Grammar", which includes a
table of available categories.

imacat

2006-02-05 16:13:30 UTC

Permalink

On Sat, 4 Feb 2006 22:12:12 +0100

Post by Stephane Bortzmeyer
On Sun, Feb 05, 2006 at 02:13:47AM +0800,

...

Post by imacat
My syslog file is growing up enormously,
and I feel the life-span of my root harddisk shortened on every scratch.

Reading the source code of BIND (because I cannot find it in the
documentation), it seems that this message is logged in the category
logging {
category lame-servers { null; };
You can read the BIND ARM to learn more about categories and
logging. See chapter "logging Statement Grammar", which includes a
table of available categories.

It works! It's magic! It's a miracle! *^_^*

Sorry I forgot that BIND ARM might have an answer. Actually, now
that you are talking about "lame-servers", I remembered this. Some how
I heard of it years ago, I did not disable it, because I was told to be
supposed to be solving the problem instead of supressing the error
messages.

I have digged into the source. I found that disabling "lame-servers"
will also disable other type of "lame-servers" messages, too. I'm a
little concerned to this. I do not want to supress error messages of my
own servers. But currently, I have to save my harddisk by doing so, or
my harddisk will die sooner or later. My harddisk is of more importance.

Is there any BIND member talking care of this DoS issue? Maybe a
new category, say, "remote-lame-servers"?

In fact, it's interesting that I got 1/5 of the syslog messages that
are "unexpect RCODE (SERVFAIL)". And I haven't got any other RCODE
other than SERVFAIL. I got no "unexpected OPCODE", too.

imacat at rinse ~ % gunzip -c messages.200601.gz| wc -l
286007
imacat at rinse ~ % gunzip -c messages.200601.gz| grep 'unexpected RCODE
(SERVFAIL)' | wc -l
46724
imacat at rinse ~ % gunzip -c messages.200601.gz| grep 'unexpected RCODE' |
grep -v SERVFAIL | wc -l
0
imacat at rinse ~ % gunzip -c messages.200601.gz| grep 'unexpected OPCODE'
| wc -l
0
imacat at rinse ~ %

--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt

<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

Stephane Bortzmeyer

2006-02-05 17:02:51 UTC

Permalink

On Mon, Feb 06, 2006 at 12:13:30AM +0800,

Some how I heard of it years ago, I did not disable it, because I
was told to be supposed to be solving the problem instead of
supressing the error messages.

This is a good advice if the error is *local*. If it is remote, you
cannot do a lot: sure you can write people but my experience with
email to DNS administrators is that you'll get a bounce or simply a
complete silence (TLD administrators are not different in that
respect). The DNS administrators who reply to email are typically not
the ones who have configuration problems :-}

I do not want to supress error messages of my own servers.

For your own servers, rather than relying on BIND errors, why not
using Zonecheck? http://www.zonecheck.fr/

imacat

2006-02-05 18:14:05 UTC

Permalink

On Sun, 5 Feb 2006 18:02:51 +0100

Post by Stephane Bortzmeyer
On Mon, Feb 06, 2006 at 12:13:30AM +0800,

Some how I heard of it years ago, I did not disable it, because I
was told to be supposed to be solving the problem instead of
supressing the error messages.

Now that I know this is all about the "lame servers" problem, in
1997. I have a little search on this issue. Is it supposed to be
something like a valid SOA record?

Well, all the SOA records of the domains hosted on my servers are
valid. I think it's a important to be a good citizen. But if BIND want
to enforce that, could you please do some other thing, for example,
notifying the TLD administrators automatically (without using mail,
please), instead of making DoS attack to the harddisk of th good
citizens? With one of my previous servers that has a /var/log partition
only 500MB, it'll be filled up in a very, very short time. And not to
mention the lifespan of the harddisk itself.

If I were to attack some DNS server, knowing that it does not
disable the lame-server messages, all I have to do is to register a
domain, configure an improper SOA record, and trigger it to do DNS query.
Bla! Either the hard disk will be filled up, or die completely.

This is somehow security-related, isn't it?

By the way, I have already made a patch that bypass all the
"unexpected RCODE (SERVFAIL)" messages, that applies to BIND 9.3.2. It
runs well without disabling the "lame-server" messages. I attached it
below. Please tell me if there is any problem.

diff -r -u bind-9.3.2.orig/lib/dns/resolver.c bind-9.3.2/lib/dns/resolver.c
--- bind-9.3.2.orig/lib/dns/resolver.c 2005-10-14 09:38:48.000000000 +0800
+++ bind-9.3.2/lib/dns/resolver.c 2006-02-06 00:33:34.000000000 +0800
@@ -1663,6 +1663,10 @@
if (reason == DNS_R_LAME) /* already logged */
return;

+ if (reason == DNS_R_UNEXPECTEDRCODE
+ && fctx->rmessage->rcode == dns_rcode_servfail)
+ return; /* no "unexpected RCODE (SERVFAIL)" */
+
if (reason == DNS_R_UNEXPECTEDRCODE) {
isc_buffer_init(&b, code, sizeof(code) - 1);
dns_rcode_totext(fctx->rmessage->rcode, &b);

--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt

<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

Mark Andrews

2006-02-04 22:35:48 UTC

Permalink

Post by imacat
Dear all,
Hi. I'm new here. I'm running BIND 9.3.2, with sendmail on a same
box. The "unexpected RCODE (SERVFAIL)" message is blowing up my syslog.
Just tonight I heard my harddisk scratching hardly again. I checked the
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:'
Feb 4 21:00:32 rinse named[2037]: unexpected RCODE (SERVFAIL) resolving
'194.183.255.216.in-addr.arpa/PTR/IN': 127.0.0.1#50053
Feb 4 21:00:34 leaf named[2008]: unexpected RCODE (SERVFAIL) resolving
'194.183.255.216.in-addr.arpa/PTR/IN': 168.95.1.1#53
...
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:' | wc -l
1277
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:00' | wc -l
24
imacat at rinse ~ %
I searched the archive. On Sep 2, 2004 Kevin Darcy said that these
are problems of others' name servers. There is nothing I can do here.

But there is. You can report it. The nameservers for
183.255.216.in-addr.arpa are lame. In this case you are
using forwarders so you are finding out second hand (hence
the SERVFAIL).

Lookup the whois contact details and inform them.

If that fails report it to the parent (ARIN in this case) so the
delegation can be withdrawn.

Mark

OrgName: InterCage, Inc.
OrgID: INTER-359
Address: 1955 Monument Blvd.
Address: #236
City: Concord
StateProv: CA
PostalCode: 94520
Country: US

ReferralServer: rwhois://rwhois.intercage.com:4321/

NetRange: 216.255.176.0 - 216.255.191.255
CIDR: 216.255.176.0/20
NetName: INTERCAGE-NETWORK-GROUP2
NetHandle: NET-216-255-176-0-1
Parent: NET-216-0-0-0-0
NetType: Direct Allocation
NameServer: NS10.INTERCAGE.COM
NameServer: NS11.INTERCAGE.COM
Comment:
RegDate: 2005-09-20
Updated: 2005-09-20

OrgAbuseHandle: ABUSE735-ARIN
OrgAbuseName: Abuse Department
OrgAbusePhone: +1-925-550-3947
OrgAbuseEmail: abuse at intercage.com

OrgNOCHandle: NETWO670-ARIN
OrgNOCName: Network Operations
OrgNOCPhone: +1-925-550-3947
OrgNOCEmail: noc at intercage.com

OrgTechHandle: INE4-ARIN
OrgTechName: IP Network Engineering
OrgTechPhone: +1-925-550-3947
OrgTechEmail: ipeng at intercage.com

# ARIN WHOIS database, last updated 2006-02-03 19:10
# Enter ? for additional hints on searching ARIN's WHOIS database.

Post by imacat
But, hey, this is my harddisk! My syslog file is growing up enormously,
and I feel the life-span of my root harddisk shortened on every scratch.
This is sort of DoS. My heart aches on every such scratch~ >_<
Is there any way I can suppress these messages? These messages are
irrevelent to my system at all. I would rather like to keep more
relevant information in my syslog, for ex, other type of failurs than
SERVFAIL, or even supress the whole 'unexpected RCODE'. Or is there
something like the "limit" target in Linux iptables/netfilter that can
prevent my harddisk from blowing up? Can anyone teach me how I can do
on this? Thank you.
--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt
<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews at isc.org

imacat

2006-02-05 16:31:16 UTC

Permalink

On Sun, 05 Feb 2006 09:35:48 +1100

Post by Mark Andrews

Post by imacat
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:'
Feb 4 21:00:32 rinse named[2037]: unexpected RCODE (SERVFAIL) resolving
'194.183.255.216.in-addr.arpa/PTR/IN': 127.0.0.1#50053
Feb 4 21:00:34 leaf named[2008]: unexpected RCODE (SERVFAIL) resolving
'194.183.255.216.in-addr.arpa/PTR/IN': 168.95.1.1#53
...
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:' | wc -l
1277
imacat at rinse ~ % grep 'unexpected RCODE (SERVFAIL)' /var/log/messages |
grep '^Feb 4 21:00' | wc -l
24
imacat at rinse ~ %
I searched the archive. On Sep 2, 2004 Kevin Darcy said that these
are problems of others' name servers. There is nothing I can do here.

But there is. You can report it. The nameservers for
If that fails report it to the parent (ARIN in this case) so the
delegation can be withdrawn.

Sorry, NO, I don't want to inform them. I listed some PTR fail
messages, but there are thousands of A fail messages. THEY ARE
SPAMMER, or spammers' victims that they pretend their senders as. It's
sendmail that is trying to bounce the undeliverable spam that causes
these errors. I have spent too much energy on these spammers regularily.
I don't want to inform them, or even check if they are really spammers
or innocent victims and decide whether to inform them.

And, with 24 messages per minute, 1277 per hour, 46724 per month,
are you asking me to check these messages for unique domains, check
these domains' whois records, inform their administrators seperately,
waiting for a couple of days, and then inform their parent
administrators? Certainly you are not so cruel, are you? I'm supposed
to have other better work to do.

For years I haven't got any "lame-server" messages other than
'unexepected RCODE (SERVFAIL)' yet. If the BIND team can provide a way
to disable only the 'unexpected RCODE (SERVFAIL)', but leave other
"lame-server" messages there, I would be very appreciated. I would not
mind if I have to patch it.

--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt

<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

Mark Andrews

2006-02-05 23:36:22 UTC

Permalink

Post by imacat
On Sun, 5 Feb 2006 18:02:51 +0100

Post by Stephane Bortzmeyer
On Mon, Feb 06, 2006 at 12:13:30AM +0800,

Some how I heard of it years ago, I did not disable it, because I
was told to be supposed to be solving the problem instead of
supressing the error messages.

Now that I know this is all about the "lame servers" problem, in
1997. I have a little search on this issue. Is it supposed to be
something like a valid SOA record?

A lame server is a server that has had a zone delegated to
it but is not serving the zone. It can also be a server
that is serving the zone but the zone has a error which
hasn't been fixed by the operator of the nameserver.

Post by imacat
Well, all the SOA records of the domains hosted on my servers are
valid. I think it's a important to be a good citizen. But if BIND want
to enforce that, could you please do some other thing, for example,
notifying the TLD administrators automatically (without using mail,
please), instead of making DoS attack to the harddisk of th good
citizens? With one of my previous servers that has a /var/log partition
only 500MB, it'll be filled up in a very, very short time. And not to
mention the lifespan of the harddisk itself.

It is logged so you have a idea why your (your clients) lookups
are failing. If you don't want to know why your lookups are
failing you are perfectly free to turn the logging off.

As for filling up the log partition, log rotation usually deals
with that.

Post by imacat
If I were to attack some DNS server, knowing that it does not
disable the lame-server messages, all I have to do is to register a
domain, configure an improper SOA record, and trigger it to do DNS query.
Bla! Either the hard disk will be filled up, or die completely.
This is somehow security-related, isn't it?
By the way, I have already made a patch that bypass all the
"unexpected RCODE (SERVFAIL)" messages, that applies to BIND 9.3.2. It
runs well without disabling the "lame-server" messages. I attached it
below. Please tell me if there is any problem.
diff -r -u bind-9.3.2.orig/lib/dns/resolver.c bind-9.3.2/lib/dns/resolver.c
--- bind-9.3.2.orig/lib/dns/resolver.c 2005-10-14 09:38:48.000000000 +
0800
+++ bind-9.3.2/lib/dns/resolver.c 2006-02-06 00:33:34.000000000 +0800
@@ -1663,6 +1663,10 @@
if (reason == DNS_R_LAME) /* already logged */
return;
+ if (reason == DNS_R_UNEXPECTEDRCODE
+ && fctx->rmessage->rcode == dns_rcode_servfail)
+ return; /* no "unexpected RCODE (SERVFAIL)" */
+
if (reason == DNS_R_UNEXPECTEDRCODE) {
isc_buffer_init(&b, code, sizeof(code) - 1);
dns_rcode_totext(fctx->rmessage->rcode, &b);
--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt
<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews at isc.org

imacat

2006-02-06 14:15:45 UTC

Permalink

On Mon, 06 Feb 2006 10:36:22 +1100

Post by Mark Andrews

Post by imacat
On Sun, 5 Feb 2006 18:02:51 +0100

Post by Stephane Bortzmeyer
On Mon, Feb 06, 2006 at 12:13:30AM +0800,

Thank you for correcting my information. I hate that, too.

Post by Mark Andrews
It is logged so you have a idea why your (your clients) lookups
are failing. If you don't want to know why your lookups are
failing you are perfectly free to turn the logging off.

In fact, the only people that will check why lookups are failing is
me and my boss, and we both know how to do that with dig or host. And
I'm turning "lame-server" off completely now.

Post by Mark Andrews
As for filling up the log partition, log rotation usually deals
with that.

Log rotation will only deal with disk fill-up, but not disk
scratching. And we are not using log rotate for security reason.
Anyway, thank you.

--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt

<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

Pete Ehlke

2006-02-06 17:00:25 UTC

Permalink

Post by imacat
Log rotation will only deal with disk fill-up, but not disk
scratching.

Erm... What is the MTBF specification for the drives you're using? And
how much data are you logging that you seriously believe it's going to
shorten the lifespan of your drives?

Post by imacat
And we are not using log rotate for security reason.

Uhh.... what? What 'security reason'?

imacat

2006-02-06 21:17:38 UTC

Permalink

On Mon, 6 Feb 2006 11:00:25 -0600

Post by Pete Ehlke

Post by imacat
Log rotation will only deal with disk fill-up, but not disk
scratching.

Erm... What is the MTBF specification for the drives you're using? And
how much data are you logging that you seriously believe it's going to
shorten the lifespan of your drives?

As I said, 24 "unexpected RCODE (SERVFAIL)" messages per minute,
1277 per hour, 46724 per month. I can hear it scratching noisily every
time. Newer harddrives are weaker than before. I would never risk it.

And, a log file with excess irrevelent messsage is hard to track,
too. It's no difference to useless. It's always a nightmare every time
I try to tail the syslog and a bunch of "unexpected RCODE (SERVFAIL)"
comes in altogether.

Post by Pete Ehlke

Post by imacat
And we are not using log rotate for security reason.

Uhh.... what? What 'security reason'?

Seriously, I don't know who is using log rotate. Maybe only those
newbies, or distrubiton packagers. We archive all the log files, so
that we can reference them in the future. Some intruders do not make
trouble when intrusion happens, but hide themself and wait a specific
time. Also sometimes we may not be able to attend to some minor
problems in time, when we are bonded to other large projects. We have
to keep the log files in order that we can return and check it in the
future. And it has statistics value, too. The Apache log files can be
analized with Analog for marketing purpose. The maillog can be parsed for
recent spammers, or abandon accounts that are frequently spammed. We
even use it to track mails weeks ago after some long vacation.

I can fully understand why distrubiton packagers need logrotate, in
order to help the newbies not fill-up their harddisks. But, we are not
newbies. We know where and how to archive them.

--
Best regards,
imacat ^_*' <imacat at mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt

<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug