Yet Another DNS 5504 Event issue

2010-05-26

Question

_{Wednesday, May 26, 2010 3:01 PM}

Two of my AD controllers (both running DNS service) appear to be having a similar issue. Both are throwing lots of events in the DNS events that look like this:

Event Type: Information
Event Source:   DNS
Event Category: None
Event ID:   5504
Date:       5/24/2010
Time:       11:51:38 AM
User:       N/A
Computer:   ALPHA
Description:
The DNS server encountered an invalid domain name in a packet from 76.74.137.6. The packet will be rejected. The event data contains the DNS packet.

Immediately after that will come another event with a packet from 76.74.137.7 as well. They always come in pairs.

I know this is "Information" not an error, but since it is new and different it bothers me (yes, I fear unexplained change!)

Both machines are running Windows 2003 R2 SP2. The DNS servers are not exposed to the internet.
Both DNS servers were configured to use OpenDNS for Forwarders. I've also changed them to point to Google's public DNS servers with the same results.
For both servers, this started about a week ago.

Any thoughts on:

should I be concerned?
how can I stop/fix this?

To keep it interesting, I have a 3rd AD / DNS box. Same domain, different Active Directory site. Same forwarders yet doesn't have this issue.

I've seen the Forum FAQ on Troubleshooting 5504 issues. In response:

Secure Cache Against Pollution was already enabled
I believe my forwarders are valid or legit and not recursive. (or they certainly were for the past couple years at any rate)
I have not installed the mentioned hotfix. Since this just recently started, I'm hesitant to grab an older hotfix
I've run the suggested dnscmd with no change

Suggestions on how to proceed?

All replies (16)

_{Saturday, May 29, 2010 4:56 AM ✅Answered}

Definitely not related to VPN connections. I guess that was a red herring. The interesting thing now is since I removed forwarders and the added them back (same one) an hour later, I still get 5504's, but now with some different IP addresses. I'm fast approaching the point where I think a reboot might be in order :-p

I'm not too worried about the WatchGuard just yet. While these servers didn't log any 5504's for a few years the WG was in front of them the whole time. This whole scenario just started this month.

Tried the nslookup test you suggested. First test gives 4 records. Then, after the "set vc" I get a bunch more when I try again. I'm confused though on how that would relate to the firewall?

Cutting to the chase though: Are these 5504's considered "normal"? Just because I'd never seen them the past few years, am I spending too much time on this?

I guess I could bust out the packet sniffer next week, it's annoying that there's no predictable pattern to these errors though. They occur all around the clock...

So the VPN users or connections are ruled out.

If using a forwarder, END0 support won't be required. I went down that road when I suggested to remove your forwarders and use the Root Hints to test this, and basically wanted to make sure you can resolve all domain names and resources on the internet. lack of EDNS0 support will prevent resolution of some resources. Since you put your forwarders back in, it's a moot point now. But in the future, FWIW, you now know if you want to use the Root Hints, your firewall needs to support EDNS0.

I don't think any event log error is normal, but some may be more benign than others. As for your 5504s, I'm starting to lean to two things: an advertising domain using those name servers, and something in the name is illegal. Other than that, this is going to be difficult, unless you actually get some sniffing in. If it's around the clock, it may point to adware on a machine in your network. I'm not trying to add more to your plate, but this is just a suggestion that you may want to check your machines for malware, the kind many AVs don't catch. I've been using malwarebytes.org's free version, but they have a paid version that runs realtime.

Ace

Ace Fekay, MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007, MCSE & MCSA 2003/2000, MCSA Messaging 2003 Microsoft Certified Trainer Microsoft MVP - Directory Services This posting is provided AS-IS with no warranties or guarantees and confers no rights.

_{Wednesday, May 26, 2010 4:48 PM}

Two of my AD controllers (both running DNS service) appear to be having a similar issue. Both are throwing lots of events in the DNS events that look like this:
Event Type: Information
Event Source:   DNS
Event Category: None
Event ID:   5504
Date:       5/24/2010
Time:       11:51:38 AM
User:       N/A
Computer:   ALPHA
Description:
The DNS server encountered an invalid domain name in a packet from 76.74.137.6. The packet will be rejected. The event data contains the DNS packet.
Immediately after that will come another event with a packet from 76.74.137.7 as well. They always come in pairs.

I know this is "Information" not an error, but since it is new and different it bothers me (yes, I fear unexplained change!)

Both machines are running Windows 2003 R2 SP2. The DNS servers are not exposed to the internet.
Both DNS servers were configured to use OpenDNS for Forwarders. I've also changed them to point to Google's public DNS servers with the same results.
For both servers, this started about a week ago.

Any thoughts on:

should I be concerned?

how can I stop/fix this?

To keep it interesting, I have a 3rd AD / DNS box. Same domain, different Active Directory site. Same forwarders yet doesn't have this issue.

I've seen the Forum FAQ on Troubleshooting 5504 issues. In response:

Secure Cache Against Pollution was already enabled

I believe my forwarders are valid or legit and not recursive. (or they certainly were for the past couple years at any rate)

I have not installed the mentioned hotfix. Since this just recently started, I'm hesitant to grab an older hotfix

I've run the suggested dnscmd with no change

Suggestions on how to proceed?

Hello Chris,

That IP in the error resolves to a server in Canada:

Name: ns1.vpsville.ca
Address: 76.74.137.6

Is this being used as a Forwarder? What fowrarders are you using? I can suggest to try 4.2.2.2 and 4.2.2.3, which are viable forwarders.

For the most part, usually an invalid domain name means that there is an illegal character in the domain name, such as an underscore or other special characters. A dash is a legal character. This data is attempting to register into a DNS server.

Assuming that your DCs and machines are only using the internal DNS servers, then it should only be registering into your own DNS server. If the internal domain name and external domain name are the same, and using a public DNS in IP properties as a DNS server, and/or the nameserver tab indicates the a public name, it may be trying to register into an external DNS. Without more specifics, it may be difficult to diagnose.

Is your internal name and external name the same?

Can you post an ipconfig /all from the two affected DNS servers, as well as one from the DC that is not being affected, please? This will also help towards a diagnosis.

Thanks,

Ace

_{Wednesday, May 26, 2010 6:01 PM}

Hi Ace, thank you for the reply.

That IP in the error resolves to a server in Canada:

Name: ns1.vpsville.ca
Address: 76.74.137.6

Is this being used as a Forwarder?

Nope. And if reverse dns isn't properly set, there's really no telling what's really at that IP. To my knowledge I have no business processes or associations to anything hosted at vpsville in Canada.

Keep in mind, 76.74.137.6 and 76.74.137.7 are the only two IP addresses that I see for this error. Always in pairs. I'd never seen this error before (we're talking years here!) until this started a couple years ago. Until this started, the only DNS events I had were the "service starting" events after a reboot :-)

What fowrarders are you using? I can suggest to try 4.2.2.2 and 4.2.2.3, which are viable forwarders.

As mentioned in my original post, I've used both the OpenDNS and Google DNS servers as forwarders. Those are 208.67.222.222 & 208.67.220.220 and 8.8.8.8 & 8.8.4.4 respectively. Been using the OpenDNS set for a couple years now. Regardless, no difference between which set I use.

For the most part, usually an invalid domain name means that there is an illegal character in the domain name, such as an underscore or other special characters. A dash is a legal character. This data is attempting to register into a DNS server.

No name issues on my internal network. Certainly nothing, to my knowledge tied to vpsville.ca either.

Assuming that your DCs and machines are only using the internal DNS servers, then it should only be registering into your own DNS server. [...]

true and true

[...] Assuming that your DCs and machines are only using the internal DNS servers, then it should only be registering into your own DNS server. If the internal domain name and external domain name are the same, and using a public DNS in IP properties as a DNS server, and/or the nameserver tab indicates the a public name, it may be trying to register into an external DNS. Without more specifics, it may be difficult to diagnose.

Is your internal name and external name the same?

No. For all practical purposes, I don't have (nor need) an external name.

What else would you need to know to help diagnose? There's really nothing fancy going on here. Just a vanilla SMB office.

While checking IPCONFIG I noticed that one of the DCs was only configured to point to the other server's DNS, not its own. Not sure if that's a factor or not, but I've fixed it. As for the other:

IP Address: 148.172.173.20
Subnet Mask: 255.255.255.0
Default Gateway: 148.172.173.1
DNS Servers: 148.172.173.20, 148.172.173.17
WINS Server: 148.172.173.20

.20 and .17 are the two DC / DNS servers in question.

And just to head off a possible response: Try not to get hung up on the IP range -- it is assigned to us by a "very large client" as part of their IP space. We've been on it for years, I'm very comfortable that it is not the issue. :-) Honest!

_{Wednesday, May 26, 2010 6:21 PM}

Hi Ace, thank you for the reply.

That IP in the error resolves to a server in Canada:

Name: ns1.vpsville.ca
Address: 76.74.137.6

Is this being used as a Forwarder?

Nope. And if reverse dns isn't properly set, there's really no telling what's really at that IP. To my knowledge I have no business processes or associations to anything hosted at vpsville in Canada.

Keep in mind, 76.74.137.6 and 76.74.137.7 are the only two IP addresses that I see for this error. Always in pairs. I'd never seen this error before (we're talking years here!) until this started a couple years ago. Until this started, the only DNS events I had were the "service starting" events after a reboot :-)

What fowrarders are you using? I can suggest to try 4.2.2.2 and 4.2.2.3, which are viable forwarders.

As mentioned in my original post, I've used both the OpenDNS and Google DNS servers as forwarders. Those are 208.67.222.222 & 208.67.220.220 and 8.8.8.8 & 8.8.4.4 respectively. Been using the OpenDNS set for a couple years now. Regardless, no difference between which set I use.

For the most part, usually an invalid domain name means that there is an illegal character in the domain name, such as an underscore or other special characters. A dash is a legal character. This data is attempting to register into a DNS server.

No name issues on my internal network. Certainly nothing, to my knowledge tied to vpsville.ca either.

Assuming that your DCs and machines are only using the internal DNS servers, then it should only be registering into your own DNS server. [...]

true and true

[...] Assuming that your DCs and machines are only using the internal DNS servers, then it should only be registering into your own DNS server. If the internal domain name and external domain name are the same, and using a public DNS in IP properties as a DNS server, and/or the nameserver tab indicates the a public name, it may be trying to register into an external DNS. Without more specifics, it may be difficult to diagnose.

Is your internal name and external name the same?

No. For all practical purposes, I don't have (nor need) an external name.

What else would you need to know to help diagnose? There's really nothing fancy going on here. Just a vanilla SMB office.

While checking IPCONFIG I noticed that one of the DCs was only configured to point to the other server's DNS, not its own. Not sure if that's a factor or not, but I've fixed it. As for the other:

IP Address: 148.172.173.20
Subnet Mask: 255.255.255.0
Default Gateway: 148.172.173.1
DNS Servers: 148.172.173.20, 148.172.173.17
WINS Server: 148.172.173.20

.20 and .17 are the two DC / DNS servers in question.

And just to head off a possible response: Try not to get hung up on the IP range -- it is assigned to us by a "very large client" as part of their IP space. We've been on it for years, I'm very comfortable that it is not the issue. :-) Honest!

Thanks for providing the info. No worries, I won't get hung up on using public IPs. I understand.

As for which IPs to point to in the NIC, one commonly agreed on configuration is to point to itself as the first entry, and one of the others as the second entry.

Take a look at EventID.net's take on it; http://eventid.net/display.asp?eventid=5504&eventno=642&source=DNS&phase=1.

The one comment, which I've found to work in the past with 5504's, is to suggest changing the Forwarders to see if it alleviates the errors.

Also, there was another comment in the link about an IP in the error message that happened to be from doubleclick.net. Now that makes me think of one other possibility to look for, is to make sure no malware is present on any of your machines. Take a look at the Cache folder in DNS (put the view in Advanced Mode), and see if you can coorelate the IPs you are seeing with a domain name that was previously resolved. It may not be that DNS server in Canada, but it may be a zone it's hosting.

Ace

_{Wednesday, May 26, 2010 6:53 PM}

Thanks again.

Just for grins, I've changed the forwarders again. I'll monitor and see if that makes a difference.

Also, there was another comment in the link about an IP in the error message that happened to be from doubleclick.net. Now that makes me think of one other possibility to look for, is to make sure no malware is present on any of your machines. Take a look at the Cache folder in DNS (put the view in Advanced Mode), and see if you can coorelate the IPs you are seeing with a domain name that was previously resolved. It may not be that DNS server in Canada, but it may be a zone it's hosting.

Now that is something I should definitely dig deeper into -- I had just possibly correlated the issue to only happening when i have remote users on the VPN. Today no VPN users connected and none of these DNS errors... Perhaps one of my remote users has an issue?

However, I'm not quite sure how to go about digging through the cache. I was able to find the vpsville.ca record in the cache easily enough, but not sure how to search for those IPs in other records without manually expanding each and every folder -- and there are a ton of 'em. Am I missing a "Search" option for the cache?

_{Wednesday, May 26, 2010 7:51 PM}

Thanks again.

Just for grins, I've changed the forwarders again. I'll monitor and see if that makes a difference.

Also, there was another comment in the link about an IP in the error message that happened to be from doubleclick.net. Now that makes me think of one other possibility to look for, is to make sure no malware is present on any of your machines. Take a look at the Cache folder in DNS (put the view in Advanced Mode), and see if you can coorelate the IPs you are seeing with a domain name that was previously resolved. It may not be that DNS server in Canada, but it may be a zone it's hosting.

Now that is something I should definitely dig deeper into -- I had just possibly correlated the issue to only happening when i have remote users on the VPN. Today no VPN users connected and none of these DNS errors... Perhaps one of my remote users has an issue?

However, I'm not quite sure how to go about digging through the cache. I was able to find the vpsville.ca record in the cache easily enough, but not sure how to search for those IPs in other records without manually expanding each and every folder -- and there are a ton of 'em. Am I missing a "Search" option for the cache?

Changing the Forwarder is a good start, but if you are saying it's correlating to VPN users, there may be a possibility one of their machines at home, or a company machine, may be compromised. It may be warranted to look into them, such as having them bring their laptops in the office and run an AV and malware scan. For malware, I usually use malwarebytes.org's free scanner. Works nicely.

As for the cache, there's is no search function, unfortunately. When you click on a folder, such as the 'com' folder, you will see the name servers associated with that folder. You will also see the various '.com' domains that were queried by internal users. If you click on any of them, they show you the resource that was queried under that domain name, such as a 'www' resource, and it's IP address. But you would have to look at each domain until you find the NS servers listed in the error to find all the resources that it resolved, which can be pretty hectic and daunting. There is an export function, but it only works on each TLD (such as com, edu, etc), and you have many, well, forget that idea! You could clear the cache, and start fresh. There wil be no real consequence with this other than for each new query that isn't cached, it would just have to resolve it and populate the cache again. So with that idea, you can wait until a VPN user connects, clear the cache, then start looking.

Another possibility is using NetMon or some other sniffer to capture packets and look for DNS queries.

I'm trying to figure out the easiest way, but whichever way you choose, seems to be some work involved!

Ace

_{Thursday, May 27, 2010 2:21 PM}

Another day dawns... :-)

The theory about the DNS events related to VPN users didn't pan out.

Do the below paragraphs sum up what appears to be happening?

Someone/something on my network makes a request for a site. My DNS servers pass it along to the Forwarders who determine that the DNS records are at the vpsville.ca IP addresses(s) mentioned in my original post.

The vpsville.ca DNS servers return a message that my DNS servers can't read -- perhaps because it is a DNAME resource record or an EDNS packet.

Close? If so, I'm not all that worried (yet) about malware.

Just wish there was a reasonable way to narrow down the cause and stop the events. Revisiting the forum's 5504 FAQ, the DNAME thing shouldn't be an issue since I'm running Win2k3 R2 SP2. At least, my understanding is that it was only an issue prior to SP2.

I've ran the suggested DNS command for EDNS, but that didn't change anything either. Does it necessitate a reboot? (I restarted the service at the time...)

_{Thursday, May 27, 2010 3:16 PM | 1 vote}

Another day dawns... :-)

The theory about the DNS events related to VPN users didn't pan out.

Do the below paragraphs sum up what appears to be happening?

Someone/something on my network makes a request for a site. My DNS servers pass it along to the Forwarders who determine that the DNS records are at the vpsville.ca IP addresses(s) mentioned in my original post.

The vpsville.ca DNS servers return a message that my DNS servers can't read -- perhaps because it is a DNAME resource record or an EDNS packet.

Close? If so, I'm not all that worried (yet) about malware.

Just wish there was a reasonable way to narrow down the cause and stop the events. Revisiting the forum's 5504 FAQ, the DNAME thing shouldn't be an issue since I'm running Win2k3 R2 SP2. At least, my understanding is that it was only an issue prior to SP2.

I've ran the suggested DNS command for EDNS, but that didn't change anything either. Does it necessitate a reboot? (I restarted the service at the time...)

Another day dawns, and I am just finishing my cup of coffee. :-)

Good summary, so far. But I don't think it's related to the DNAME issue.

I don't think it's related to EDNS0, either, since you are using a forwarder. Forwarders bypass routers that don't support EDNS0. I wouldn't suggest disabling the EDNS0 feature on the server, either. That just supports larger than 512 byte UDP query packets, which makes resolution more efficient.

Now if you removed the forwarders completely, do the errors still appear?

Ace

_{Thursday, May 27, 2010 4:41 PM}

Now if you removed the forwarders completely, do the errors still appear?

Well... I'll expose my ignorance. If I remove the forwarders, what else should I change to keep external DNS resolution working? Or do Root Hints take over?

(If Root Hints, then am I specifying forwarders mainly to help speed things up and ease root server load?)

_{Thursday, May 27, 2010 6:06 PM}

Now if you removed the forwarders completely, do the errors still appear?

Well... I'll expose my ignorance. If I remove the forwarders, what else should I change to keep external DNS resolution working? Or do Root Hints take over?

(If Root Hints, then am I specifying forwarders mainly to help speed things up and ease root server load?)

If you remove the forwwarders, it will use the Root Hints. Forwarders are more efficient, but I wanted to see if the errors disappear if you eliminate the forwarders, or change them.

Don't forget to enable EDNS0 on the server again.

Does your router support EDNS0? What brand/model is it?

Ace

_{Thursday, May 27, 2010 6:20 PM}

Thanks again.

Ran dnscmd /Config /EnableEDnsProbes 0 on the DNS servers and removed the Forwarders. Now just waiting to see what happens (this, of course, eliminates all of my OpenDNS filtering. *sigh* Worth it to test though)

The office firewall is a WatchGuard Firebox x55e. No clue if it supports EDNS0, but nothing here points to it for DNS so that's moot, right?

[updated]

Removing the forwarders seems to be a good way to generate LOTS of 5504 events for quite a wide range of IP addresses.

_{Friday, May 28, 2010 2:20 PM}

Thanks again.

Ran dnscmd /Config /EnableEDnsProbes 0 on the DNS servers and removed the Forwarders. Now just waiting to see what happens (this, of course, eliminates all of my OpenDNS filtering. *sigh* Worth it to test though)

The office firewall is a WatchGuard Firebox x55e. No clue if it supports EDNS0, but nothing here points to it for DNS so that's moot, right?

[updated]

Removing the forwarders seems to be a good way to generate LOTS of 5504 events for quite a wide range of IP addresses.

So removing the forwarders generated lots of 5504's. Interesting. And that's not just for the VPN folks?

I couldn't find any info about your Watchguard whether it supports it or not. It's best to contact them. You can test it if you like, using nslookup. Here's a series of steps to test it:

nslookup
set q=mx
hotmail.com

You should see about 24 MX entries in the results.

Now force TCP with the following command:
set vc

Hit arrow Up to re-run the hotmail.com query. If you see a substantially larger number of MX entries return in the results, then it means that it may possibly mean it doesn't support it. This test is not necessarily a definite indication, but it is an indication that TCP is pulling more data than UDP, which is what EDNS0 is supposed to allow.

Your best bet is to contact them.

I would at this time sugget to use a packet sniffer to find out which machine is querying for something generating the 5504 errors. Here's an old post of mine, along with other responses, regarding this similar issue when we tried to help another person a few years back, to see what I mean.

http://www.keyongtech.com/2872264-event-id-5504-a

Ace

_{Friday, May 28, 2010 5:12 PM}

Definitely not related to VPN connections. I guess that was a red herring. The interesting thing now is since I removed forwarders and the added them back (same one) an hour later, I still get 5504's, but now with some different IP addresses. I'm fast approaching the point where I think a reboot might be in order :-p

I'm not too worried about the WatchGuard just yet. While these servers didn't log any 5504's for a few years the WG was in front of them the whole time. This whole scenario just started this month.

Tried the nslookup test you suggested. First test gives 4 records. Then, after the "set vc" I get a bunch more when I try again. I'm confused though on how that would relate to the firewall?

Cutting to the chase though: Are these 5504's considered "normal"? Just because I'd never seen them the past few years, am I spending too much time on this?

I guess I could bust out the packet sniffer next week, it's annoying that there's no predictable pattern to these errors though. They occur all around the clock...

_{Wednesday, June 2, 2010 2:07 PM}

Hi Chris,

Any update on your progress?

Ace

_{Friday, June 4, 2010 3:24 PM}

Over the past week the 5504 errors were from a new set of IP addresses -- and then stopped mid week. A mystery for the ages :-)

Thanks for the diagnostic ideas and help, Ace!

_{Friday, June 4, 2010 4:00 PM}

Over the past week the 5504 errors were from a new set of IP addresses -- and then stopped mid week. A mystery for the ages :-)

Thanks for the diagnostic ideas and help, Ace!

You are welcome! It appears that it may be advertising sites, and maybe if intermittent like that, one possibility is they could be html based advertisements in emails where when a user opens it in the preview pane and loads up, it maybe loading the site. Just conjecture.

Otherwise, relax, unless you see them constantly, I would just concentrate on other things. :-)

Have a nice weekend! Cheers!

Ace

Share via

Yet Another DNS 5504 Event issue

Question

All replies (16)

Additional resources