Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Wednesday, July 1, 2015 5:25 PM
We’re having a weird issue with our DHCP server and how it is handling address renewals. What is apparently happening is occasionally the system gets a renewal request, then for some reason thinks it is already checked out (which it is.. to the machine requesting renewal) but then the address gets marked as a bad_address. I’ve actually sat and monitored the dhcp logs for a while now and what I have found is that when this “bad_address” link happens to an address in the normal allocation pool.. it simply gets flagged, assigned a reverse hexadecimal version of the IP address in question as it’s “MAC” address, but after a short while, it gets erased and the IP can be re-allocated to the next requesting machine.. all in all.. this is not much of a problem as the address gets put back into production in a relatively short time (when I was watching it normally took just a few minutes to less than an hour).
The PROBLEM comes when the IP address in question is a reserved IP address, and is supposed to be given to a specific mac address.. when the “conflict” is somehow detected and the reverse hex IP gets put into the mac field.. it overwrites the MAC address that is supposed to be getting the IP address, and the only way to get it cleared out so far is to manually go in to the reservation, and rename it and re-enter the correct MAC address, which usually sets the machine back to an operational state almost immediately.. but until that manual step is done.. the machine in question is OFFLINE!!!
I’ve looked around the forums and have seen some similar topics, the closest was apparently on specific printers..
I’m wondering if anyone has any suggestions.. I have been monitoring for a rogue dhcp server.. cannot find one as of yet, and while I cannot 100% state that malware is not involved, I doubt it is the cause in this particular issue…
I’m suspecting that something between the dhcp renewal and the DNS update that seems to go along witih it each time, something is getting hashed…
DHCP server is virtual machine-- 2012R2 box.. 1 network port enabled on it, 1 IP address allocated (not multi homed).
I have rebuilt the dhcp scope over the last couple weeks, so I’m fairly sure the scope is correct and all that.
Any suggestions?
here is a picture of what I am finding
Address Lease view in the DHCP mmc module
and here is what the reservation looks like before I modify it back to normal..
Note the mac address is not a reverse Hex version of the IP address
F4=244 15=21 84=132 0a=10
All replies (15)
Thursday, July 2, 2015 2:28 AM ✅Answered
Hi Halidan,
Are there any errors in the events of DHCP server? It may help to analyze the problem.
When creating DHCP reservation, we need to ensure there is no device using the IP address (DHCP or static assigned). We could use ping to test.
Best Regards,
Leo
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected].
Wednesday, July 1, 2015 7:49 PM
OK.. previous attempt to insert a picture failed.. I'll try again with a different one...
and here is what the reservation screen looks like...
the first 2 characters are different than my first example, but the point is still valid..
Monday, July 13, 2015 4:20 PM
Hello again. I can't find any errors on the DHCP server.. and even turned on debugging on the DNS servers for a while.. I'm not finding anything in either log that explains any of this.
I did notice that I get several "conflict's" logged, that usually results in the corresponding IP address getting lines up with the malformed MAC. but normally this issue resolves itself.. UNLESS it is a reserved IP/Mac entry. for reservations.. this quirk overwrites the listed MAC, and will not correct itself.. Non-reservations entries seem to clear themselves out after a short bit of time...
Any suggestions on where or how to setup logging on these systems to figure this out? I've been trolling my dhcp logs for weeks .. not noticing much.. dns logs.. they don't seem to show anything either...
Tuesday, July 14, 2015 1:33 AM
Hi Halidan,
We need to ensure that devices configured with static IP addresses will not conflict with the IP addresses of DHCP clients. And there is no other device offering IP addresses for clients.
About the DHCP audit logging, we could check the configuration in Properties of IPv4.
Here is the guide for analyzing log files:
Analyzing server log files:
https://technet.microsoft.com/en-us/library/cc776384(v=ws.10).aspx
Best Regards,
Leo
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected].
Tuesday, July 14, 2015 1:24 PM
Thank you Leo. I've been going over that page for a while now trying to make head or tails of what I'm seeing.
I have a sample of the dhcp log I'll post here for you to look at.. but I'm not finding much in it. looks like normal operatins have the workstation doing a 3 line set for each renewal..
1. DNS Update request
2. Renew request for the IP address
3. DNS Update successful
When this problem hits.. normally it looks like it's just trying to renew the IP address to fast.. and causing this issue... which results in a "bad address" line in the dhcp log.. then I get a bunch of NACK messages that very quickly fill up the dhcp log completely.. so I dont' get much more information...
here is a sample I pulled a while back.
30,06/23/15,23:38:03,DNS Update Request,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
11,06/23/15,23:38:03,Renew,10.1.2.3,WorkstationName,MACADDRESS,,300134334,0,,,,0x4D53465420352E30,MSFT 5.0,,,0x010600040001020502080006C067AF29C778,0
32,06/23/15,23:38:03,DNS Update Successful,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
30,06/23/15,23:38:48,DNS Update Request,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
11,06/23/15,23:38:48,Renew,10.1.2.3,WorkstationName,MACADDRESS,,96647880,0,,,,0x4D53465420352E30,MSFT 5.0,,,0x010600040001020502080006C067AF29C778,0
32,06/23/15,23:38:48,DNS Update Successful,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
30,06/23/15,23:38:54,DNS Update Request,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
11,06/23/15,23:38:54,Renew,10.1.2.3,WorkstationName,MACADDRESS,,587883992,0,,,,0x4D53465420352E30,MSFT 5.0,,,0x010600040001020502080006C067AF29C778,0
32,06/23/15,23:38:54,DNS Update Successful,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
30,06/23/15,23:39:59,DNS Update Request,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
11,06/23/15,23:39:59,Renew,10.1.2.3,WorkstationName,MACADDRESS,,3564107093,0,,,,0x4D53465420352E30,MSFT 5.0,,,0x010600040001020502080006C067AF29C778,0
32,06/23/15,23:39:59,DNS Update Successful,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
30,06/23/15,23:40:05,DNS Update Request,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
13,06/23/15,23:40:05,Conflict,10.1.2.3,BAD_ADDRESS,,,0,6,,,,,,,,,0
32,06/23/15,23:40:05,DNS Update Successful,10.1.2.3,WorkstationName,,,0,6,,,AAEBdS/vzYKTv6oJaDpPs7AYmrV+/RDEXvuT5w9S9iqwhvM=,,,,,,0
15,06/23/15,23:40:15,NACK,10.1.2.3,,MACADDRESS,,0,6,,,,,,,,,0
*** edit *** normally the "NACK" line will be repeated until the log is full, then there will be no more entries.. this generally takes just a minute or two for the log to fill up so there normally isn't any more usefull information until the bad address is corrected AND the log is manually reset, or when the next days log begins.. without the NACK entry filling it up immediately.
Tuesday, July 14, 2015 1:34 PM
Leo.. I'm about 98% positive there are no duplicate IP addresses being handed out.. the ip addresses that are reserved are being excluded from the distribution list. and they are being handled by the same DHCP server that hands out the rest of the addresses... so theoretically it is fully aware of all addresses it is handing out. to the best of my knowledge, I don't have any duplicate MAC addresses.. and if I did, wouldn't that result in the other machine getting the IP, not the correct one?
I don't have any other DHCP servers working on this VLAN (that I have detected.. fairly sure no rogue dhcp servers).
I'm suspecting something in the DHCP server / DNS server setup somewhere/somehow is getting corrupted when renewal requests come to quickly for it's liking, and that is causing the problem.
As I've said.. if I watch the server long enough, I see this happen on NON-reserved addresses, but they clean themselves up in short order... it's the overwriting of the MAC on reserved addresses that is causing this issue.. what causes the dhcp server to overwrite the MAC address on a reservation?
Thursday, July 16, 2015 6:35 AM
Hi Halidan,
Are you using Cisco switch? I found a similar case that Cisco switch caused the problem.
If yes, we could follow the solution:
http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/8021x/116529-problemsolution-product-00.html
Best Regards,
Leo
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected].
Thursday, July 16, 2015 1:08 PM
Thanks Leo.. as a matter of fact.. I am using a cisco switch.. attempting to get the suggested workaround/fix from the article you mentioned put in place now.
Thursday, July 16, 2015 1:09 PM
As a note to MS. (if anyone is listening) I still wonder at how the system can overwrite the dhcp reservation anyhow.. this would not be an issue if the reservations put in place by an admin could not be overwritten by the system.......
Tuesday, July 21, 2015 2:17 PM
trying the suggested fix.. so far i'm still getting the same issue.. but I haven't applied the suggested command to each and every single switch in my system yet.. getting there...
Thursday, July 23, 2015 5:07 PM
so far the suggested fix for the cisco switch is NOT resolving the issue..
it "could" be that the majority of these problems are coming from wireless machines, which go through a wireless controller before getting to the switch, and I have not found anything on the wireless controller side of things that is comparable to the functionality on the switch as of yet.. but I'll keep looking.
I still say that the system should not be able to overwrite the MAC in the reservation whenever it wants to.. and that if the dhcp server could not alter those settings, then it would likely not be a problem....
Thursday, July 23, 2015 6:36 PM
What is the OS of the DHCP server?
How many DHCP servers are in your network?
The DHCP database can be corrupted if the AV is scanning the c:\Windows\System32\DHCP folder while the database is being written. Hence, you should always exclude the AV from scanning the DHCP folder.
If you think this is a case of database corruption, you may want to move the existing DHCP database file "dhcp.mdb" to another location and create a new database with a fresh reservation and check if the issue is reproducible.
A bad address is marked in DHCP when the same IP address is assigned to another machine in the network
After the client completes the DORA process, it broadcasts a GARP request on the network to find out if someone is using the same IP address. If the client gets a reply to the broadcast, the DHCP server will mark the address BAD.
If you have turned on DHCP conflict detection on the server, you may want to turn it off as the client will send a GARP broadcast after the DORA process
You will have to take a network monitor trace on the DHCP server and the client while trying to obtain the IP address to check if someone else is responding to the GARP request and what's happening on the wire
If a relay agent is in the picture, you may want to check:
https://support.microsoft.com/en-us/kb/2831920
Thursday, July 23, 2015 7:41 PM
Good points.. I've thought about some of that as well..
1.. AV was removed from the server a few weeks back, to test against exactly your point.. did not make a difference there.
2. OS is 2012R2 originally was a copied over dhcp profile from a 2008r2 machine, but I've since deleted that config and built it brand new on the current OS.
3. I don't think it's corruption, or I would expect to be seeing this much more widespread than just about 1/3 of my wireless system.. and also much more prominent on the wired section as well.. it's rather consistently the same group of machines that get caught up in this.. and only really seems to affect the addresses being reserved..
what troubles me about this is that addresses that are set aside and administratively "reserved" are the ones that are having their MAC address overwritten.. which is why this is such a problem..
4. There are never other machines with the same address when this happens.. (I've been scanning for them each time I come across it..) and once the reservation is corrected to the actual MAC address, the IP address usually starts responding to a ping immediately...
5. Good call on trying to do a network capture.. I"ll try to get that setup asap (should have done that a while back)
Friday, October 16, 2015 11:44 AM
Hello all,
we got the same issue in our company. We also using cisco switches and upgraded our W2k8r2 DHCP Server to w2k12r2 to fix this issue but the bad adress still exists and causes Network Broadcast if a reserverd IP Adress is causing the NACK.
Have you find a solution for this?
Greetings
Jürgen
Friday, October 16, 2015 3:40 PM
Well.. yes, I got it to stop happening.. and then forgot to post what I did.. sorry..
So.. I actually opened a ticket with MS, who basically said they aren't going to fix the fact that their dhcp server is getting overwritten by the network.. but I did track the issue down to a little item on the cisco switches..
on your switches, I recommend looking for info on the command " ip device tracking maximum #" this command has to be run on each interface via the cli, but setting it to 0 is what stopped the issue on my end. You can run the "show ip device tracking all" command and if you are getting any hits on that switch.. that's prolly where your problem is coming from.. for me.. disabling that option with the " IP device tracking maximum 0 " command .. and running it on each and every port that is flagging something.. was what I had to do.
I still wish MS had the ability to not have reservations get overwritten.. but even then it wouldn't help with the non-reserved addresses getting eaten up ....