Thursday, January 6, 2011

Hyper V Time Synchronization on a Windows Based Network

A very common query that I get asked to help out with in relation to Hyper-V installations is the small (but very important!) task of getting time synchronization working between the physical environment and the virtual environment.

There are a lot of blog posts out there at the moment covering this topic but I thought I'd throw my 2 Cents in based on the solution that works for me.

Background
 
Firstly, I need to explain a little about how time synchronization works in a domain environment.

When you install the first domain controller in your domain, this DC is the owner by default of all 5 FSMO roles within Active Directory. The PDC Emulator Role is the one that is responsible for Active Directory Time Synchronization.

When you add member servers or client computers to your new domain, they then default to using the PDC Emulator Role holder as their time synchronization source. If you move the PDC Emulator Role to another Domain Controller, then this will become the time synchronization source for that domain in place of the original DC.

When you install a Member Server into Active Directory and add the Hyper-V role onto that member server things start getting a little bit more interesting!

The physical Hyper-V host that is now a member server in your new domain will use the PDC Emulator Role holder as its time synchronization source as you would expect it to, however, when you add virtual machines to that Hyper-V host and the 'Hyper-V Integration Services' are installed onto each virtual machine, then that VM starts to synchronize its time from the Hyper-V host it is based on and not from the PDC Emulator Role!

This 'VM to Host' time sync generally isn't a problem when all of your domain controllers are physical machines as the Hyper-V host updates itself from a physical machine separate from the virtual environment and this keeps the internal VM's up to date.

What happens though when your PDC Emulator Role holder Domain Controller (gasp for air!) is based on the Hyper-V host as a virtual machine? You guessed it - time sync issues occur and the virtual DC's clock starts to move out away from the time on the Hyper-V host, which in turn then knocks the rest of the domain off-skew as they are- as mentioned above, configured by default to sync from the PDC Emulator Role Holder.

The Wrong Way
 
A lot of people - including myself for a long time - took the stance that to resolve this problem, the best and easiest solution was to just remove the 'Time Synchronization' tick box from the 'Integration Components Offered' section on the virtual Domain Controller that holds the PDC Emulator Role. Once this step was completed and the time within the virtual DC was configured correctly, this nearly always resolved the time sync issues on site. I use the word 'nearly' in that sentence though as this workaround didn't always work.

The Right Way
 
So what's the solution? A couple of months ago, I came across a blog post from Ben Armstrong -Virtualization Program Manager at Microsoft -covering this subject and specifically around the Hyper-V 'Time Synchronization Integration Service'. In this post, Ben states categorically that you SHOULD NOT disable the 'Time Synchronization Integration Service' on any virtual machine within Hyper-V and instead you should manipulate the Windows Time Service (w32tm service) from within the virtual DC to get the results that you need for a coherent time sync within your domain!

The link to Ben Armstrong's blog post is as follows:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx

To summarise his post and outline my steps when faced with time sync issues in a Hyper-V environment, I carry out the following procedure on all my VM's (most importantly, my virtual DC's):
 
 
Enable all of the Hyper-V Integration Services (see below)


Check each virtual domain controller's time source using the "w32tm /query /source" command

If the virtual DC is using the 'VM IC Time Synchronization Provider', you need to type the following commands into the command line within the virtual DC to leave the Hyper-V time sync enabled for VM reboots but not for when the VM is up and running:

reg add HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider /v Enabled /t reg_dword /d 0

(Select 'Yes' if requested after you enter this command)

w32tm /config /syncfromflags:DOMHIER /update

(This command tells the virtual DC to sync from within the Domain)

net stop w32time & net start w32time

(This command stops and then restarts the Windows Time Service)

w32tm /resync /force

(This command queries the Windows Time Synchronization Service again and should now detail the internal time server - hopefully the DC with the PDC Emulator Role)

At this point, your Virtual Domain Controller should use itself for time sync when it is online and it will use the Hyper-V host for when it is rebooting or coming out of a 'Saved State' and before the Windows O/S loads

Finally, all that is left to do now is to configure the Virtual Domain Controller and each Hyper-V host that you have in your domain to synchronize with an external time source provider (NTP Provider)

Note: Make sure to follow the instructions in this next paragraph exactly as you read them as clicking on the 'Internal Time Source' option will not give you the desired result!
 
Use this link from Microsoft to automatically configure an external authoritative time source by selecting the 'Microsoft Fix It' button half way down the page under the 'Configure the Windows Time Service to use an External Time Source':

http://support.microsoft.com/kb/816042

The 'Microsoft Fix It' button should initiate a wizard that will prompt you to enter your external NTP servers (you can enter two in here for redundancy) and will then configure the registry to reflect these changes automatically.

 As I'm based in Ireland, the NTP provider that I prefer to use is  - ie.pool.ntp.org


Hopefully this will help you to get a better handle on Time Sync in a Hyper-V environment.

25 comments:

  1. kevin,

    Had the same problem but may have a quicker solution. see http://lcomerford.blogspot.com/ latest post ( ignore the rest of my ramblings.

    Louis

    ReplyDelete
  2. Hi Louis,

    I took a look at your solution and this solution works only some times depending on the Hyper-V environment the problem manifests itself.

    Your solution would have been the one I used in the past but on certain sites, the issue would come back.

    I found Ben Armstrongs fix the most comprehensive and will sort out Hyper-V time sync every time!

    Kev.

    ReplyDelete
  3. Good article but surely you want eveything to sync on your network from the PDC server? Only the PDC should have external time?

    ReplyDelete
  4. Hi James,

    Thanks for the comment and apologies about the delay in responding.

    To clarify, everything will synchronise with the PDC Emulator as it normally should do however, if your PDC Emulator is a virtual machine based on a Hyper-V host or cluster, then your Hyper-V host machines will need to be configured to synchronise with an external time source to compensate for the possibility that the virtual PDC Emulator might be powered down or rebooting and the virtual PDC Emulator will have to take it's initial time settings from the Hyper-V host before it boots into Windows to access it's external NTP source.

    Hope this clears things up for you!

    ReplyDelete
  5. Hi Kevin,

    So I read Ben's post, and followed yours.

    My setup is Server 2008 DC running inside Hyper-V, however, my PDC is SBS 2003.

    I have been having several time-related errors, as well as errors documenting failed connection to the PDS, that I believe are related to the time errors. (however, upon checking time sync the computers appear to be syncing perfectly anyway?)

    When I first ran the /source command I was indeed syncing to the VM IC TIme Provider, so I followed the instructions and the /source command showed me that I was indeed now syncing from my PDC, however the time errors persisted.

    So I ran the fix-it from Microsoft successfully, but now when I check /source it tells me I'm running from the Local CMOS Clock, so I guess not syncing at all? I gathered that it would still try to sync from the PDC first.

    Have any idea what's up?

    Also something that may be of note, my Hyper-V host machine is running server 2008 core, is not part of my domain, and I prefer to leave it entirely disconnected from the network when not connecting to it for security reasons. So it is obviously not syncing to any external time source.

    Thanks for reading,
    Crystal

    ReplyDelete
  6. Hi Crystal,

    Thanks very much for your comment, i will try my best to assist you with your problem.

    It is a long time since I had to virtualize an SBS 2003 server with Hyper-V, but I do recall having a lot of strange issues with a number of different virtualized SBS 2003 instances based around time sync and also the 'Server' service crashing / hanging too.

    In the cases with time sync issues, I recall having to actually 'Untick' the 'Time Synchronization' option in the Hyper-V Integration Components (contrary to what I have blogged about above in fairness) and then ensure that the SBS 2003 server was syncing with an external time source. Once this was configured, the time sync issues in the virtualized SBS 2003 VM went away.

    Hope this helps!

    ReplyDelete
  7. Thanks for your valuable posting. This post is really good and blog is really interesting. It gives good details.

    Windows Network Setup

    ReplyDelete
  8. I have read the post. It did make sense until it linked to the MS website that detailed how to get the DC to connect to an external time server...

    ReplyDelete
  9. Ok, I have been working with Server 2008R2 and various Hyper-V machines.

    I followed everything here as best I could.

    It appeared to be working wonderfully. I got the primary DC (Which is a virtual machine) to connect to an external time server, it looked like it was updating and all the other VMs and the physical machine were using the DC as it's time host.

    I found a problem today that took 2 weeks to develope. The physical machine was too fast, it had gained 4mins. The DC that was hosted on it took that time and not the external NTP time.

    I forced an NTP update on the DC (w32tm /resync /force) and it got the NTP time but within 10 seconds, it has reverted back to the Hyver-V host's time.

    How do I get it to stop taking the hosts time?

    I hope I can get this resolved soon

    Brad

    ReplyDelete
  10. Hi Brad,

    Thanks for the comments.

    If you want to get the VM to stop taking the Hyper-V hosts time, then you need to untick the 'Time Synchronization Integration Service' from within the Virtual Machine properties. However, the point I make in this blog post is that we leave this option enabled and we ensure that the physical Hyper-V host is configured to sync with an external NTP time source.

    Hope this helps!

    Kevin.

    ReplyDelete
  11. Hello and thanks for the great post. I am confused though. In my environment, all my DCs are virtual. So I should disable time sync partially as you described?

    Then,I should run: w32tm /config /syncfromflags:DOMHIER /update on all DCs (even the PDC)?

    And finally, why "..all that is left to do now is to configure the Virtual Domain Controller and each Hyper-V host that you have in your domain to synchronize with an external time source provider (NTP Provider)" Why do you do this on each host and DC. If I just set an external time source on the PDC, shouldn't every other member of the domain sync from the PDC?

    Thank you for any help you can provide!

    ReplyDelete
  12. HI Kevin
    Thanks for taking the time to publish this excellent article.
    In my case I have A windows2008R2 DC in a Hyper V VM and one other DC (win2008R2) in a separate Physical machine.
    Once I transfer the PDC role to the separate Physical DC do I need to synchronize with an external time source?Also Do I need to leave the Hyper v Time synchronization service enable in the Hyper V Integration service?Thank You for any help you can provide!!!

    ReplyDelete
    Replies
    1. Hi there and thanks for the comment!

      If you move the PDC role to a physical computer, then I'd still recommend that you sync it with either an external ntp source or at least a hardware time provider that is onsite. Time sync is such and important part of Active Directory so ensuring your DC's are up to date is crucial.

      In relation to Hyper-V, as per my post, you should still leave the Hyper-V Integration Service enabled for all VM's. Just follow the steps above and it'll work perfect.

      Kevin.

      Delete
  13. Hi Kevin (Assuming this is still read)

    I hope you can help

    I have a single physical server running 9 VMs (Two of which are my primary and secondary AD DC). I originally followed your post and got it working wonderfully.

    3 months in, I found that the time had crept out of line.

    I checked the physical machine, ntp was not set correctly. It was set to "Local CMOS Clock". I thought it strange, used the MS Fix It to set an external NTP server and got it updating from that. So the physical machine works. The other VMs sync to my primary DC. There lies the problem.

    My DC is not doing what it should. I tried following all the steps in this post again to get my DC to syncronise with an external NTP server (as the post suggests) and yet nothing I do can get it to change from "Local CMOS Clock". I can set it to an internal machine, but not an external NTP.

    Any ideas? I have tried the MS Fix-It, restarting the machine, running through all the steps again and yet nothing changes.

    ReplyDelete
  14. I am pretty much in love with you right now. Thank you so much for this information!

    ReplyDelete
  15. Superb post.. Resolved my time sync issues in the Hyper V nodes and VMs running on it with DC as VM..

    ReplyDelete
    Replies
    1. Glad to hear it helped Veera - thanks for the great comment!

      Kevin.

      Delete
  16. Ok, this article is awesome... Seriously.
    Question.

    I have 4 AD servers, all of them Hyper-V servers running 2008 SP2.

    2 of them are syncing perfectly. One of them is the PDC, which syncs to a Symmetricom s250 for it's time and is passing it on to another AD machine flawlessly.

    Then there's AD03 and AD04...

    I've ran the commands you've given up there and it works for a split second, but almost immediately it gets thrown back to it's +992.2633750s offset.

    w32tm /monitor gives me the following output.

    AD01 **** PDC **** x.x.x.33:123
    ICMP 0ms
    NTP +0
    RefID Unknown
    Stratum 2

    AD02 x.x.x.34:123
    ICMP 4ms
    NTP -0.0077464s offset from AD01

    AD03 ::1:123 (which is strange it's seeing an IPv6 address here since I thought I had it off)
    ICMP 0ms
    NTP +985.9040000s offset from AD01
    RefID Unspecified / unsync 0x00000000
    Stratum 0

    AD04 x.x.x.34
    ICMP 0ms
    NTP 992.2633750s offset from AD01
    RefID Unspecified / Unsync 0x0000000
    Stratum 0


    I'm pulling my hair out because this is causing a massive issue, I can't RDP to any desktops because it can't find DNS names, Kerberos tickets aren't being created correctly, and like I said w32tm /resync /force works for about .5 seconds and then it throws it back out of whack.

    Only thing I can think is that somehow, AD03 thinks its a PDC or something along those lines?

    Any clues? Any registry settings I should check?

    ReplyDelete
    Replies
    1. Hi Matt,

      Sounds like a strange one but here's a couple of things that I'd try:

      Review the MS FixIt link again and ensure that you've carried out the correct fix against your servers. There are two fixes, one for synching with an internal time source and one for synching with an external one.

      I guess you'd need to use the internal option as you have the Symmetricom device. Try to redo the steps again with this option. If you've already tried that FixIt, then do the opposite and try the other one instead.

      The IPv6 resolution might be something to look at though. I'd always recommend leaving that setting enabled on DC's as it's the primary path for Win 2008 DC's to resolve to each other. If you simply uncheck that box in TCP/IP it doesn't actually turn off IPv6 - there's a whole lot more that has to be done (which isn't recommended) to properly disable it.

      At worst though, DC's are easy enough to demote and promote again. Why not simply DCPROMO AD03 and leave AD off of it for a day or so and then see if the rest of the servers synch properly. If they do, then you know that you've an issue with AD03 and can then investigate further.

      Hope this helps,

      Kevin.

      Delete
  17. Kevin, thanks for your thoughtful reply!
    What I did and has worked now, is I disabled time sync on the hyper-v settings for the two ad servers and so far, so good. It has kept time.

    The servers have no internet connection, it's in a lab with no access to the outside world so, promoting / demoting is fine since it isn't production, and in production it's not hyper-v it's a real actual blade so, hopefully this isn't something we need to worry about there :)

    Why do you recommend not turning off ipv6 via netsh? I don't want any IPv6 happening, nothing, nada. Reason being a certain device that I'll leave unnamed doesn't cooperate well with it so we've disabled it across the network.

    Any other ideas as to why though? I didn't want to disable the time sync, but well, it seems that it was necessary.

    ReplyDelete
  18. Hi Kevin,

    Thanks for this great post!

    When I follow this procedure on my virtual DC which is running the The PDC Emulator Role I get stuck when running the following command: w32tm /resync /force

    It says 'the computer did nog resync because no time data is available'. I've tried to unregister and register the time service, this worked but still I get the same message when I use the command: w32tm /resync /force.

    Any ideas?

    The procedure worked fine on the other VM's!

    Best regards,

    Harrie

    ReplyDelete
    Replies
    1. Harrie,

      I noticed the same exact thing in our environment. If it helps, what I did was leave our PDC VM set to use VM IC Time Sync Provider (I just skipped/removed the registry entry step Kevin included). The reason being since we configured the Hyper V host server to sync to an external source, I want the PDC to sync it's time to it essentially using the Hyper V host as it's time server. If you sync that to the NTP Pool server properly and set the time skew parameters in the MS KB, it should not skew.

      Delete
  19. Hi Kevin,

    are you still there? :-)

    I've read you post many times, but to me it remains contradictory, sure because I'm not native speaker. You say:

    1. Leave Time Synchronisation between DC and Host via Integration Services intact
    2. Let DC use itself for time sync ("At this point, your Virtual Domain Controller should use itself for time sync when it is online...")
    3. "...configure the Virtual Domain Controller [...] to synchronize with an external time source provider."

    I read that like DC should sync with host via Integration Services, then it should sync with itself and finally it should sync with external source? Hope it's clear what I mean. I'm trying to understand that topic now for days and I would appreciate it if someone could clarify this to me.

    Thanks and regards,
    Stefano

    ReplyDelete
  20. Just a comment to thank you for this blog. I was trying to fix an out of sync time on my secondary virtual domain controller running 2012 R2. I followed the instructions on other sites and ended up being stuck on "Local CMOS clock" for the time source. After running the second download option in the linked Microsoft support article, it is now sourcing from the PDC. Thanks again.

    ReplyDelete