OpsMgr 2012 Database Out of Sync issue when you remove/re-add a management server

Recently I came across an issue after performing CU 2 / Rollup 2. The update continued to fail on one of my Secondary Management Servers. Since my environment is relatively light on number of servers, I moved all agents over to other Management Servers and proceeded to Uninstall OpsMgr 2012 Console & Management Server from the offending device. Followed all the required steps, paying close attention to deleting the Management Server form the Management Group. Then reinstalled OpsMgr 2012, rebooted and applied the Updates. Everything looked hunky dory….. Until I tried to run Reports, find Custom Group, and create new notifications. In each of these cases the custom information was missing. It wasn’t until I got creative and changed the Dates on the Filters that I was able to see custom groups or any data (in terms of reports content).

This had gone on for several weeks each time these little issues were popping up, all seeming to be related. But where is the connection? Luckily, I have been working with a Microsoft PFE on the issues and we finally got an Event Log error (Completely not related) that lead us to ask the right questions. What made this problem so elusive is that we were getting Performance Data, Availability Data, and State Data. It was only when you did a Report Regarding Alerts that anything was missing. Because of that we thought that the reports themselves were the problems.

How could you have all that other data flowing in and not Alerts?

Answer the OpsMgr Databases are out of sync.

We ran a simple SQL Query to check the OperationsMangerDW DB last modified for Alerts and we found out that it hadn’t been updated since I performed the Rollups.

select DWLastModifiedDateTime, r.RuleSystemName

from SynchronizationProcessState s

join [Rule] r

on s.SynchronizationProcessId = r.RuleGuid

order by DWLastModifiedDateTime desc

  • Note: You should have one that lists an older date (from the time of installation I think), the rest should be the day you ran the query.

You may be asking, “Why didn’t you get the DB is out Sync Alert in SCOM?” The answer is because of the issue it self.  What happens is when you delete/uninstall a Management Server from SCOM it does a bunch of house cleaning in the background. When you add the exact same server back into the mix….. it doesn’t add it back/restart the Database Sync jobs. So it doesn’t know its broken because it never looks again to be synced.

After running a quick PowerShell script (Provided by MS) and letting the databases re-sync for a couple of days, all I lost was a months worth of Alert Data.

From what I have been told this issue is going to be fixed in the next Service Pack release that is scheduled for later this year. Unfortunately if you having this issue now, the only way to fix it is to contact Microsoft and get the PS script directly from them.
Good Luck and I hope that I am the only one who encounters this issue.

Josh Ancel

Advertisements

~ by Josh on August 31, 2012.

3 Responses to “OpsMgr 2012 Database Out of Sync issue when you remove/re-add a management server”

  1. Did you ever determine why the “update continued to fail”, or if there was anything you could’ve done differently to avoid re-building the MS and creating the out-of-sync condition?

    • Bryan,
      Unfortunately, nothing was logged with the failures. It was a very strange issue as it would start the process and the “progress window” would disappear, never to be seen again. The process wasn’t running anymore, and nothing was logged in the event logs or any typical MSI installation log. It was a very strange situation. I attempted Reboots, running as different users, applying it both through command prompt and right click method. The very odd part of it was that this one MS was the only one that encountered the issue, all others went off without a hitch.

      Thanks,
      Josh Ancel

  2. […] found the answer in this article by Josh Ancel, thanks a […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: