OpsMgr 2012 Database Out of Sync issue when you remove/re-add a management server
- UPDATE: MS Released KB Article KB2771934 for this issue: http://support.microsoft.com/kb/2771934
Recently I came across an issue after performing CU 2 / Rollup 2. The update continued to fail on one of my Secondary Management Servers. Since my environment is relatively light on number of servers, I moved all agents over to other Management Servers and proceeded to Uninstall OpsMgr 2012 Console & Management Server from the offending device. Followed all the required steps, paying close attention to deleting the Management Server form the Management Group. Then reinstalled OpsMgr 2012, rebooted and applied the Updates. Everything looked hunky dory….. Until I tried to run Reports, find Custom Group, and create new notifications. In each of these cases the custom information was missing. It wasn’t until I got creative and changed the Dates on the Filters that I was able to see custom groups or any data (in terms of reports content).
This had gone on for several weeks each time these little issues were popping up, all seeming to be related. But where is the connection? Luckily, I have been working with a Microsoft PFE on the issues and we finally got an Event Log error (Completely not related) that lead us to ask the right questions. What made this problem so elusive is that we were getting Performance Data, Availability Data, and State Data. It was only when you did a Report Regarding Alerts that anything was missing. Because of that we thought that the reports themselves were the problems.
How could you have all that other data flowing in and not Alerts?
Answer the OpsMgr Databases are out of sync.
We ran a simple SQL Query to check the OperationsMangerDW DB last modified for Alerts and we found out that it hadn’t been updated since I performed the Rollups.
select DWLastModifiedDateTime, r.RuleSystemName
from SynchronizationProcessState s
join [Rule] r
on s.SynchronizationProcessId = r.RuleGuid
order by DWLastModifiedDateTime desc
- Note: You should have one that lists an older date (from the time of installation I think), the rest should be the day you ran the query.
You may be asking, “Why didn’t you get the DB is out Sync Alert in SCOM?” The answer is because of the issue it self. What happens is when you delete/uninstall a Management Server from SCOM it does a bunch of house cleaning in the background. When you add the exact same server back into the mix….. it doesn’t add it back/restart the Database Sync jobs. So it doesn’t know its broken because it never looks again to be synced.
After running a quick PowerShell script (Provided by MS) and letting the databases re-sync for a couple of days, all I lost was a months worth of Alert Data.
From what I have been told this issue is going to be fixed in the next Service Pack release that is scheduled for later this year. Unfortunately if you having this issue now, the only way to fix it is to contact Microsoft and get the PS script directly from them.
Good Luck and I hope that I am the only one who encounters this issue.