Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Sunday, January 5, 2020 10:55 PM
I've exhausted everything I can find on the Internet. In the past, Exchange Server 2016 (and older versions) logs truncate just fine following a full MS Backup of the drive/volume with Exchange as long as it's set to do a VSS Full Backup under VSS settings. This time, that's not having any effect.
I'm running a DAG with 2 Exchange Servers and 1 Witness. All are on Hyper-V VM's on Windows Server 2016.
I can confirm Exchange believes the backup was done via:
Get-MailboxDatabase -Server <servername> -Status | fl Name,*FullBackup
At first glance, all databases appear healthy and active where expected, but using:
Get-HealthReport -Identity <servername> | where { $_.alertvalue -ne "Healthy" }
I did find 1 Unhealthy report for Mailboxspace on both servers and one Monitoring Unhealthy on one of the servers. I cleared those by deleting and recreating the Powershell Virtual Directory, but still no log file reduction. I re-ran the full backup. No change.
Any ideas? What else can I try to truncate the log files? Also, should I have to run the backup on both members of the DAG or just one to see the log file reductions on both servers?
Colin
All replies (23)
Monday, January 6, 2020 4:09 AM
Hi
Have you tried enabling circular logging on the DB, restarting the Information Store, letting it flush the logs and then disabling circular logging and restarting the IS again and then do a full backup and see if it does anything?
Hope this helps. Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
Monday, January 6, 2020 4:26 AM
Hi Colin,
Did you find any event logs about the backup from Event Viewer?
Please make sure Microsoft Exchange Information Store service and Microsoft Exchange Replication service are running well on two servers.
Log truncation will be delayed by the Exchange Replication service until all necessary log files are replayed into all other copies. You just have to run the backup on one active database. After the logs being replayed to other copies, log truncation will occur automatically on the passive copies.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Monday, January 6, 2020 12:19 PM
Have you verified if all copies of Databases is in healthy state? If there are unhealthy passive copied then these logs are required for log shipping.
Have you ever deleted logs manually from the folder? I would suggest that you create a new Database with required number of passive copied, move couple of mailboxes, run the backup and verify if that helps.
Else, you may have to enable circular logging for a while, to truncate all logs and disable it again.
https://expert-advice.org/exchange-server/move-truncate-logs-exchange-server-2013/
Monday, January 6, 2020 1:17 PM
Try to run vsstester and see what happens please check if logs are getting purged or not.
https://github.com/Microsoft/VSSTESTER
Monday, January 6, 2020 2:10 PM
I have not done this (enabled circular logging, then restarting the IS). I suppose if it's just to clear the logs, then maybe this is OK. We wanted to make the backup for the added safety that provides too. I'm still hoping for a solution that leverages the backup, but if not, it's good to have this as an alternative path. Thanks!
Curious: given that we believe the backup itself went fine and the problems is with the data syncing across the DAG, is there any reason to believe circular logging would work any better?
Colin
Monday, January 6, 2020 2:28 PM
Lydia, there are some errors that seem to be new. I had not seen these before I ran the last full backup.
Event 1006
The performance counter '\<<servername>>\MSExchange Assistants - Per Database(msexchangemailboxassistants-<<mailbox DB name>>)\Event Dispatchers Catching Up' sustained a value of '400.00', for the '30' minute(s) interval starting at '1/6/2020 9:12:00 AM'. Threshold breached since '1/5/2020 7:12 PM'. None Trigger Name:EventDispatchersCatchupQueueTrigger. Instance:msexchangemailboxassistants-mailbox database 0321554407
There are also recurring warnings:
Event 1040
The average of the most recent heartbeat intervals [315] for request [Ping] used by clients is less than or equal to [540].
Make sure that your firewall configuration is set to work correctly with Exchange ActiveSync and direct push technology. Specifically, make sure that your firewall is configured so that requests to Exchange ActiveSync do not expire before they have the opportunity to be processed.
For more information about how to configure firewall settings when using Exchange ActiveSync, see Microsoft Knowledge Base article 905013, "Enterprise Firewall Configuration for Exchange ActiveSync Direct Push Technology" (http://go.microsoft.com/fwlink/?linkid=3052&kbid=905013).
The server with the Event Error 1006 shows everything as healthy when looking at the databases or performing a Get-HealthReport
I also see that there is now one unhealthy response on the other server (yesterday, everything showed up as healthy following the Powershell Virtual Directory rebuild):
Server State Name TargetResource HealthSetName AlertValue ServerComponent
<<servername>> NotApplicable EacBackEndLogonMonitor ECP Unhealthy None
The only event error on that server is:
Event 1012
Data loss occurred in RetentionAgent: RetentionAgent: Data loss occurred. The size of this folder C:\Program Files\Microsoft\Exchange Server\V15\Logging\Diagnostics\DailyPerformanceLogs has reached the max size allowed - 5120 MB. Some files will be purged.
All Exchange Services are running on both servers. Is there anything specific I could check to confirm they are "running well" as you put it? Did any of these details give you any further ideas? Anything else I can try?
Monday, January 6, 2020 3:13 PM
Databases all appear healthy. Only "Unhealthy" reports found via Get-HealthReport (they had all appeared as Health yesterday, but this has since changed):
Server State Name TargetResource HealthSetName AlertValue ServerComponent
<<servername>> NotApplicable EacBackEndLogonMonitor ECP Unhealthy None
Colin
Monday, January 6, 2020 3:19 PM
Try to run vsstester and see what happens please check if logs are getting purged or not.
As noted in my original post, logs are not being purged. That's the problem. Database backup runs and completes fine. Exchange shows that the backup has been done, via:
Get-MailboxDatabase -Server <<servername>> -Status | fl Name,*FullBackup
This shows SnapshotLastFullBackup: True for for all databases with a timestamp of the completion of the last full VSS backup.
I think this means that the problem is purely internal to Exchange -- for some reason, something's not syncing between the DAG members, so it's not willing to purge/truncate the old log data yet. But what and how can I fix?
Colin
Tuesday, January 7, 2020 11:06 PM
The errors returned. I don't see much of anything in the Event Log, but do get these. Are these a clue? How can I fix so I can truncate the logs?
Colin
Wednesday, January 8, 2020 8:36 AM
Hi Colin,
The event logs you provided show little connection to the log truncation. Did you find any other similar events mentioned in the following article: Exchange 2019 Transaction Log Truncation Issue
Please also check the content indexing health for the database copies:
Get-MailboxDatabaseCopyStatus | fl name,status,contentindexstate
Make sure the Microsoft Exchange Replication service and Microsoft Exchange Information Store service are running well. You can try to restart them.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Wednesday, January 8, 2020 7:42 PM
Everything else looks healthy and good. The Get-MailboxDatabaseCopyStatus shows:
Name : Mailbox Database ...407\<Exch Serv 1>>
Status : Mounted
ContentIndexState : Healthy
Name : Mailbox Database ...903\<Exch Serv 1>>
Status : Mounted
ContentIndexState : Healthy
And on the other server:
Name : Mailbox Database ...903\<Exch Serv 2>>
Status : Healthy
ContentIndexState : Healthy
Name : Mailbox Database ...407\<Exch Serv 2>>
Status : Healthy
ContentIndexState : Healthy
I don't see anything at that link that suggests anything new.
Further, I have not only restarted individual services, just to be sure, I've restarted both servers with a complete power cycle.
Any other suggestions? Should I try to clear the MailboxSpace and Monitoring Unhealthy messages shown in my screen shot above, or are those unrelated and unimportant?
Thanks,
Colin
Colin
Friday, January 10, 2020 2:47 AM
ActiveSync health set monitors the overall health of the ActiveSync service for mobile clients in your organization, it is unrelated to the log truncation issue. However, you can check this article for suggestions, if you want fix ActiveSync health set: Troubleshooting ActiveSync Health Set
I did some research for MailboxSpace Health Set, StorageLogicalDriveSpaceMonitor regarding the Exchange disk space monitoring, which by default sets 175GB of log disk threshold, and generates alerts if the disk has less space than this. At present, you can try to remove old and unneeded logs to guarantee enough disk space. If available, you can try to expand the disk space.
We can try to solve the MailboxSpace Health Set issue to make sure Exchange server and database can work normally. After that, run the VSS full backup again to check any differences.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Friday, January 10, 2020 1:46 PM
ActiveSync health set monitors the overall health of the ActiveSync service for mobile clients in your organization, it is unrelated to the log truncation issue. However, you can check this article for suggestions, if you want fix ActiveSync health set: Troubleshooting ActiveSync Health Set
I did some research for MailboxSpace Health Set, StorageLogicalDriveSpaceMonitor regarding the Exchange disk space monitoring, which by default sets 175GB of log disk threshold, and generates alerts if the disk has less space than this. At present, you can try to remove old and unneeded logs to guarantee enough disk space. If available, you can try to expand the disk space.
We can try to solve the MailboxSpace Health Set issue to make sure Exchange server and database can work normally. After that, run the VSS full backup again to check any differences.
Regards,
Lydia Zhou
We don't have any problems with mobile devices, from older Windows Phones to modern Android devices. Email and calendar updates work just fine on mobile, so whatever the issues with ActiveSync, they don't (yet) present any functional problems. For now, I'd like to focus on the issues preventing truncation, before we run out of drive space and have a catastrophic problem.
If you think the MailboxSpace Health Set could be related, that sounds promising. How do we fix that? You said "remove old and uneeded logs", but I thought manually removing logs was destructive and that logs should ONLY be removed by the truncation process. What should I do?
Colin
Monday, January 13, 2020 2:35 AM
Since the database is backed up successfully, you can move some old logs to other disk if you don't want to delete them directly. As mentioned above, you also can expand the disk space to make sure you have enough free space.
Here is a similar issue, enough free space can make VSS full backup and log truncation work well: Transaction Logs not truncating on backup
Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make sure that you completely understand the risk before retrieving any suggestions from the above link.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Monday, January 13, 2020 2:54 AM
Since the database is backed up successfully, you can move some old logs to other disk if you don't want to delete them directly. As mentioned above, you also can expand the disk space to make sure you have enough free space.
Here is a similar issue, enough free space can make VSS full backup and log truncation work well: Transaction Logs not truncating on backup
Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make sure that you completely understand the risk before retrieving any suggestions from the above link.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
I don't think that applies to my situation (unless I'm looking at the wrong parts of the page). I am using Windows Server Backup, not a third-party backup utility. Also, there are plenty of GB still free on the volume (for now, getting less every week as the logs continue to bloat), so more space won't help unless you're imagining an unlimited amount of space where I never purge/truncate the logs, and that's not realistic.
Colin
Monday, January 13, 2020 3:11 AM
Lydia, given that we're not seeing any errors, could this be a DAG issue rather than a backup issue? If so, perhaps an easier solution to this ongoing diagnostics would be to shutdown one of the members of the DAG, temporarily reducing everything to a single server, truncate the logs, then reformat the removed server, and then finally add it back in a new server to the DAG.
Would that work? Is there a way to force the logs to truncate if we shrink the DAG?
Colin
Tuesday, January 14, 2020 9:07 AM
Hi Colin,
Before you shutdown one DAG member, you can move all active databases to the other one. After truncating logs, you can bring the server up and update database copies. You don't have to remove and reformat one DAG server.
Since you have a two nodes DAG, shutdown one member server and make sure the witness server works. Then the DAG can work normally. You don't have to remove and reformat one DAG member.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Thursday, January 16, 2020 6:03 PM
Hi Colin,
Before you shutdown one DAG member, you can move all active databases to the other one. After truncating logs, you can bring the server up and update database copies. You don't have to remove and reformat one DAG server.
Since you have a two nodes DAG, shutdown one member server and make sure the witness server works. Then the DAG can work normally. You don't have to remove and reformat one DAG member.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
I'm a little confused by this: if I simply set the Active databases to be all on one server (which is not a problem), then shutdown (but don't destroy) the other DAG client member, won't it still have the same problem where it believes that the logs are not finished syncing between all DAG members? And then, because it thinks it still needs to sync the logs before restoring, wouldn't I still be in exactly the same place I am now?
Colin
Tuesday, January 21, 2020 2:12 AM
Hi Colin,
Yes, log truncation will be delayed by the replication service until all necessary log files are replayed into all other copies. Since the other DAG member is shutdown, there are no available copies in DAG. The logs on the active database can be truncated successfully. If you still worry about the replication of database copies, you can remove the copy, and add back the database copy after truncation.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Monday, January 27, 2020 8:56 AM
Any updates so far? If you have solved your problem, could you share with us? Maybe it will help more people with similar problems.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Monday, January 27, 2020 11:45 AM
Any updates so far? If you have solved your problem, could you share with us? Maybe it will help more people with similar problems.
Regards,
Lydia Zhou
Not yet. This got more complicated than I've had time to attack, but as the log files fill up the server, I will need to do something soon.
Regarding "shutting down the other DAG member," is that literally as simple as just issuing the shutdown command to the computer (virtual server) and rerunning the backup on the active server while the secondary DAG member is offline, or do you mean taking action through Exchange Management Shell to terminate the DAG member? If the latter, what do I need to do so the remaining active DAG member will know the other has been shutdown?
Colin
Wednesday, January 29, 2020 5:43 AM
You don't need additional action in EMS to shut down the server. However, before shutting down the server, you can active the database on other DAG member manually to make sure user mailboxes work well. For reference: Activate a mailbox database copy
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Friday, February 7, 2020 11:21 AM
Just checking in to see if above information was helpful. If you have any questions or need further help on this issue, please feel free to post back.
Regards,
Lydia Zhou
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].