Hello again,
I've been encountering some odd behavior seemingly at random after some backups complete. The issue seems to occur primarily with GRT backups, however is not limited to GRT backups as I've seen the same behavior when non-GRT backups complete. I can only assume that the actual backup process completes successfully as the system generates an alert notification of ;SDR Success...' and I see the correct amount of data (xx GB) was backed up during a given job, however the actual backup job is reported as 'Failed' with an error code of E0000602 which is ...query for media sequence...unsuccessful... I can't be certian that the non-GRT backups are the same exact root cause, but they report the same error code even though I have an SDR Success alert corresponding to the backup that has reported as 'Failed'.
I have a feeling that the error code may be a bit of a red herring in this situation as well; after the backup completes, generates the 'SDR Success' message and begins to verify, the verification process can't find some of the backup sets created by the recently completed backup job. As such, the entire job fails. All jobs in question are B2D jobs, so I don't have any reason to suspect any configuration issues as we've had the same configuration and backup jobs running weekly for several years now and this behavior has only just begun to occur within the past few months.
What is leading me to believe that there is something else going on here is that consistantly, when this behavior occurs, I can look at the backup sets list in the B2D storage location that was used for the failed job and some of the backup sets that I expect to see are missing. This is reliabally the case when our Exchange backup reports failure after 'SDR Success...' alert is generated, I check the B2D storage location and I can see the 'system state' and individual Drive letters were backed up, however I don't see the mailbox or public folder databases backup sets. I'll then check the actual file system (Windows Explorer) and verify that there are indeed img###### folders which contain the mailbox and public folder databases - created at the date/time of the backup job that failed. If I then inventory/catalog the B2D location from within the storage view of the BE GUI, the missing backup sets appear. I can then manuallly re-run the failed catalog job (corresponding to the failed GRT backup job) and everything succeeds as if nothing was ever wrong.
So, as far as I can tell, the first behavioral problem contributing to this issue is that when the backup completes, the backup sets that were just created are not being listed in the backup sets list in storage - before the verification process for that particular backup job begins. As such, the verification process can't find one or more expected backup sets and fails the job with the E0000602 error code citing that it can't find the media. A manual inventory/catalog of the storage location finds the missing backup sets, lists them in the backup sets view in the GUI and then all is happy once again in backup land until the next failure occurs.
At one point I was under the impresssion that it had to do with the fact that some backup jobs run a little long sometimes and due to our short backup job expiration times the media was being deleted before the verification process could complete. (We B2D2T all the time, so having short expiration for the B2D portion keeps the B2D locations empty and available for subsequent backup jobs). This however is not the case since I'm beginning to see the same behavior occur more frequently when there is only one backup job running with no duplciate backup sets linked job and so the expiration should make no difference since BE won't delete the most recent available backup unless it has been duplicated by BE to other media.
So in short, the issue here seems to be that Backup Exec is not properly adding newly created backup sets to the storage inventory in all cases, seemingly at random. The bahavior does not occur only for one backup job and occurs for both GRT and non-GRT backup jobs. After a failure is reported, I can manually inventory/catalog the storage location containing the backup sets from the failed job which fills in the missing backup sets and allows a manual re-run of the catalog operation to succeed.
Any chance anyone has seen this before and/or is this possibly a known issue? If so, I could use a fix or a handy work-around so I don't have to continually baby-sit my backup jobs and re-inventory the B2D storage locations sevaral times a week.
Much appreciate any insight,
Thank you,
