I need a solution
Good Evening,
I'm hoping someone can help me determine if there is a fix for the problem I am experiencing. I have had several recent environment changes but 90% of my backups are working flawlessly at great speeds. Here are the details for the last 10%:
When backing up smaller VMs of a size less than 100GB my backups are quick and progress smoothly. My bpbkar32 job will start usually a minute or two after position the tape to the proper location. Speeds typically around 55 - 65 MB/sec on an LTO4 tape drive.
When backing up large VMs of a size greater than 100GB my backups are initially crawling.
Here is a snipet from a job backing up a 600GB server:
11/30/2012 7:03:20 PM - started process bpbrm (5260)
11/30/2012 7:03:20 PM - connecting
11/30/2012 7:03:21 PM - Info bpbrm(pid=5260) starting bpbkar32 on client
11/30/2012 7:03:21 PM - connected; connect time: 00:00:01
11/30/2012 7:03:24 PM - Info bpbkar32(pid=5128) Backup started
11/30/2012 7:03:24 PM - Info bptm(pid=3984) start
11/30/2012 7:03:24 PM - Info bptm(pid=3984) using 65536 data buffer size
11/30/2012 7:03:24 PM - Info bptm(pid=3984) setting receive network buffer to 263168 bytes
11/30/2012 7:03:24 PM - Info bptm(pid=3984) using 30 data buffers
11/30/2012 7:03:25 PM - Info bptm(pid=3984) start backup
11/30/2012 7:03:25 PM - Info bptm(pid=3984) Waiting for mount of media id 000406 (copy 1) on server xxxxx
11/30/2012 7:03:25 PM - mounting 000406
11/30/2012 7:04:02 PM - Info bptm(pid=3984) media id 000406 mounted on drive index 0, drivepath {2,0,3,0}, drivename IBM.ULT3580-HH4.000, copy 1
11/30/2012 7:04:02 PM - mounted; mount time: 00:00:37
11/30/2012 7:04:02 PM - positioning 000406 to file 11
11/30/2012 7:04:55 PM - positioned 000406; position time: 00:00:53
11/30/2012 7:04:55 PM - begin writing
11/30/2012 8:03:59 PM - Info bpbkar32(pid=5128) INF - Transport Type = san
As you can see the bpbkar32 job doesn't kick of for about an hour. After this job kicks off, I see about 200 KB/sec transfer speeds for another hour. My network connection shows no real traffic. My storage device shows no contention for resources. After this second hour of waiting, my speeds just pick up out of nowhere and I start streaming at around 55-65 MB/sec again.
I'm not sure why it takes approximately 2 hours for NetBackup to really start sending data over my storage fabric on large VMDK files. Here is a snipet from a 90GB server job details to show the contrast:
11/30/2012 6:00:15 PM - Info bpbrm(pid=4196) reading file list from client
11/30/2012 6:00:15 PM - Info bpbrm(pid=4196) starting bpbkar32 on client
11/30/2012 6:00:15 PM - connecting
11/30/2012 6:00:15 PM - connected; connect time: 00:00:00
11/30/2012 6:00:44 PM - Info bpbkar32(pid=4544) Backup started
11/30/2012 6:00:44 PM - Info bptm(pid=5488) start
11/30/2012 6:00:44 PM - Info bptm(pid=5488) using 65536 data buffer size
11/30/2012 6:00:44 PM - Info bptm(pid=5488) setting receive network buffer to 263168 bytes
11/30/2012 6:00:44 PM - Info bptm(pid=5488) using 30 data buffers
11/30/2012 6:00:44 PM - Info bptm(pid=5488) start backup
11/30/2012 6:00:44 PM - Info bptm(pid=5488) Waiting for mount of media id 000406 (copy 1) on server xxxxxxx.
11/30/2012 6:00:44 PM - mounting 000406
11/30/2012 6:01:23 PM - Info bptm(pid=5488) media id 000406 mounted on drive index 0, drivepath {2,0,3,0}, drivename IBM.ULT3580-HH4.000, copy 1
11/30/2012 6:01:23 PM - mounted; mount time: 00:00:39
11/30/2012 6:01:23 PM - positioning 000406 to file 5
11/30/2012 6:03:43 PM - positioned 000406; position time: 00:02:20
11/30/2012 6:03:43 PM - begin writing
11/30/2012 6:04:21 PM - Info bpbkar32(pid=4544) INF - Transport Type = san
11/30/2012 6:18:11 PM - Info bptm(pid=5488) waited for full buffer 23591 times, delayed 25269 times
11/30/2012 6:18:11 PM - Info bpbkar32(pid=4544) bpbkar waited 677 times for empty buffer, delayed 3689 times.
11/30/2012 6:18:21 PM - Info bptm(pid=5488) EXITING with status 0 <----------
11/30/2012 6:18:21 PM - Info bpbrm(pid=4196) validating image for client xxxxxxx
11/30/2012 6:18:21 PM - Info bpbkar32(pid=4544) done. status: 0: the requested operation was successfully completed
11/30/2012 6:18:21 PM - end writing; write time: 00:14:38
Thank you for your time!