Hi,
We’ve been having issues setting up a Dell EMC ML3 tape library with our backup server running DPM 2019. Having been through the motions with Dell they’ve established that there’s no fault with the hardware as we can run all the tape library/tape drive operations through IBM’s ITDT tape diagnostic tool.
Additionally, it was checked that DPM is compatible with the Dell EMC ML3 tape library
With the latest nonexclusive_dpf drivers for the medium changer and tape drive installed DPM can identify the library and tape drive. However, the tape drive states that it is constantly “in use” and when running a detailed/fast inventory to find the tapes (10 tapes – in one logical library, random access partition) the DPM console crashed with an error ID: 917.
” Connection to the DPM service has been lost.
Review the application event log for information about a possible service shutdown. Verify that the following services are not disabled:
DPM
DPM Replication Agent
SQLAgent$MSSQLSERVER
MSSQL$MSSQLSERVER
Virtual Disk Service
Volume Shadow Copy
ID: 917 “
All of the services were checked to see if they were running. After making sure that they were I tested the inventory again, it once again crashed. In event viewer there were a number of crash logs.
“The detailed inventory of tape failed for the following reason: (ID: 3316)
Operation failed since the slot 1 in library IBM TotalStorage 3573 Tape Library-2 was empty. This tape is currently present in offline (ID: 24027)”
And
“Library drive IBM ULTRIUM 7 3580 TAPE DRIVE (1164F0605B) in IBM TotalStorage 3573 Tape Library-2 is not functioning and library jobs may fail until the drive is repaired. The drive is not functioning for the following reason:
(ID: 3303)
Drive IBM ULTRIUM 7 3580 TAPE DRIVE (1164F0605B) was detected as not ready. (ID: 25015)”
According to the first error (24027) the way to fix this is to move a tape to the slot specified, however there is already a tape in slot 1 (slot 1-10 full of tapes).
The second error (25015) the solution for resolving the error is:
- If the drive is being serviced, you can ignore this alert.
- Make sure that the drive is powered on.
- Make sure that the drive is connected properly.
- In Device Manager, if the drive is not enabled or is not visible, refer to your hardware manual for other troubleshooting steps.
All of these do not apply as the drive is not being serviced, is powered on and correctly connected and within device manager both the medium changer and tape drive are visible in device manger and there are no errors with the drivers.
Changing the Dell supplied drivers to alternate versions i.e. exclusive, exclusive_dpf, exclusive_no_dpf, nonexclusive and nonexclusive_no_dpf we end up with driver errors (37) and can only identify the tape drive standalone. We have also used the IBM specific drivers for the Tape library and drive version. These provided no different results.
The DPM drive mapping tool has been used to ensure that the correct ISCSI addresses are being used for the library and drive after completing the remapping no changes to the behaviour of the library/drive.
Also used the DPM Library compatibility test tool running tests 1, 2, 4 and 5 all of which failed.
- dpmlibrarytesttool.cpp(1026) NORMAL CMDLINE: DPMLibraryTest.exe /CERTIFY /TL \\.\Changer0 /AT 1 2 4 5
- utility.cpp(274) WARNING Failed: Hr: = [0x80070424] QueryServiceStatus failed
- utility.cpp(350) WARNING Failed: Hr: = [0x80070424] GetServiceStatus for NtmsSvc service failed, ignoring the failure
- utility.cpp(267) WARNING Failed: Hr: = [0x80070424] : Encountered Failure: : lVal : (HANDLE)(schService = OpenServiceW( schSCManager, strServiceName.PeekStr(), 0x0004))
- dpmlibrarytesttool.cpp(990) NORMAL By default only RSM_COMPAT_INIT_ELEMENT_STATUS - RSM Compat mode is enabled
- utility.cpp(196) WARNING Failed: Hr: = [0x80070002] : Encountered Failure: : lVal : HRESULT_FROM_WIN32(2)
- cfgclass.cpp(1104) [0000000001650820] NORMAL We got 1 devices for class 0
- cfgclass.cpp(176) [00000000031210E8] NORMAL Initializing Device with path \\?\scsi#sequential&ven_ibm&prod_ult3580-td7#5&3158837b&0&000000#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}
- system.cpp(1183) WARNING Failed: Hr: = [0x80070002] Failed to initialize IOCTL timeout from registry. Initializing with default timeout [900000 ms].
- system.cpp(1168) WARNING Failed: Hr: = [0x80070002] : Encountered Failure: : lVal : r.GetValue(L"LibraryIoctlTimeout", &s_dwIoctlTimeoutInMs)
- cfgclass.cpp(1104) [0000000003121890] NORMAL We got 1 devices for class 1
- cfgclass.cpp(176) [0000000003121938] NORMAL Initializing Device with path \\?\scsi#changer&ven_ibm&prod_3573-tl#5&3158837b&0&000001#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}
- cfgnudev.cpp(984) [0000000000BFF1B8] NORMAL Found a Tape Drive with Serial Number match
- dpmdevicelayer.cpp(1528) [0000000000BFF1B0] NORMAL Found a device with following details
- dpmdevicelayer.cpp(1529) [0000000000BFF1B0] NORMAL Device Name : \\.\Changer0
- dpmdevicelayer.cpp(1530) [0000000000BFF1B0] NORMAL Device Serialnumber : 55L3A7800RBNLL01
- dpmdevicelayer.cpp(1531) [0000000000BFF1B0] NORMAL Device Vendor Name : IBM Corporation
- dpmdevicelayer.cpp(1532) [0000000000BFF1B0] NORMAL Device Product Name : IBM TotalStorage 3573 Tape Library
- dpmdevicelayer.cpp(1533) [0000000000BFF1B0] NORMAL FirmWare Revision : 1300
- dpmdevicelayer.cpp(1535) [0000000000BFF1B0] NORMAL Device SCSI Props : (P,B,T,L)5,0,0,1
- dpmdevicelayer.cpp(1543) [0000000000BFF1B0] NORMAL Driver Name : ibmcg2k19
- dpmdevicelayer.cpp(1567) [0000000000BFF1B0] NORMAL Driver Version : 6.2.6.8x64 built by: WinDDK
- dpmdevicelayer.cpp(1528) [0000000000BFF1B0] NORMAL Found a device with following details
- dpmdevicelayer.cpp(1529) [0000000000BFF1B0] NORMAL Device Name : \\.\Tape0
- dpmdevicelayer.cpp(1530) [0000000000BFF1B0] NORMAL Device Serialnumber : 1164F0605B
- dpmdevicelayer.cpp(1531) [0000000000BFF1B0] NORMAL Device Vendor Name : IBM Corporation
- dpmdevicelayer.cpp(1532) [0000000000BFF1B0] NORMAL Device Product Name : IBM ULTRIUM 7 3580 TAPE DRIVE
- dpmdevicelayer.cpp(1533) [0000000000BFF1B0] NORMAL FirmWare Revision : J4D0
- dpmdevicelayer.cpp(1535) [0000000000BFF1B0] NORMAL Device SCSI Props : (P,B,T,L)5,0,0,0
- dpmdevicelayer.cpp(1543) [0000000000BFF1B0] NORMAL Driver Name : ibmtp2k19
- dpmdevicelayer.cpp(1567) [0000000000BFF1B0] NORMAL Driver Version : 6.2.6.8x64 built by: WinDDK
- dpmdevicelayer.cpp(1085) [0000000000BFF1B0] NORMAL Rescan Operation completed with hr - 0x0
- dpmdevicelayer.cpp(1479) NORMAL Opening handle for device \\.\Changer0
- dpmdevicelayer.cpp(1492) NORMAL Open Handle Operation completed with hr - 0x0
- dpmdevicelayer.cpp(854) [0000000000BFF1B0] NORMAL Doing Fast Inventory
- dpmdevicelayer.cpp(857) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_PARAMETERS
- dpmdevicelayer.cpp(870) [0000000000BFF1B0] NORMAL Library information Number of Slots - 35, Number of Drives - 1, Number of I/E ports – 5
- dpmdevicelayer.cpp(883) [0000000000BFF1B0] NORMAL Doing Slot Inventory
- dpmdevicelayer.cpp(1614) [0000000000BFF1B0] NORMAL Calling InitializeElementStatus for element type - 2, first element - 0, Number of elements
- dpmdevicelayer.cpp(1640) [0000000000BFF1B0] NORMAL InitializeElementStatus Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 2, first element - 0, Number of elements – 5
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 2, first element - 5, Number of elements – 5
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 2, first element - 10, Number of elements – 5
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 2, first element - 15, Number of elements - 5
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 2, first element - 20, Number of elements – 5
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 2, first element - 25, Number of elements – 5
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 2, first element - 30, Number of elements – 5
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1797) [0000000000BFF1B0] NORMAL Get Element Status Operation completed with hr - 0x0
- dpmdevicelayer.cpp(891) [0000000000BFF1B0] NORMAL Doing I/E port Inventory
- dpmdevicelayer.cpp(1614) [0000000000BFF1B0] NORMAL Calling InitializeElementStatus for element type - 3, first element - 0, Number of elements
- dpmdevicelayer.cpp(1640) [0000000000BFF1B0] NORMAL InitializeElementStatus Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 3, first element - 0, Number of elements – 1
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 3, first element - 1, Number of elements – 1
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 3, first element - 2, Number of elements – 1
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 3, first element - 3, Number of elements – 1
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- dpmdevicelayer.cpp(1659) [0000000000BFF1B0] NORMAL Calling IOCTL_CHANGER_GET_ELEMENT_STATUS for element type - 3, first element - 4, Number of elements – 1
- dpmdevicelayer.cpp(1673) [0000000000BFF1B0] NORMAL IOCTL_CHANGER_GET_ELEMENT_STATUS Succeeded
- lacommonlib.cpp(167) NORMAL Element Type 3, Element Number 4 is in Exceptional State
- dpmdevicelayer.cpp(1797) [0000000000BFF1B0] NORMAL Get Element Status Operation completed with hr - 0x33e7210
- dpmdevicelayer.cpp(950) [0000000000BFF1B0] NORMAL Fast Inventory Operation completed with hr - 0x33e7210
- dpmlibrarytesttool.cpp(788) NORMAL Result: 0 tests completed successfully, 5 failed
- dpmlibrarytesttool.cpp(794) NORMAL Would you like to submit error report? [Y/N :Default(Y)]:
- dpmlibrarytesttool.cpp(806) NORMAL Submitted the error report successfully
- dpmlibrarytesttool.cpp(1085) NORMAL Operation Failed....
- utility.cpp(706) NORMAL Error Code: 0x80990c1b
- dpmlibrarytesttool.cpp(1087) NORMAL VERBOSE INFORMATION FILE:
- dpmlibrarytesttool.cpp(1087) NORMALC:\...\DPMLibraryTest\LibraryTestTool-27-Curr.errlog
- lacommonlib.cpp(406) WARNING Failed: Hr: = [0x80990c1b] : Unhandled error : Fast inventory is going to FAIL: lVal : E_LIBRARYAGENT_INVENTORY_EXCEPTION
- dpmdevicelayer.cpp(1708) [0000000000BFF1B0] WARNING Failed: Hr: = [0x80990c1b] : Encountered Failure: : lVal : CLACommonLib::UpdateStateFromFastInventory(0, ioctlParams, arrMedia, arrAbsentSlots, arrOpenIEPorts, arrUnusableSlots, arrUnusableDrives, &numMedia, &numAbsentSlots, &numOpenIEPorts, &numUnusableSlots,&numUnusableDrives, m_pErrorAndCancelHelper, false)
- dpmdevicelayer.cpp(921) [0000000000BFF1B0] WARNING Failed: Hr: = [0x80990c1b] : Encountered Failure: : lVal : GetElementStatus(hDevice, sourceElementType, dwFirstElementNumber, dwNumberOfElemetsPerCall, dwNumberOfElemets, (pLibParams->Features0 & 0x00000001) ? true : false, pStatus, *ppFIResponse)
- dpmlibrarytesttool.cpp(582) WARNING Failed: Hr: = [0x80990c1b] : Encountered Failure: : lVal : deviceLayer.DPMFastInventory( hChangerDevice,&pLibParams,&pFIResponse)
- dpmlibrarytesttool.cpp(907) WARNING Failed: Hr: = [0x80990c1b] : Encountered Failure: : lVal : ExecuteLibraryTest(ssDeviceName, false, dwNumberOfTestCase, arrTestCase)
- dpmlibrarytesttool.cpp(1043) WARNING Failed: Hr: = [0x80990c1b] : Encountered Failure: : lVal : LibraryTestMode(argc-2, wszArgv+2)
We are running Win Server 2019, DPM 2019 and SQL 2017 with the latest updates.
If anyone has seen this before or has any possible solutions it would be greatly appreciated.