Matthew D. Hendel Software Engineer BaseOS/Drivers

1. Introducing Storport Matthew D. Hendel Software Engineer BaseOS/Drivers

2. Contents • This presentation will be an introduction to Storport programming. The target audience are those people familiar with storage devices and programming, but not well acquainted with programming Storport or SCSIport. • Throughout this presentation we will be using the I2O sample to discuss Storport concepts. The sample is freely available in the DDK and the I2O specification that was used to create this sample is available on the web. • Storport Advanced Topics discusses advanced Storport programming in more depth.

3. Outline • I/O architecture – Review of the I/O architecture in the Windows Operating Systems • Discuss Storport features using I2O sample miniport as an example • Miniport Callbacks • Buffers exposed by the port driver, explanation of Common Buffer, Device Extension Uncached Extension, LUN Extension, SRB Extension • HW_INITIALIZTAION_DATA and PORT_CONFIGURATION_DATA • I2O Hardware and I2O message format • Walk through I2O callbacks HwFindAdapter, HwInitialize, HwStartIo, HwBuildIo, HwBusReset, HwAdapterControl • Walk I2O Inquiry and Write commands through submission and translation of return values

4. I/O Subsystem • I/O Subsystem • Decides if IRP is necessary • Creates and initializes IRP • Initializes top of IRP stack to request type (Read, Write, Flush, etc.) • Passes IRP to top level driver, File System, in our example I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

5. File System • Imposes file structure on raw disk • Passed in a file object, offset and length, translates into volume offset and length • Last driver that is within the process context of the calling process • Implies all other drivers must not access the StartVa field of the MDL I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

6. Volume Shadow Copy • Takes snapshots of volumes • Snapshots used for rollback of files • Used for live backup I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

7. Volume Manager • Volume managers present user level volumes, “C:”, “D:”, etc. • Devices can come and go, volumes are permanent • Two volume managers • Basic disks use FtDisk as a volume manager • Dynamic disks use DMIO as a volume manager • Bottom of Volume Stack I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

8. Partition Manager • Partition Manager tells volume manager when partitions come and go • Also has performance counters for device I/O • Top of the device stack I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

9. Class Driver • Class drivers manage a specific class of devices • Disk for disk devices • CD-ROM for CD-ROMs • Tape class driver for tapes • Class uses a separate SCSI-based interface to communicate with Port Drivers • Class/Port interface based on the SCSI_REQUEST_BLOCK data structure I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

10. Port Driver • Port Layer manages a specific transport • SCSIPORT for SCSI buses • SBP2PORT for 1394 buses • ATAPI for IDE buses • Port layer may translate the SCSI SRB to hardware-specific request, e.g., IDE • Port driver may create a miniport model for hardware specific details I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

11. Miniport • Miniport is a vendor-supplied driver used to access hardware • Used to hide differences in hardware • Port driver defines port/miniport interface – there may be multiple port/miniport interfaces I/O Subsystem File System Volume Shadow Copy Volume Manager Partition Manager Class Miniport Port Hardware

12. Example: I2O Miniport

13. Miniport Callbacks • FindAdapter – Detect if adapter is present and operating • Initialize – Initialize adapter previously found via Find Adapter • Interrupt – Handle adapter interrupt • StartIo – Issue I/O command to adapter • ResetBus – Reset a bus • AdapterControl – Stop or restart an adapter • BuildIo (Optional) – Perform deserialized per command processing

14. Find Adapter • Called to discover an adapter • Holdover from pre-PnP days • Now used to perform basic initialization and reset the hardware • Interrupts are not connected at the time that Find Adapter is called

15. I2O Find Adapter I2O FindAdapter pseudocode if (InLdrOrDump (ArgumentString)) deviceExtension->Flags |= FLAGS_LOADER_OR_DUMP; newAddress = (ULONG_PTR)StorPortGetDeviceBase( deviceExtension, PCIBus, ConfigInfo->SystemIoBusNumber, accessRange->RangeStart, accessRange->RangeLength, !accessRange->RangeInMemory); I2O InitializeDeviceExtension (deviceExtension); I2O InitializePortConfiguration (ConfigInfo); deviceExtension->ReplyBuffer = StorPortGetUncachedExtension ( deviceExtension, ConfigInfo, totalSize); I2OResetIop (deviceExtension); • Check if we are running under loader or crashdump. This is important because under loader and crashdump, we have limited resources. • Map the I/O Processor’s access range into memory, either in I/O space or in virtual address space • Initialize the device extension • Initialize the Port Configuration Information • Get a pointer to the uncached extension for the device • Reset the I/O controller by sending it ExecIopReset message

16. Storport Buffers • Storport does not provide direct buffer allocate and free routines, instead it provides a set of per-object buffers used by the miniport. The buffers are implicitly allocated and freed by Storport on behalf of the miniport. The four buffers are: • Device Extension • Uncached Extension • SRB Extension • Logical Unit Extension • Two basic buffer types • Common Buffer is physically contiguous region of memory that, under Storport, will not exceed the four gigabyte range • Pool is any other buffer, which is not guaranteed to be physically contiguous or below four gigabytes

17. Pool Memory • Pool memory is not physically contiguous • Difficult to share between hardware and software Virtual Address Physical Address F0000000 F0001000 F0002000 F0003000 F0004000 F0005000 F0006000 F0007000 F0008000 00000000 00001000 00002000 00003000 00004000 00005000 00006000 00007000 00008000

18. Common Buffer • Common Buffer is physically contiguous, making it easy to share data structures between software and hardware • In Storport and SCSIPORT Common Buffer is below 4 GB Physical Address Virtual Address 00000000 00001000 00002000 00003000 00004000 00005000 00006000 00007000 00008000 F0000000 F0001000 F0002000 F0003000 F0004000 F0005000 F0006000 F0007000 F0008000

19. Storport Buffers • Device Extension • Passed in by Storport for each miniport callback • Not physically contiguous • Anywhere in physical address space • Uncached Extension • Obtained through StorPortGetUncachedExtension call • Physically contiguous • Below 4 GB • Logical Unit Extension • Obtained through StorPortGetLogicalUnit call • Not physically contiguous • Anywhere in physical address space • SRB Extension • SrbExtension field of SRB • Physical contiguous if spans a page • Below 4 GB

20. Miniport Callback Data Structures • HW_INITIALIZATION_DATA • Device Extension Size • Logical Unit Extension Size • SRB Extension Size • Map Buffers • PORT_CONFIGURATION_INFORMATION • Many of the fields from HwInitializationData are the same • SynchronizationModel • MaximumNumberOfLogicalUnits • Dma64BitAddress • CachesData • MaximumTransferLength

21. I2O Hardware • Two FIFOs allocated from host memory • Inbound FIFO to I/O Processor • Outbound FIFO from I/O Processor • Each FIFO contains a Free_List and a Post_List; the Free_List contains unused entries for the FIFO and the Post_List contains queued entries Inbound (to IOP) Free_List Post_List Outbound (from IOP) Free_List Post_List

22. I2O Hardware • Packet architecture with separate submission and completion FIFOs • Command Submission • Remove entry from Inbound Free_List • Fill in packet • Submit to Inbound Post_List • Command Completion • Remove entry from Outbound Post_List • Process command • Return it to Outbound Free_List • I2O specific command sets • Random Block Commands • Sequential Storage • LAN • WAN • Fibre Channel • SCSI Peripheral • Executive

23. I2O Request Message Format 3 2 1 0 31 24 23 16 15 8 7 0 0 Message Size Flags Version Offset 4 Function Initiator Address Target Address 8 Initiator Context 12 Transaction Context 16 Message Details (Function Specific) n Scatter Gather List

24. I2O Reply Message Format 3 2 1 0 31 24 23 16 15 8 7 0 0 Message Size Flags Version Offset 4 Function Initiator Address Target Address 8 Initiator Context 12 Transaction Context 16 Status reserved Detailed Status Code 20 Reply Payload

25. I2O Scatter Gather List Format 3 2 1 0 31 24 23 16 15 8 7 0 1 Flags Scatter Gather Element 2 Flags Scatter Gather Element 3 Flags Scatter Gather Element

26. I2O Simple Scatter Gather Element Format 3 2 1 0 31 24 23 16 15 8 7 0 0 Flags Byte Count 0 Optional Buffer Context 4 Physical Address

27. I2O Find Adapter I2OFindAdapter pseudocode if (InLdrOrDump (ArgumentString)) deviceExtension->Flags |= FLAGS_LOADER_OR_DUMP; newAddress = (ULONG_PTR)StorPortGetDeviceBase( deviceExtension, PCIBus, ConfigInfo->SystemIoBusNumber, accessRange->RangeStart, accessRange->RangeLength, !accessRange->RangeInMemory); I2O InitializeDeviceExtension (deviceExtension); I2O InitializePortConfiguration (ConfigInfo); deviceExtension->ReplyBuffer = StorPortGetUncachedExtension ( deviceExtension, ConfigInfo, totalSize); I2O ResetIop (deviceExtension); • Check if we are running under loader or crashdump. This is important because under loader and crashdump, we have limited resources. • Map the I/O Processor’s access range into memory, either in I/O space or in virtual address space • Initialize the device extension • Initialize the Port Configuration Information • Get a pointer to the uncached extension for the device • Reset the I/O processor by sending it ExecIopReset message

28. I2O Hardware Initialize • Setup Inbound and Outbound FIFOs • Enable interrupts • Setup other hardware parameters • Note that interrupts are enabled by the port driver upon exit

29. I2O Hardware Initialize I2O Initialize pseudocode if (!I2OInitOutbound(deviceExtension)) { return SP_INTERNAL_ADAPTER_ERROR; } if (StorPortReadRegisterUlong (deviceExtension, deviceExtension->IntControl) != 0)) { StorPortWriteRegisterUlong (deviceExtension, deviceExtension->IntControl, 0); } More hardware initialization if (!I2OClaimDevice(deviceExtension)) { return SP_INTERNAL_ADAPTER_ERROR; } if (!I2ORegisterForEvents(deviceExtension, I2OMP_EVENT_MASK)) { return SP_INTERNAL_ADAPTER_ERROR; } • Initialize outbound FIFOs • Enable interrupts in the hardware; note that interrupts are still being held off by the port driver we need to poll for results when sending the next two messages • I2O specific hardware initialization • Notify the device that we intend to send commands to it using UtilClaim message • Register for hardware events from the device using UtilEventRegister message

30. I2O StartIo Routine I2OStartIo pseudocode switch (Cdb->ScsiCommand) { case SCSIOP_MEDIUM_REMOVAL: Srb->SrbStatus = SRB_STATUS_SUCCESS; break; case SCSIOP_REQUEST_SENSE: Srb->SrbStatus = SRB_STATUS_INVALID_REQUEST; break; case SCSIOP_MODE_SENSE: Srb->Status = I2OModeSense (DeviceExtension, Srb); break; case SCSIOP_READ_CAPACITY: Srb->Status = I2ODriveCapacity (DeviceExtension, Srb); break; case SCSIOP_READ: case SCSIOP_WRITE: Srb->Status = I2OReadWrite (DeviceExtension, Srb); break; case SCSIOP_INQUIRY: Srb->SrbStatus = I2OInquiry (DeviceExtension, Srb); break; • Start I/O routine translates between SCSI command and I2O command

31. I2O Block Commands

32. I2O Miscellaneous Commands

33. SCSI Primary And Block Commands

34. I2O Write Command I2O BuildReadWrite pseudocode msg->Write.StdMessageFrame.VersionOffset = I2O_VERSION_11 | I2O_WRITE_SGL_VER_OFFSET; msg->Write.StdMessageFrame.MessageSize = messageSize; msg->Write.StdMessageFrame.TargetAddress = deviceInfo->Lct.LocalTID; msg->Write.StdMessageFrame.InitiatorAddress = I2O_HOST_TID; msg->Write.StdMessageFrame.Function = I2O_BSA_BLOCK_WRITE; msg->Write.StdMessageFrame.InitiatorContext = 0; if (cdb->CDB10.ForceUnitAccess == TRUE) msg->Write.ControlFlags = I2O_BSA_WR_FLAG_WRITE_THRU; callback->Callback.CallbackAddress = I2OWriteCompletion; blockOffset.QuadPart *= deviceInfo->BlockSize; msg->Read.LogicalByteAddress.HighPart = blockOffset.HighPart; msg->Read.LogicalByteAddress.LowPart = blockOffset.LowPart; Build I2O scatter-gather list from Storport scatter-gather list • Build an I2O write message in the SRB extenion • Translate SCSI ForceUnitAccess flag to I2O write through • Set completion routine • Translate SCSI block addressing to I2O byte addressing mode • Build the I2O scatter-gather list

35. I2O Write Command I2OSendReadWrite pseudocode messageFrameAddress = StorPortReadRegisterUlong ( DeviceExtension, DeviceExtension->MessageAlloc); if (messageFrameAddress == (ULONG)-1) { return FALSE; } StorPortWriteRegisterBufferUlong( DeviceExtension, (PULONG)((PUCHAR)DeviceExtension->MappedAddress + messageFrameAddress), (PULONG)MessageBuffer, MessageBuffer->MessageSize); StorPortWriteRegisterUlong (DeviceExtension, DeviceExtension->MessagePost, messageFrameAddress); • Get the next available message from the Inbound Free_List • If there is no free message available, return failure • Copy the contents of the I2O message buffer to allocated message frame • Post the message to the Inbound Post_List

36. I2O Inquiry Command I2OInquiry pseudocode msg.StdMessageFrame.VersionOffset = I2O_VERSION_11 | I2O_PARAMS_SGL_VER_OFFSET; msg.StdMessageFrame.MsgFlags = 0; msg.StdMessageFrame.TargetAddress = deviceInfo->Lct.LocalTID; msg.StdMessageFrame.InitiatorAddress = I2O_HOST_TID; msg.StdMessageFrame.Function = I2O_UTIL_PARAMS_GET; msg.StdMessageFrame.InitiatorContext = 0; callback->Callback.CallbackAddress = I2OInquiryCallBack; scalar = (PI2O_PARAM_SCALAR_OPERATION)((PUCHAR)&msg.SGL + sizeof(I2O_SGE_IMMEDIATE_DATA_ELEMENT)); scalar->OpList.OperationCount = 1; scalar->OpBlock.Operation = I2O_PARAMS_OPERATION_FIELD_GET; scalar->OpBlock.FieldCount = (USHORT) -1; // Get all fields in the group • Setup the I2O UtilParamsGet message • Setup a completion routine to translate I2O Parameter data into SCSI Inquiry data • Get all data parameter data

37. I2O Interrupt I2O Interupt pseudocode msg = StorPortReadRegisterUlong ( DeviceExtension, DeviceExtension->MessageReply); while (msg != -1) { callBack = I2OGetMessageCallbackRoutine (msg); callBack (msg); StorPortWriteRegisterUlong (DeviceExtension, DeviceExtension->MessagePost, msg); msg = StorPortReadRegisterUlong ( DeviceExtension, DeviceExtension->MessageReply); } • Read the message from the Outbound Post_List • While there are still valid messages • Get and invoke the per-message callback routine • Return the message to the Outbound Free_List • Read the next message from the I/O processor

38. I2O Callback for Write Command I2O WriteCompletion pseudocode flags = Msg->BsaReplyFrame.StdMessageFrame.MsgFlags; if (flags & I2O_MESSAGE_FLAGS_FAIL) { I2OHandleError (msg, DeviceExtension, TransactionCxtOffset); return; } if (srb->DataTransferLength > Msg->TransferCount) { srb->SrbStatus = SRB_STATUS_DATA_OVERRUN; srb->DataTransferLength = Msg->TransferCount; } else if (srb->DataTransferLength < Msg->TransferCount) { srb->SrbStatus = SRB_STATUS_ERROR; } else { srb->SrbStatus = SRB_STATUS_SUCCESS; } StorPortNotification (RequestComplete, DeviceExtension, Srb); • Handle any errors • Translate I2O transfer count to SRB DataTransferLength, noting if there was an overrun or underrun • Notify the port driver that the command has been completed.

39. I2O Callback for Inquiry Command I2O InquiryCallback pseudocode flags = Msg->BsaReplyFrame.StdMessageFrame.MsgFlags; if (flags & I2O_MESSAGE_FLAGS_FAIL) { I2OHandleError (msg, DeviceExtension, TransactionCxtOffset); return; } inquiryData->DeviceType = i2OData->DeviceType; inquiryData->RemovableMedia = i2OData->DeviceCapabiltiy & I2O_BSA_DEV_CAP_REMOVABLE_MEDIA; inquiryData->VendorId = “Vendor ”; inquiryData->ProductId = “I2O Disk ”; inquiryData->ProductRevisionLevel = “2.00”; StorPortNotification (RequestComplete, DeviceExtension, Srb); • Handle error if there was one • Translate device type • Translate removable media bit • Hardcode vendor, product and revision level • Finally, complete the request back to the port driver

40. I2O Bus Reset I2O BusReset pseudocode status = I2OGetStatus (DeviceExtension); if ( status is operational ) { I2OBSADeviceResetAll (deviceExtension); } else { I2OSysQuiesce (DeviceExtension); I2OResetIop (DeviceExtension); I2OInitialize (DeviceExtension); } StorPortCompleteRequest (DeviceExtension, PathId, 0xFF, 0xFF, SRB_STATUS_BUS_RESET); • Get status from the I/O processor • If the I/O processor is working send BsaReset message to each target • Otherwise, send a ExecSysQuiesce message to the I/O Processor then reinitialize it • Complete all outstanding I/Os to port driver with status BUS RESET

41. I2O Adapter Control Routine I2O AdapterControl pseudocode case ScsiStopAdapter: I2OSysQuiesce (DeviceExtension); I2ODisableInterrupts (DeviceExtension); break; case ScsiRestartAdapter: I2OEnableInterrupts (DeviceExtension); I2OResetIop (DeviceExtension); I2OInitialize (DeviceExtension); StorPortNotification (BusChangeDetected, DeviceExtension, 0); break; • Send ExecSysQuiesce message to I/O Processor • Disable interrupts from the processor • Enable Interrupts from the processor • Reset I/O processor • Reinitialize the I/O processor • Tell the port driver to re-enumerate the bus in case hardware has been added or removed

42. Conclusion • We have looked in depth at the routines required to write a Storport driver. Samples for these routines have been reviewed and discussion of implementation details. • This information should enable you to understand the Storport model, and begin work on a Storport driver. • Please attend the Storport Advanced Topics session following this one for more in depth information on writing a Storport driver.

43. Resources • Further information on Storport is in available the Microsoft Windows Server 2003 DDK. • Windows Server 2003 DDK has full source for Storport I2O sample driver. • I2O web site www.intelligent-io.com has complete specs for I2O. This is very useful when reading through the I2O sample miniport. • Links • http://www.microsoft.com/windowsserversystem/storage/storport.mspx • http://www.microsoft.com/whdc/hwdev/tech/storage/entstor/default.mspx

44. © 2003 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.