770 likes | 781 Views
Learn about the benefits and use cases of HP-UX Dynamic Root Disk (DRD) boot disk cloning. Reduce downtime, simplify software management, and provision systems efficiently.
E N D
HP-UX Dynamic Root DiskBoot Disk Cloning Benefits and Use CasesDusan Baljevic 2013
Acknowledgements These slides have been used in various presentations in Australia over the last four years. This is a work-in-progress. I bear full responsibility for any error, even though it is purely unintentional. I cannot claim credits solely, nor can I claim that I know everything about Unix. I consider myself to be a Unix Apprentice. For that reason I need to give special credits to our colleagues Nobuyuki Hirota (TCE&Q BCS ERT), Daniel Bambou (TC&Q BCS ERT), and Leon Strauss (GSE) for their continuous support, advice, comments, and guidance. Wisdom of many helped in creation of the presentation (seminars at HP, HPWorld, ITRC/HPSC forums, HP Ambassadors and Unix Profession forums, HP Education courses, individual contributions on the Net).
What Kind of Use Cases? This presentation is not displaying formulated textual, structural and visual modeling techniques for specifying use cases with the HP-UX DRD. In software and systems engineering, a use case is a list of steps, typically defining interactions between a role (known in UML as an "actor") and a system, to achieve a goal. The actor can be a human or an external system. In systems engineering, use cases are used at a higher level than within software engineering, often representing missions or stakeholder goals. The detailed requirements may then be captured in SysML or as contractual statements. Rather, in our context, use cases are practical examples of HP-UX DRD usage.
Bootable System Images in Unix/Linux Many tools available. For the sake of brevity, to mention a few: AIX mksysb, Network Installation Manager (NIM) HP-UX make_tape_recovery/make_net_recovery, Dynamic Root Disk (DRD)*, VM mirroring Linux Mondo Rescue, Clonezilla Solaris ufsdump, fssnap+ufsdump, flash/JumpStart Tru64 btcreate
Why Boot Disk Cloning is Critical Today? • Creates a "point-in-time“ O/S image, • On-line patching and configuration changes of the inactive O/S, • Easier change management approvals because the active O/S is not affected (risk is eliminated), • Some tasks make dynamic changes of the O/S during the cloning, without affecting the active O/S, • Boot disk mirroring does not prevent disasters caused by human error, • If boot disks are on the samecontroller, mirroring is not a perfect protection.
Dynamic Root Disk Mission * Significantly reduce the downtime needed to perform HP-UX software maintenance Reduce the downtime required for recovery from administrative errors Perform software update work during normal business hours, or whenever convenient Provision systems quickly and efficiently Simplify testing
Dynamic Root Disk Cycles Provision - [Re-]Ignite - Recover - Clone • Software • Management • - Identify • - Acquire • Organize • Deploy Bare Metal - unused HW Maintain - Monitor - Patches - Applications - Recovery Upgrade or Recycle - update-ux - re-ignite
HP-UX Dynamic Root Disk Features 1 of 4 • Dynamic Root Disk (DRD) provides the ability to clone an HP-UX system image to an inactive disk. • Supported on HP PA-RISC and Itanium-based systems. • Supported on hard partitions (nPars), virtual partitions (vPars), and Integrity Virtual Machines (Integrity VMs), running the following operating systems with roots managed by the following Volume Managers (except as specifically noted for rehosting): o HP-UX 11i V2 (11.23) September 2004 or later o HP-UX 11i V3 (11.31) o LVM (all O/S releases supported by DRD) o VxVM 4.1 o VxVM 5.x
HP-UX Dynamic Root Disk Features 2 of 4 • Product : DynRootDisk Version: A.3.12.316 (DRD_1131_WEB1301.depot) (DRD_1123_WEB1301.depot) • The target disk must be a single physical disk, or SAN LUN. • The target disk must be large enough to hold all of the root volume file systems. DRD allows the cloning of the root volume group even if the master O/S is spread across multiple disks (it is a one-way, many-to-one operation). • On Itanium servers, all partitions are created; EFI and HP-UX partitions are copied. This release of DRD does not copy the HPSP partition. • Copy of lvmtab on the cloned image is modified by the clone operation to contain information that will reflect the desired volume groups when the clone is booted.
HP-UX Dynamic Root Disk Features 3 of 4 • Only the contents of vg00 are copied. • Due to system calls DRD depends on, DRD expects legacy Device Special Files (DSFs) to be present and the legacy naming model to be enabled on HP-UX 11i v3 servers. HP recommends only partial migration to persistent DSFs be performed. • If the disk is currently in use by another volume group that is visible on the system, the disk will not be used. • If the disk contains LVM, VxVM, or boot records but is not in use, one must use the “-x overwrite” option to tell DRD to overwrite the disk. Already-created clones will contain boot records; the drd status command will show the disk that is currently in use as an inactive system image.
HP-UX Dynamic Root Disk Features 4 of 4 • All DRD processes, including “drd clone” and “drd runcmd”, can be safely interrupted issuing Control-C (SIGINT) from the controlling terminal or by issuing kill –HUP <pid> (SIGHUP). This action causes DRD to abort processing. Do not interrupt DRD using the kill -9 <pid> command (SIGKILL), which fails to abort safely and does not perform cleanup. Refer to the “Known Issues” list on the DRD web page (http://www.hp.com/go/DRD) for cleanup instructions after drd runcmd is interrupted. • The Ignite server will only be aware of the clone if it is mounted during a make_*_recovery operation. • DRD Revision A.3.12 DRD supports SoftReboot feature if a machine is installed with SoftReboot on a supported platform.
HP-UX Dynamic Root Disk versus Ignite-UX • DRD has several advantages over Ignite-UX net and tape images: * No tape drive is needed, * No impact on network performance will occur, * No security issues of transferring data across the network. • Mirror Disk/UX keeps an "always up-to-date" image of the booted system. DRD provides a "point-in-time“ image. The booted system and the clone may then diverge due to changes to either one. Keeping the clone unchanged is the Recovery scenario. DRD is not available for HP-UX 11.11, which limits options on those systems.
HP-UX Patching Versus Update-UX 1 of 2 The update-ux method is not only used to update from a lower to a higher version (for example, 11i v2 to v3), but also to update from an older to a newer release within the same version. For many reasons, we encourage usage of update-ux with Dynamic Root Disk (DRD). If O/S is upgraded through update-ux process, the best practice recommends cold installs; incremental upgrades might create possibility that some obsolete software and libraries exist afterwards.
HP-UX Patching Versus Update-UX 2 of 2 We recommend customers develop a release “cycle” through DRD implementation: Run update-ux every year (18 months or maximum two years is acceptable in some circumstances). Only break this cycle if they must have some new functionality in a bi-annual release. Unless specifically requested differently, the patch/update level should be at latest release, if practicable, or LATEST-1.
HP-UX DRD: Minimizing Planned Downtime • DRD enables the administrator to create a point-in-time clone of the vg00 volume group: • Original vg00 image remains active; • Cloned vg00 image remains inactive until needed; • Unlike boot disk mirrors, DRD clones are unaffected by vg00 changes. • DRD is an optional, free product on the 11i v2 and v3 application media. Install patcheson the clone;applicationsremain running lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 boot disk clone disk boot mirror clone mirror vg00(active) cloned vg00 (inactive/patched) Activate theclone to makechanges takeeffect lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 boot disk clonedisk boot mirror clone mirror vg00 (inactive) cloned vg00 (active/patched) 17
DRD Clones Minimize Unplanned Downtime • Without DRD: In case of O/S mis-configuration, it may be necessary to restore from tape. • With DRD: In case of O/S mis-configuration, simply activate and boot the clone. lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 Originalboot VG iscorrupted boot disk clone disk boot mirror clone mirror original vg00 (unusable) cloned vg00 (inactive) lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 Activatethe clone! boot disk clone disk boot mirror clone mirror original vg00 (unusable) cloned vg00 (active) 18
DRD Clones Minimize Planned Downtime • Without DRD: Software and kernel management may require extended downtime. • With DRD: Install/remove software on the clone while applications continue running. Install patches &tune the kernelon the clone;applicationsremain running lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 boot disk clone disk boot mirror clone mirror vg00 (active) cloned vg00 (inactive/patched) Activate theclone to makechanges takeeffect lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 boot disk clone disk boot mirror clone mirror vg00 (inactive) cloned vg00 (active/patched) 19
HP-UX DRD Pros 1 of 2 • Fully supported by HP. • Full clone. • Complements other HP solutions by reducing system downtime required to install and update patches and software. • Copy operation is currently done by fbackup and frecover. • kctune command can be used to modify kernel parameters in the clone. • The ioconfig file and the entire /dev directory are copied by the DRD clone operation, so instance numbers will not change when the clone is booted.* • Supports nPars, vPars, and Integrity VMs.
HP-UX DRD Pros 2 of 2 • No tape drive is needed. • No impact on network performance. • No security issues of transferring data across the network. • All DRD processes, including drd clone and drd runcmd, can be safely interrupted issuing Control-C (SIGINT) from the controlling terminal or by issuing kill -HUP<pid> (SIGHUP). This action causes DRD to abort processing and perform any necessary clean up. Do not interrupt DRD using the kill -9 <pid> command (SIGKILL), which fails to abort safely and does not perform cleanup.
HP-UX DRD Cons 1 of 4 • Target disk must be a single disk or mirror group only. • Not easy to list all differences between Active and Inactive image (drd sync * is the simplistic option). • Cloning should be done when the server’s activity is at a minimum. • DRD can clone root volume group that is spread across multiple disks. The target must be a single disk or mirrored pair.
HP-UX DRD Cons 2 of 4 • Contents of root volume group are copied. A system that has /opt (or any file system that is patched) not in root volume group is not suitable for use with DRD. • Does not provide a mechanism for resizing file systems during a DRD clone operation. However, after the clone is created, you can manually change file system sizes on the inactive system without needing an immediate reboot. The whitepaper, Using the Dynamic Root Disk Toolset describes resizing file systems other than /stand. The whitepaper Using the DRD toolset to extend the /stand file system in an LVM environment describes resizing the boot (/stand) file system on an inactive system image. • Current release of DRD does not copy the Itanium Service Partition (s3 or _p3).
HP-UX DRD Cons 3 of 4 • Command /opt/drd/lbin/drd_scan_hw_host hangs occasionally. This is a hardware issue as it is trying to scan all connected hardware. Check it before using DRD and maybe even remove stale devices with rmsf –x if necessary: # ioscan -s # lssf -s • Too many tiny files on root disks can cause significant performance problem when DRD is used. When there are large number of files in the root VG (for example, two millions), drd clone / drd sync might fail with error "Out of memory". It is suggested to increase maxdsiz kernel parameter, or use "-x exclude_list" option, or remove unnecessary user files.
HP-UX DRD Cons 4 of 4 • We might see the following error message during the execution of drd runcmd if the nsswitch.conf file contains the "hosts: nis" entry: Error: Could not contact host "myserver". Make sure the hostname is correct and an absolute pathname is specified (beginning with "/"). • We might see the following error message during the execution of drd runcmd if the nsswitch.conf file contains the "passwd: compat" or "group: compat" entries: Error: Permission is denied for the current operation. There is no entry for user id 0 in the user database. Check /etc/passwd and/or the NIS user database.
Supported Versions of DRD • Versions of DRD are supported for at least two years. • Versions not listed in the "Supported DRD Releases" section of the latest Release Notes are no longer supported. • We always recommend to have the latest DRD installed.
Installing DRD • DRD is included in current 11i v2 and v3 operating environments or ... • Download and install DRD from http://software.hp.com Install DRD with swinstall (no reboot required) # swinstall –s /tmp/DynRootDisk*.depot DynRootDisk 29
DRD Commands Most DRD tasks require a single command, drd, which supports multiple “modes”. Example # drd clone –t /dev/disk/diskY–x overwrite=true Other available modes # drd view available modes and options # drd clone ... create a DRD clone # drd mount ...mount the DRD clone’s file systems # drd umount ...unmount the DRD clone’s file systems # drd runcmd ... execute a command on the clone’s file systems # drd activate ...make the DRD clone the default boot disk after next reboot# drd deactivate retain the current active image as the default boot disk# drd status display information about active/inactive DRD images DRD offers several common options that are supported in all modes# drd mode -? view available options # drd mode –x ? view available extended options # drd mode [-x verbosity=3] ... specify stdout/stderr verbosity, 0-5 # drd mode [-x log_verbosity=4] ... specify log file verbosity, 0-5# drd mode [-qqq|qq|q|v|vv|vvv] ...alternative to –x verbosity=n# drd mode [–p] ... preview but don’t execute the operation 30
HP-UX DRD Restrictions on update-ux and sw* Commands Invoked by drd runcmd • Options on the Software Distributor commands that can be used with drd runcmd need to ensure that operations are DRD-safe: • The -F and -x fix=true options are not supported for drd runcmd swverify operations. Use of these options could result in changes to the booted system. • The use of double quotation marks and wild card symbols (*, ?) in the command line must be escaped with a backslash character (\), as in the following example: # drd runcmd swinstall –s depot_server:/var/opt/patches \* • Files referenced in the command line must both: o Reside in the inactive system image o Be referenced in the DRD-safe command by the path relative to the mount point of the inactive system image • This applies to files referenced as arguments for the -C, -f, -S, -X, and -x logfile options for an sw command run by drd runcmd and update-ux command -f option.
HP-UX Issue when DRD versions different in booted and cloned environment 1 of 2 # drd runcmd swinstall -s /tmp/ignite/Ignite-UX-11-ALL_C.7.7.98.depot • ======= 11/28/12 00:42:22 IST BEGIN Executing Command On • … /opt/drd/wrappers/start_fsdaemon[22]: start_fsdaemon: not found. * Stopping swagentd for drd runcmd /opt/drd/wrappers/stop_fsdaemon[22]: stop_fsdaemon: not found. ERROR: Command executed on inactive system image returned an error - One or more postcommands for /usr/sbin/swinstall failed. - One or more precommands for /usr/sbin/swinstall failed. /usr/sbin/swinstall will not be executed. - The precommand "/opt/drd/wrappers/start_fsdaemon" fails with the return code "1". - The postcommand "/opt/drd/wrappers/stop_fsdaemon" fails with the return code "1". Executing Command On Inactive System Image failed with 1 error.* Cleaning Up After Command Execution On Inactive System Image
HP-UX Issue when DRD versions different in booted and cloned environment 2 of 2 This problem is triggered by having one version of DRD installed on the booted system, and a previous release on the inactive image. If the clone is not very new, just re-run drd clone. If you do not want to re-create the clone, the following workaround will help: # drd mount # cp /var/opt/drd/mnts/sysimage_001/opt/drd/wrappers/common_utils /var/opt/drd/mnts/sysimage_001/opt/drd/wrappers/common_utils.orig # cp /opt/drd/wrappers/common_utils /var/opt/drd/mnts/sysimage_001/opt/drd/wrappers/common_utils (If you are booted on the clone, replace "sysimage_001" with "sysimage_000".) The steps above will enable drd runcmd to succeed. However, the file change would cause a swverify error on the version of DRD in the clone. To repair this, install the new version of DRD to the inactive image: # drd runcmd swinstall -s <depot> DynRootDisk
HP-UX DRD Updates from multiple-DVD media DRD updates directly from media require the September 2010 OE (or later) versions of DRD, SWM and SW-DIST products. In order to use a media depot to do a DRD update, first install September 2010 or later versions of DRD, SWM, and SW-DIST products from the media. This must be done before the clone is created, so the new DRD, SWM, and SW-DIST are on the active system and on the clone.
Creating and Updating DRD Clone • Use the drd clone command to create a DRD clone of the active boot disk: • DRD identifies the current active boot disk • DRD builds a similarly structured clone disk • DRD copies the current disk’s file system contents to the clone • DRD builds a mirror of the clone, too, if requested • DRD records log messages in /var/opt/drd/drd.log Identify available disk(s) # ioscan –funC disklist all disks on the system# lvmadm –l or strings /etc/lvmtab*which disks are LVM disks?# vxdisk listwhich disks are VxVM disks?# diskinfo/dev/rdisk/disk3 verify the disk sizeClone the current active boot disk # drd clone –t /dev/disk/disk3 \specify a target disk (required!)[–x overwrite=true] \overwrite data on target [-x mirror_disk=/dev/disk/disk4]create a mirror of the DRD Update an existing clone (overwrite=true required!) # drd clone –t /dev/disk/disk3 \specify a target disk (required!) –x overwrite=true \overwrite data on target [-x mirror_disk=/dev/disk/disk4]create a mirror of the DRD 37
Verifying DRD Clone Status # drd status ======= 07/23/08 12:13:57 EDT BEGIN Displaying DRD Clone Image Information (user=root) (jobid=myhost) * Clone Disk: /dev/disk/disk3 * Clone EFI Partition: Boot loader and AUTO file present * Clone Creation Date: 07/18/08 21:07:29 EDT * Clone Mirror Disk: None * Mirror EFI Partition: None * Original Disk: /dev/disk/disk1 * Original EFI Partition: Boot loader and AUTO file present * Booted Disk: Original Disk (/dev/disk/disk1) * Activated Disk: Original Disk (/dev/disk/disk1) ======= 07/23/08 12:14:04 EDT END Displaying DRD Clone Image Information succeeded. (user=root) (jobid=myhost) 38
DRD-Safe Commands • Files in the inactive system image are not accessible, by default, to HP-UX commands. • “DRD-Safe” commands can be executed on the inactive image via drd runcmd • Temporarily imports and mounts the inactive image’s volume group and file systems, • Executes the specified command using executables & files on the inactive image, • Ensures that the active image remains untouched, • Unmounts and exports the inactive image’s file systems and volume group. • DRD-safe commands currently include: swinstall swremove swlist swmodify swverify swjob kctune update-ux view
Managing Patches with DRD-Safe Commands • Installing patches and software sometimes requires a reboot and downtime. • Minimize downtime by installing software/patches/updates on an inactive image. • Changes take effect when you activate and boot the inactive image. • Only DRD-Safe patches/products can be installed via DRD. List software installed on the inactive image using the DRD-Safe swlist command # drd runcmd swlist Check if product or patch is DRD-Safe # swlist –l fileset –a is_drd_safe product_name|patch Install software on the inactive image using the DRD-Safe swinstall command # drd runcmd swinstall –s server:/mydepot PHSS_NNNNN Remove software from the inactive image using the DRD-Safe swremove command # drd runcmd swremove PHSS_NNNNN View the inactive image SDUX log file using the DRD-Safe view command # drd runcmd view /var/adm/sw/swagent.log Update to a more recent 11i v3 media kit # drd runcmd swinstall –s server:/mydepot Update-UX# drd runcmd update-ux –s server:/mydepot# drd runcmd view /var/adm/sw/update-ux.log # drd runcmd view /var/opt/swm/sw.log 40
Accessing DRD Inactive Images • The drd runcmd utility only executes DRD-safe executables on an inactive image. • To access other files on the inactive image, mount the image via drd mount • Imports the inactive image volume group, typically as drd00, • Mounts the image file systems under /var/opt/drd/mnts/sysimage_001 • Warnings: • Be careful not to unintentionally modify the active system image! • Only use read-only commands like view and diff to access inactive images. Mount the inactive image file systems # drd mount# mount -v Access the inactive image file systems, being careful not to modify the active image!# diff /etc/passwd /var/opt/drd/mnts/sysimage_001/etc/passwd Unmount the inactive image file systems # drd umount 41
Activating and Deactivating Inactive DRD Image • Use drd activate to make the inactive image the primary boot disk • DRD updates the boot menu • DRD can optionally reboot the system immediately Promote the inactive system image to become primary boot disk (with preview) # drd activate [-x reboot=false] -p Check the bootpath # setboot -v If –x reboot=true wasn’t specified, manually reboot# shutdown –ry 0 If you change your mind before rebooting, use drd deactivate to undo the activation# drd deactivate Use drd status to determine which disk is the currently active boot disk # drd status 42
DRD Inactive Image Synchronization • The drd sync command was introduced in release B.11.xx.A.3.5 of Dynamic Root Disk (DRD) to propagate root volume group file system changes from the booted original system to the inactive clone image. Running drd sync command updates/creates the files on Inactive Image (Clone Disk) which were modified on Active Image (Boot Disk) after last successful execution of drd clone command. • pax archive is used for drd sync while fbackup/frestore is used for clone. • To preview differences between the Active Image and the DRD Inactive Image • # drd sync –p • It creates file /var/opt/drd/sync/files_to_be_copied_by_drd_sync • Once the preview is checked, a resync of the cloned image can be initiated • # drd sync 43
drd sync • With DRD Sync • A system administrator creates a DRD clone on a Thursday. • The administrator applies a collection of software changes to the clone on Friday using the drd runcmd command. • On Friday, several log files are updated on the booted system. • On Saturday, the clone is synced then booted – log files and other files that have changed on the original system have automatically been copied to the clone. • Without DRD Sync • A system administrator creates a DRD clone on a Thursday. • The administrator applies a collection of software changes to the clone on Friday using the drd runcmd command. • On Friday, several log files are updated on the booted system. • On Saturday, the clone is booted, however the log files are not up to date, so the administrator must copy over the log files and any other files from the original system that changed after the clone was created – for example, /etc/passwd
drd sync • The list of files on the active system whose modification date is newer than or equal to the clone creation time provides the initial list of files to be synchronized Trimming the list of files to be synchronized The following locations are not synchronized: /var/adm/sw, /tmp, /var/tmp, /var/opt/drd/tmp, /stand, /dev/<clone_group>, Files that have changed on the clone are not synchronized Nonvolatile files in the Software Distributor Installed Products Database (IPD) are not synchronized Volatile files in the Software Distributor Installed Products Database (IPD) are not synchronized
HP-UX DRD Examples for Different O/S Releases HP-UX 11iv2: # drd clone -t /dev/dsk/c2t1d0 -x \ overwrite=true [-x mirror_disk=/dev/dsk/c3t0d1] HP-UX 11iv3, use agile views: # drd clone -t /dev/disk/disk32 -x \ overwrite=true [-x mirror_disk=/dev/disk/disk4] Note that all partitions on Itanium disk are created, and s1 and s2 (_p1 and _p2) are copied.
HP-UX 11i v2 To v3 Upgrade with DRD 1 of 3 Original image: /dev/dsk/c0t0d0 (HP-UX 11i v2) Clone disk: /dev/dsk/c1t0d0 What to apply: HP-UX 11i v3 Update 9, Virtual Server OE Depot with patches depsvr:/var/depots/1131_VSE-OE Version of DRD: B.11.31.A.3.3 or later Objective: Utilize DRD to help adjust file systems sizes when performing an HP-UX 11i v2 to v3 update • Create the clone: # drd clone –t /dev/dsk/c1t0d0 • Use drd status to view the clone: # drd status
HP-UX 11i v2 To v3 Upgrade with DRD 2 of 3 • Run update-UX in preview mode on the active disk: # update-ux -p -s depsvr:/var/depots/1131 HPUX11i-VSE-OE • Adjust file system sizes on the clone as needed • Activate and boot the clone, setting the alternate bootpath to the HP-UX 11i v2 disk: # drd activate -x alternate_bootdisk=/dev/dsk/c0t0d0 -x reboot=true • Update the active image to HP-UX 11i v3, Virtual Server OE: # update-ux -s depsvr:/var/depots/1131_VSE-OE HPUX11i-VSE-OE There will be a reboot executed at this time.
HP-UX 11i v2 To v3 Upgrade with DRD 3 of 3 • Ensure that the software is installed properly: # swverify \* • Verify all software has been updated to the HP-UX 11i v3: # swlist • Ensure the integrity of your updated system by checking the following log files /var/adm/sw/update-ux.log and /var/opt/swm/swm.log
HP-UX How to Interrupt DRD processes All DRD processes, including “drd clone” and “drd runcmd”, can be safely interrupted issuing Control-C (SIGINT) from the controlling terminal or by issuing kill –HUP <pid> (SIGHUP). This action causes DRD to abort processing. Do not interrupt DRD using the kill -9 <pid> command (SIGKILL), which fails to abort safely and does not perform cleanup. Refer to the “Known Issues” list on the DRD web page (http://www.hp.com/go/DRD) for cleanup instructions after drd runcmd is interrupted.