Repack Operational Items Tim Bell Charles Curran Gordon Lee July 1 st 2008

1. Repack Operational Items Tim Bell Charles Curran Gordon Lee July 1st 2008

2. Bulk Repack Outlook • Bulk repack is much more difficult use case than repairing bad tapes or reclaiming holes • Time pressure • Repack to free slots or stop an old robot • Large scale drive allocation during campaign so downtime means lost resources • Volumes are much bigger • 100s of tapes per day • Requires planning and careful monitoring of progress and resouces used 2

3. Operational Items • Reliability needs further improvement • Migrations stuck • Stageins failing and tapes being remounted • Problem diagnosis remains difficult • Is it running OK or blocked ? • Why did a repack fail ? Repack –e will help but not cover many errors such as checksums or bad blocks • Still requires extensive developer debug and hand-holding. Support is limited when key developer is away • Workload management required • User must not submit too many requests in parallel or too close together to avoid meltdowns • Rtcpclientd dying with manual cleanup • Migrator problems with bestTapeForCopy timeouts • Repack server throttles stage-in requests to maximum 500 files at one time 3

4. Operational Items (cont) • Selection of destination tape pool • Allow multiple users of single repack instance • Tape recovery in parallel in c2public (rather than using c2pps) • Defragmentation • Avoid need for wrapper scripts • Support bulk repack • Repack by smallest number of files not by pool • Unmounts on failure (#35319) • When an error occurs, the tape is unmounted and remounted rather than skipping the bad file • Cannot repack a disabled tape (#35953) • If a tape has been disabled to stop the users reading the files, the repack program cannot process it. It must be enabled first and therefore the user’s recalls can also proceed giving errors. 4

5. Operational Items (cont) • Recaller is stuck if tape repacked (#32430) • If a recall of a tape occurs while it is being repacked, the recall will block. The recall needs to be manually cancelled. This is because it tries to read the file from the tape where it originally resided when the recall job was submitted even though it has moved to another tape. • Handling bad and disabled segments (#31772) • An interface is needed to handle disabling tape segments in an automated fashion so that bad files on tape can be quickly marked as inaccessible. • Further enhancements not yet raised • Move tape to different pool on completion or reclaim tape • Cancel repack request should cancel outstanding recalls • Log of progress (start/regular snapshot/completed/rate/volume) • Being fixed but not yet in production • Repack double copy files • Dedicated thread pool for repack to avoid starvation of other clients • Files not being recalled in correct order • Repack server locks up when archiving certain requests • Repack –e provides improved error reporting of stager errors 5

6. Additional Information • Repack Options • https://twiki.cern.ch/twiki/bin/view/FIOgroup/TapeBulkRepack • Repack Performance Analysis • http://it-div-ds.web.cern.ch/it-div-ds/HO/repack_challenge.html • Label Options • https://twiki.cern.ch/twiki/bin/view/FIOgroup/TapeLabelOptions 6