1 / 21

IN2P3 HPSS Migration report V 5.1  6.2.2.2

IN2P3 HPSS Migration report V 5.1  6.2.2.2. Pierre-Emmanuel Brinette IN2P3 Computing Center (Lyon – France) pbrinette@cc.in2p3.fr. Overview. Context of the migration What’s new in HPSS 6.2 IN2P3 HPSS Configuration Migration Planning Metadata Migration Conclusion. Context of migration.

oihane
Download Presentation

IN2P3 HPSS Migration report V 5.1  6.2.2.2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IN2P3 HPSS Migration reportV 5.1  6.2.2.2 Pierre-Emmanuel Brinette IN2P3 Computing Center (Lyon – France) pbrinette@cc.in2p3.fr

  2. Overview Context of the migration What’s new in HPSS 6.2 IN2P3 HPSS Configuration Migration Planning Metadata Migration Conclusion Pierre-Emmanuel Brinette

  3. Context of migration • IN2P3 use HPSS as main MSS since 1999. • The last major upgrade (to v 5.1) has been made in 2005. • Migration to HPSS 6.2 planned since 2007: • HPSS 6.2.1 test platform setup in mid 2007. • IN2P3 HPSS staff has changed in 2008. • The engineers that manage HPSS leave the center. • The new team has no experience in HPSS. • In 2009 upgrade to HPSS 6.2 is necessary • Support for T10k-B drives. • Remove End-Of-Life equipments. Pierre-Emmanuel Brinette Wednesday, August 27, 2014 3

  4. Overview • Context of the migration • What’s new in HPSS 6.2 • IN2P3 HPSS Configuration • Migration Planning • Metadata Migration • Conclusion Pierre-Emmanuel Brinette Wednesday, August 27, 2014 4

  5. What’s new in HPSS 6.2.2.2 • Major changes: • DCE is no longer used as security mechanism. • Some changes in the HPSS API. • New Authentication/Authorization mechanisms: • Unix/Unix : /etc/passwd or NIS • Kerberos/LDAP. • Officially supported platform: • AIX 5.3 and RHEL 4.7. • DB2 v 9.1. Pierre-Emmanuel Brinette Wednesday, August 27, 2014 5

  6. Overview • Context of the migration • What’s new in HPSS 6.2 • IN2P3 HPSS Configuration • Migration Planning • Metadata Migration • Conclusion Pierre-Emmanuel Brinette Wednesday, August 27, 2014 6

  7. HPSS HPSS-API SRB RFIO xrootd dcache IN2P3 HPSS Configuration (early 2009) • 25 M files / 6 Pb • + 2000 users • Main access protocol is RFIO • HPSS is used as tape backend for others storage systems: • xrootd: 1 M files /1 Pb • SRB used for files transfers • dcache 2 M files / 2 Pb for grid Local users only Local and remote users Grid users (SRM) Pierre-Emmanuel Brinette Wednesday, August 27, 2014 7

  8. IN2P3 HPSS Configuration (early 2009) • HPSS 5.1 • 42 disk movers • 2600 disk volumes • 28 tape movers • 66 drives • Gbits LAN Pierre-Emmanuel Brinette Wednesday, August 27, 2014 8

  9. Overview • Context of the migration • What’s new in HPSS 6.2 • IN2P3 HPSS Configuration • Migration Planning • Metadata Migration • Conclusion Pierre-Emmanuel Brinette Wednesday, August 27, 2014 9

  10. Migration Planning • Migration goal • Clean-up configuration • Remove EOL hardware • Decrease the number of volumes Migrate to HPSS 6.2 Migrate to HPSS 6.2 Migration to HPSS 6 Migrate to HPSS 6.2 Migrate to HPSS 6.2 Pierre-Emmanuel Brinette Wednesday, August 27, 2014 10

  11. Migration Planning New 10 Gb • Migration goal • Introduce new hardware • Use 10 Gbits for mover New New New Pierre-Emmanuel Brinette Wednesday, August 27, 2014 11

  12. Migration Planning • IBM Migration requirements • Migration procedure applied on the production server. • Metadata migration procedure assume a 1 core server and a 1 subsystem. • Empty Disk volumes. • UID/GID in DCE consistent with new auth/authz system (ie NIS). Pierre-Emmanuel Brinette Wednesday, August 27, 2014 12

  13. Migration Planning • IN2P3 CC constraints • Change the HPSS Core server and remove secondary server (EOL) • Use NIS for Auth/Authz • Modify metadata of 4 subsystems • Drastically reduce the number of Disk Volumes/hpss mover/servers • Introduce the new tape movers Pierre-Emmanuel Brinette Wednesday, August 27, 2014 13

  14. Migration Planning (Non linear time line) Dec 08 Feb 09 Apr 09 May 1, 09 May 15, 09 Jun 1, 09 Jun 5, 09 Migration case study with IBM Support HPSS 6.2 Test platform setup Writing migration procedure HPSS 6.2 metadata migration platform setup Clean-up production config Setup new production server Migration Pierre-Emmanuel Brinette Wednesday, August 27, 2014 14

  15. Migration Planning (Non linear time line) May 25 May 31 Jun 1 Jun 2 Jun 3 Jun 4 Jun 5 Reduce disk volume Stop HPSS access Purge disk Shutdown HPSS Export & Convert DB2 metadata Import in new server Configure new server HPSS mover installation RFIO Installation Tests Pierre-Emmanuel Brinette Wednesday, August 27, 2014 15

  16. Overview • Context of the migration • What’s new in HPSS 6.2 • IN2P3 HPSS Configuration • Migration Planning • Metadata Migration • Conclusion Pierre-Emmanuel Brinette Wednesday, August 27, 2014 16

  17. Metadata migration • HPSS 6.2 introduce new DB2 Tablespace containers scheme • Use a production metadata backup after HPSS shutdown • On a temporary HPSS systems: • Create exactly the same raw devices than the ones used on the production server • Restore the backup, run metadata conversion script, and export DB content • On the new production system: • Create the databases with HPSS install procedure (mkhpss) • Populate the databases with the DB content just exported Pierre-Emmanuel Brinette Wednesday, August 27, 2014 17

  18. Overview • Context of the migration • What’s new in HPSS 6.2 • IN2P3 HPSS Configuration • Migration Planning • Metadata Migration • Conclusion Pierre-Emmanuel Brinette Wednesday, August 27, 2014 18

  19. Conclusion • Migration issues • Even with a good preparation we have encountered (small) issues during the migration. • Details available on demand. • Benefits • Supported HPSS version. • Stability improved. • Configuration simplified. • Metadata queries now runs faster. • Gain lot of experience for the new staff ! Pierre-Emmanuel Brinette Wednesday, August 27, 2014 19

  20. Conclusion • After migration • Repack of remaining 9940 tapes • Remove of last PowderHorn library • Repack of T10k-A  T10kB (Big files SC) • Repack 9840  T10k-A Sport • Future • HPSS 7.1 migration in 2010 • 4th SL8500 with T10K-B • Add more disk (probably DDN) Pierre-Emmanuel Brinette Wednesday, August 27, 2014 20

  21. Thanks for your attention Pierre-Emmanuel Brinette Wednesday, August 27, 2014 21

More Related