1 / 20

Audio Visual archiving : User requirements and challenges on infrastructure (AVATAR-m view)

Audio Visual archiving : User requirements and challenges on infrastructure (AVATAR-m view). Rajitha Weerakkody 22 Nov 2010. Outline. Audio visual archiving challenges Obsolescence Scalability Degradation Digitisation Avatar-m approaches Storage Management

aggie
Download Presentation

Audio Visual archiving : User requirements and challenges on infrastructure (AVATAR-m view)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Audio Visual archiving: User requirements and challenges on infrastructure (AVATAR-m view) Rajitha Weerakkody 22 Nov 2010

  2. Outline • Audio visual archiving challenges • Obsolescence • Scalability • Degradation • Digitisation • Avatar-m approaches • Storage Management • Scalable storage technologies • Video coding for graceful degradation

  3. The challenge: in perspective.. • BBC Archive has 650K hrs of video and 350K hrs of audio • (Estimated 100M hrs of film, video, audio in Europe, 200M worldwide!!) • Stored in various media types (2” tapes, 1” tapes, Betacam, Digibeta, D3, LTO, VCD, DVD, disk storage,……) • In both analog and digital …

  4. Obsolescence is a significant factor • Formats, media, .. • Long lasting media is not a solution • If no equipment to playback.. • New technologies are emerging • Migration is a must • Migration cycles getting shorter (4 years!!)

  5. They keep degrading.. • From physical degradation of storage media – to digital data corruptions..

  6. But we want to keep them in perpetuity!!

  7. Digitisation efforts • Digitisation is theway for long term (perpetual) preservation • Not all collections will ever be digitised – choices will have to be made • BBC currently runs a mass digitisation project for D3 tapes • Uncompressed video and audio in MXF files • Stored in LTO3 tapes-on-shelves • Generate up to 2 peta bytes per year!! • “Preservation factory” • File-based production and Archive workflows • Born-digital material also need to be preserved

  8. To recap, there are challenges around: • Digital storage system scalability and management • Constant migration cycles • Content degradation • Supporting file-based production workflows • Access to the archives • With HD material coming in, and SHV (16xHD) in the horizon, the pressure on the infrastructure is more ..

  9. What is AVATAR-m project.. • British government (Technology Strategy Board) funded industrial and academic collaboration • Completion November 2010 • Key research areas • Intelligent storage Management • Scalable storage technologies • Video coding for graceful degradation

  10. Intelligent Storage Management • Archive storage planning and management • Tools for planning for the archive growth, obsolescence and automating migration • Avatar archive storage manager • Some of the key features: • Centralised management of heterogeneous storage systems • Intelligent data handling, integrity checking and recovery with data chunking • Policy based management • Open file system interfaces • Seamless migration of data

  11. Avatar aggregated storage

  12. Scalable Storage Technology • Why digital storage doesn’t scale infinitely? • Interconnection fabric limitations – particularly with distributed storage technologies • I/O request handling • Indexing, metadata servers and databases • What we did.. • Detailed investigation of the state-of-the art, and various commercial products • Produce scalable architectures, with prototypes • Key recommendations • Ring architectures with high speed interconnectivity of nodes • Hierarchical grouping of I/O requests • Networked metadata server clusters • Parity declustering techniques to supersede RAID

  13. Archive Coding • Rationale: • AV files are subjected to data corruptions when retained for long periods in digital storage systems • Encoded video streams are made of a number of distinct components of varying “significance” (e.g. headers, various coding parameter blocks, sub band coefficient data etc.) • Thus the sensitivity of video files to data corruptions depends on the location of errors on the bit stream. E.g.: • Could be even worse! Files could be unreadable. Errors on a DC sub band data (Dirac Pro coding) Errors on a high freq. sub band data (Dirac Pro coding)

  14. Archive Coding • Proposal: • Significance classification for the bit stream elements • E.g.: • “Graded” protection based on Significance index • Split the video file into multiple sub streams • Use tiered storage for variable protection

  15. Archive Coding • Encoder process:

  16. Archive Coding • Retrieval process: • Combine multiple sub streams • Exploit redundancies to recover from data corruptions

  17. Archive Coding • Benefits: • Enables graceful degradation in the face of storage system errors • Significantly less bit rate overhead (1~10%) compared to keeping multiple copies (>100%) of the video files • Reduced total cost of storage • Enhanced value for the stored video content

  18. Demonstration • Demo • Dirac Pro coding • 10:1 compression • BER: 1E-6 • Error type: random bit errors

  19. Conclusions • Digital preservation of AV archives is a challenging task • Creates significant stress on the digital infrastructures • AVATAR-m project addressed some of these challenges • Intelligent storage management • Ultra scale storage technology • Video encoding for graceful degradation.

  20. Questions ? www.Avatar-m.org.uk Rajitha.Weerakkody@bbc.co.uk

More Related