Content Management Systems

Professional Content Management Systems6th & 7th Lecture: System Architecture & Service Oriented Principles in CMSDr. Andreas MautheSCC – Lancaster University

Content Management Systems • Elements • Hardware infrastructure • Servers • Storage infrastructure • Communications infrastructure • Software architecture • Modules for essence & data management • Added & auxiliary services • Unifying system architecture • Requirements on the CMS Architecture • Support workflows • Specific workflows in an integrated system • Consider features of content components • Essence • Various formats • Various copies &versions • Metadata • Integration of legacy & third-party systems • Facilitate emerging fully digital tape less production • Characteristics • Flexible • Adaptable

CMS Architectural Components

CMS Processes and Data Flow: Content Ingest • Main Modules • EssenceManagement • Storage, provision & manipulation of essence • Data Management • Support of documentation, search and retrieval • Advanced Services • Application Elements

Essence Management Features • Main Essence Management Components • Storage management • Automatic migration of essence items within the storage hierarchy • Transcoding • File format and encoding format conversions, • Delivery services • Such as file transfer and streaming services, • Ingest and provision of essence • A database • Technical metadata • Formats, bit rates, colour depth, etc.) • Management information • Locations, file sizes, etc. • Managed Essence Objects • Original tape copy • High quality file copy • EDL browse copy • Content browse copy • Keyframes

Data Management Features • Main Data Management Components • Database • DBMS • Structured files • Search engines • Including fulltext search engines • Supported Indexing Workflows • Real-time annotation • Application area • News feeds • Live sports events • Feature production • Limited set of metadata, • Identification of material and to allow successful • Retrieval during recording • In-depth cataloguing • Longer-than-real-time • Supported by • Thesauri • Legal lists, • Complex user interfaces for detailed annotation of content. • Automatic feature extraction • Automatic classification of content

Software Design Principles • Basics • Concepts to deal with • Heterogeniety • Complexity • Modularisation as key design principle • Object-oriented design • Componentware in application development • Scalability by Distribution • Increase number of resources by adding new instances • Scalability requirements • Each component of the architecture must be able to run on a dedicated server platform. • Communication must be fully network based • WAN support • Pyhsical distribution in a wider area • Across cities and countries • Mulitple instances should be able to run in paralle • Hardware neutral • It should be possible to launch components on any server platform within the CMS hardware architecture. • CMS resilient against unavailability or failure of other CMS components • Reconnect strategies • Recovery procedures

Switch n2 Resources n3 Resources n4 Resources n5 Resources n6 Resources Service Group • A Server runs on a dedicated server platform • A Server offers ni resources • A Server is restart capable n1 Resources • m Servers running on m server platforms join a Services Group • A Services Group looks to the outside world like a single Service with • niresources • A transactionsecure job list • An event and status log functionality • A processing time estimation • A resource reservation • The internal organisation of these central tasks is managed by the Services Group • Each Server can register with and un-register from the Services Group • One Server launched the coordinator of the group (the Manager) • In case of failure of the Manager another Server automatically launches a new Manager

Broker-Manager-Concept

Broker-Manager Relationship • Basic Concept • To structure • Multiple instances of specific server components for optimum performance and availability on single server level • Multiple storage clusters for maximum flexibility and scalability at a single site of an organisation • To integrate multiple sites of an organisation into a single unified system • To interface multiple organisations allowing them to share selected items seamlessly • Characteristics of Broker • Distribution and collection node • E.g.Database Broker • Accepting queries from a client • Federating these queries to multiple system and existing databases • Collecting the results from these databases • Passing the consolidated result back to the client. • Act as a Factory • E. g. Stream Broker • Accepting requests from a client to stream essence to the client • Selecting a suitable stream server from a pool of servers • Creating an instance of a control interface on this server • Passing the interface back to the client • Control passes to the client • No further broker involvement

CMS Software Architecture Principles • Idea • Platform for the integration of services and applications • Comparable to OS • Complex, federated architecture • Open framework • Allows integration of addition (third-party) components • Main Architectural Parts • Core Plane • Management & administration of content • Control of related tools & devices • Services • Enhancing functionality • Workflow support • Applications • User applications and GUI • Vertical Services • Job Management • System Management • System Administration

Component Software Architecture Application Plane Session Layer Session Manager Job Mgmt SystemAdministration Application Layer System Mgmt Plan&Draft Logging Browsing Export Cataloging Import Admini-stration Configuration Service Remote Installation Service User Management Accounting&Licensing Service Messaging Service Resource Broker Ingest Retrieval Editing Clearance Naming Service Event Log Service Process Monitor Commis-sioning Workflow Engine Task Management Service Transaction Service Services Plane Ingest Service IndexingService ImportService Water- marking Service Web Service Analysis Service Export Service Editing Service Conversion Service Ordering Service Core Plane Distribution Layer Essence Broker DeviceBroker Data Broker Abstraction Layer EssenceManager EssenceManager EssenceManager DeviceManager DataManager DataManager DataManager Foundation Layer Archive Servers Streaming Servers DeviceServer Legacy System DBMS Full Text Search ImageSimilaritySearch AudioSimilaritySearch Cache Servers Transfer Servers

CMS Design Principles • Component Based Design • Well-specified components • Described by • Functional Design Specification (FDS) • Interface Design Specification (IDS) • Functionally complete modules • Core and service modules can build on each other • Applications provide user front-end • Operative Characteristics • Job oriented processing • Each task can be broken down into a set of jobs • Jobs are atomic units processed by one module • Job management to co-ordinate & monitor task execution • Splits tasks into jobs • Schedules jobs • Monitors jobs • Communication • Open communication standards • Interprocess communication • Standard schemes • CORBA, D-COM • SOAP

Essence Management (EM) • Tasks • Storing, managing and providing essence objects • Individual tasks are: • Access to mass storage systems by addressing the Archive Management Server • File migration between online and near-online storage by controlling the Archive Transfer Servers • The creation of multiple copies of essence files in the archive • The contents and integrity of the online storage systems by addressing the Cache Servers • The delivery of content to clients by linking them to the Streaming Servers • Can be third-party streaming server • Introducing essence from or delivering essence to remote storage systems by accessing the Transfer Servers • Recording essence • Read-while-recording • Essence Management Components • Archive Manager • Acrhive Transfer Manager • Archive Transfer Servers • Cache Servers • Streaming & Transfer Manager • Streaming Servers • Transfer Servers

Essence Management Structure

EM Archive Manager • Tasks • The Brain of the Essence Management • Keeps track of storage units in mass storage system • Tapes, tape pools • Disk pools • Archival and restoral of files • To other CMS components • Restoration of files from near-online storage to server • Library control • Movement between slots and tape drives • Ensure storage media integrity • Operational Considerations • Priorities tasks • Interactive tasks vs. automatic background tasks • Distribute jobs considering • Load • Location • Mass Storage Alternatives • CD Libraries • Mixed media libraries • Studio tape libraries

Storage Media Integrity Check • Procedure • Mounting tapes written by one tape drive in another tape drive and re-reading the data  to eliminate the possibility of write errors due to misalignment of heads or other technical defects in the writing unit. • Rewinding and re-tensioning tapes that have not been accessed for a given amount of time  to avoid sticking of tapes. • Reading tapes in regular intervals  to check the bit error rate. • Copying tapes that have a bit error rate higher than a given safe bit error rate to new tapes • Copying tapes that have reached end of life to new tapes. • A tape has reached end of life when its l • Overal age due to its time spent inside of a controlled environment • Time spent outside of a controlled environment is larger than a given safe age.

EM Archive Transfer Manager • Task • Archiving and restoring of files • Request from Archive Management Server • Copying files between online and near-onlinestorage • Operational Features • Archive Management Services has • Transaction secure queue • Priority based queuing • Archive Transfer Manager • Processes requests accordign to • Priorities • FiFO • Manages multiple Archive Transfer Servers

EM Archive Transfer Server • Task • Handles the migration between near-online and online • Near-online • Tape drive • As part of tape library • Disk pool • Online • Hard disk system • Network Attached Storage (NAS) • Storage Area Network (SAN) •  Full SAN environment may make Archive Transfer Manager obsolete • Remote storage systems • Accessible via FTP • Integration of optical media • Transfer Policy • Optimised batch transfer • User triggered transfer • Media managers, archivists, etc. • Priority based queuing

EM Cache Server • Task • Manages online storage systems • Essence storage for online access • Provide information about essence online • Identification of essence & content object it belongs to • Encoding format • Other technical metadata • Keep amount of essence between high and low “watermarks“ • Caching Policy • Migration strategy • Keep the most frequently accessed material online • User definable rules • Deleting essence that has not been touched for the longest time • Deleting essence that occupies a large amount of disk space • Minimum time in cache • Mark essence undeletable • Locking of essence if it is expected to be used in future • Migration requirements • Essence must have been successfully transferred to near-online • Essence must not be locked by an application or a service

EM Streaming & Transfer Manager • Streaming Manager • Task • Manages & co-ordinates concurrent requests for streaming • Manages groups of Streaming Server • Selection of most suitable Streaming Server for a request • Connection establishment between Client and Streaming Server • Operational Features • Selection of Streaming Server according to • Load situation • Physical proximity • Location • Connection characteristics (i.e. QoS) • Logical proximity • Content is already served to other clients • Transfer Manager • Task • Manages & co-ordinates requests for file transfer • Operational Features • Similar to Streaming Manager

EM Streaming Server • Task • Negotiating and establishing connection between client and streaming product (i.e. essence on online storage) • Streaming essence to clients • From online storage • Operational Features • Wrapper around • Existing video & audio server • Keyframe servers • As fast as possible, not necessarily continuous • Advantage: no header metadata with every frame • Triggers staging from near-online if required • Multiple Streaming Servers can access the same online copy • Streaming • Synchronous • Playout • Ensured QoS (e.g. Bandwidth less or equal maximum allocable bandwidth) • Asynchronous • Download

EM Transfer Server • Task • File exchange with remote content server • Upload & download • Encoders, decoders, production systems, media juke boxes, • Cart machines, playout server, NLE • Physical file transfer • Characteristics • Co-located with Cache Server or on proxy • Provide control API for content exchange • Communication support for essence exchange • Including standard protocols such as FTP • Proprietary mechanisms

CSM and SOA • Why the SOA Paradigm for CMS • Capabilities are provided by Services • Business logic is provided by Orchestration • Communication is accomplished through open schemas and protocols

Reminder: SOA in iT • Services • Repeatably and reliably provide capabilities that can be used to • Process tasks in business processes (Workflow Services) • Mangage and operate the platform (Infrastructure Services) • Provide information and/or transform business data from one consistent state into another • Are loosely coupled, stateless and autonomous • Offer well-defined, technology & platform independent, standard interfaces • Publish functions and data but hide their implementation • Are offered by a Provider • Orchestration • Manages overarching business processes • Aggregates higher level capabilities by invoking services • Capabilities are in the Services, intelligence is in the orchestration

SOA Basics “An SOA is an architectural style that is reflected in an enterprise-scale IT architecture. It enables access to distributed resources, on demand. An SOA consists of a set of business-aligned IT services that collectively fulfil an organization’s business processes and goals. These services can be choreographed into composite applications and can be invoked through open protocols.” (Asarjani, A.: The SOA Best Practices Blog) 3:Compare request to published offers Manages published Service descriptions Registry 2:Search Services 4:Indicate suitable Provider 1:Publish Service 5:Request Service Requestor Provider 6:Service result Requires a certain Service Wants to publisha Service

SOA in Media vs. SOA in IT • These aspects of SOA in Media are also recognized by the AMWA Application Specification „Framework for Interoperable Media Services“ (FIMS): • Asynchronous communication is required for long lasting activities • Long lasting activites benefit from Job Management • Essence and Metadata may travel along different paths • Essence transfers and delivery require various physical resources • Media SLAs (M-SLA) are required for guaranteed Media Delivery • Guaranteed essence deliveries benefit from Resource Management • Security of media adds requirements to conventional IT security

Example: Video File Import Workflow Import request (Workflow Engine) Receive Request User (Workflow Engine) Request Job Card (Job Card Provider) Create Job Card Database Template (Workflow Engine) Send metadata (Data Management) Create/update record Database Metadata Inbox (Workflow Engine) Import file (Essence Management) File transfer High Res Archive High Res Archive High Res (Workflow Engine) Create proxy (Transcoding) Create proxy Archive Low Res Archive Low Res (Workflow Engine) Create key frames (Video Analysis) Extract key frames Archive Frames (Workflow Engine) Archive Job Card (Job Card Archive) Create record Database Job Card

Literatur • P. Thomas, A. Mauthe: SOA für Media-Asset-Managementsysteme, in FKT 8-9/2008, 62. Jahrgang, Seite 490-494, September 2008 • Service Oriented Architectures For Broadcast Media Asset Management • P. Thomas, A. Mauthe, in Proceedings of NAB Broadcast Engineering Conference, Las Vegas, USA, April 2007

Content Management Systems