“Resource Management on Keystone Devices" by Justin Sobota

“Resource Management on Keystone Devices" by Justin Sobota 1/16/13

Agenda • Why is Resource Management Needed on Keystone Devices? • Keystone I Resource Manager (RM) LLD • Keystone I RM Lessons Learned • Keystone II RM • Q & A • Demo

Why is Resource Management Needed on Keystone Devices? • Resource sharing problems introduced with release of multicore devices based on Keystone architecture • Resource contention was managed by LLDs on DSP-only multicore solutions • NySh • Resource contention not managed on (ARM + DSP) solutions • ARM (Linux kernel) doesn’t use LLDs • Resource contention between ARM & DSPs – running Linux, BIOS, and bare-metal • No easy way to sync resource usage between ARM and DSP cores • Combination of SYS/BIOS and Keystone IP initialization uncovered sharing problems at application/task level • RTSC allows for plugging of pre-main startup routines • Arbitration of IP hardware startup when integrating an existing post-main application with app that utilizes SYS/BIOS pre-main capabilities • OpenMP & QMSS • Keystone II architecture introduces devices with multiple ARMs, as well as DSPs • Resource sharing problems will only get worse if not solved in a robust manner

Keystone I Resource Manager (RM) LLD • Prevents DSP applications from stepping on resources taken by ARM Linux • Initialization and usage permissions for select LLD resources defined in user-defined resource table • QMSS, CPPI, and PA LLDs • Operates under hood of LLDs • Resource init and usage requests made between LLDs and RM directly • LLD operates normally if resource request “allowed” • LLD returns error if resource request “denied” • Application must take steps to handle • Permission tables located in shared memory • Restricts RM to DSP cores only • LLD resource storage still in hands of LLD • Full backwards compatibility with applications not utilizing RM Quick, “dirty” resource management solution for Keystone I devices

Keystone I RM Lessons Learned • Very tedious to add new resources since RM has static definitions in both code and data of resources it manages • Privileges for DSP cores only skims surface • Resource table with privileges defined at compile time • Privileges cannot be manipulated at runtime • System integrator must align Linux DTB resources with DSP RM resource table • Tedious and error-prone • Shared memory architecture provides no communication path to ARM • Not easily portable to new devices

Keystone II RM – Major Requirements • Remove static definitions of managed resources from RM • Easy addition of new resources • Enable management of resources at all levels within system software architecture • Core, task, application component (LLD) • Pre/post-main execution • Runtime modification of resource permissions • Automate reservation of resources taken by Linux kernel • Generic, processor-independent transport interface that allows RM instances to communicate regardless of device hardware architecture • Transport glue logic provided by application • Easy to port RM to new devices

Keystone II RM - Overview • Instance-based Client/Server Architecture: • Three instance hierarchy: • RM Server – Global management of resources and permission policies • RM Client Delegate (CD) • Offloads management of resource subsets from Server. • Provides singular data path to Server • RM Client – Provide resource services to system software elements • Resource services provided via instance service API • No restrictions on RM instance topology within system • RM Instances Communication Over Generic Transport Interface • Application must setup data paths between RM instances • Allows RM to run on any device architecture without modification to RM source • Resources Tracked by RM Defined in Global Resource List (GRL) • GRL captures all resources that will be tracked for a given device • Based on device spec and CSL • Facilitates automatic extraction of resources used by ARM Linux from Linux DTB • Policies Specify RM Instance Resource Privileges • Resource initialization, usage, and exclusive right privileges assigned to RM instances • Resource assignment to RM instances allows resource management at all software system levels • Runtime modification of policy privileges • APIs and Linux CLI (Planned) • Resources Stored within Balanced Search Tree Allocators • Reduce memory usage and resource lookup times • Allocators facilitated by NameServer

Keystone II RM - Overview ARM/DSP n ARM/DSP n+1 User Mode (ARM) Resource Policies Global Resource List (GRL) Linux DTB Memory Allocator QMSS Available resources are inverse of Linux DTB CPPI RM Server Instance RM CD Instance Allocation policies Resource Allocators PA QMSS CD Service Transaction Handler Resources Allocated from Server Etc CPPI CD Service Transaction Handler Service Port Transport API PA Service Port ARM  DSP Transport DSP  DSP Transport Transport-Specific Data Path Transport API Etc ARM  DSP Transport ARM/DSP n+2 DSP  DSP Transport ARM/DSP n+3 DSP  DSP Transport QMSS QMSS RM Client Instance Transport API RM Client Instance Transport API CPPI CPPI Client Service Transaction Handler Client Service Transaction Handler PA PA Service Port Service Port Mem Alloc Mem Alloc Etc Etc

Keystone II RM - Instances

Keystone II RM – Instance Topology Example 1 DSP Multicore Application IPC BIOS Linux User-Space LLD Etc LLD Etc LLD Etc RM Server RM Client Delegate RM Client ARM DSP 1 DSP 2

Keystone II RM – Instance Topology Example 2 DSP Multicore Application Linux User-Space LLD Etc IPC BIOS RM Server RM Client Delegate LLD LLD RM Client RM Client ARM DSP

Keystone II RM - Services • Services available from any RM instance • Instance name used as the basis for permission checks • Current services • Allocate (initialization, usage) • Allocate static resource • Allocate resource of unspecified base and alignment, length must be specified • Allocate resource based on existing NameServer name • Free • Free static resource • Free resource based on existing NameServer name • Map resource(s) to NameServer name • Get resource(s) tied to existing NameServer name • Unmap resource(s) from existing NameServer name • Non-blocking service requests directly return result • Blocking service requests return ID to system • Result of service returned at a later time via callback function provided with original service request • Service result ID will match ID returned to system at time of request

Keystone II RM – Transport Interface • RM instances communicate via generic transport API • Application provides transport path between RM instances • Transport paths registered with RM instances • Destination RM instance type specified • Transport path send/receive function pointers provided • Management of transports between RM instances handled by application • Facilitates ability of RM to run on any device topology • RM source code porting not required in order to run on new devices and architectures

Keystone II RM - Transport Example ARM DSP 2 RM Server RM Client 1 RM Transport APIs RM Transport APIs App Transport Routing Map App Transport Routing Map ARMDSP Transport DSPDSP Transport DSP 1 App Transport Routing Map RM Transport APIs RM Client Delegate RM Transport APIs RM Client 2

Keystone II RM - Global Resource List (GRL) • Specified in Device Tree Source (DTS) format • Open source, dual GPL/BSD-licensed LIBFDT used for parsing GRL • Input to Server on initialization • Server instantiates allocator for each resource specified in GRL • A GRL specification for a resource includes: • Resource name • Resource range (base + length) • Linux DTB alias path (if applicable) • Resource NameServer assignments (if applicable) • Permissions not specified in GRL

Keystone II RM - GRL Example /dts-v1/; / { device-name = "TCI6614"; /* Device Resource Definitions based on HW spec and CSL */ timers { resource-range = <0 16>; ns-assignment = “Core_0_Watchdog”, <0 1>; }; qmss { pdsps { resource-range = <0 2>; }; memory-regions { resource-range = <0 20>; linux-dtb-alias = "hwqueue@2a00000 regions region-12 id", <1 0>; }; accumulator-ch { resource-range = <0 48>; }; gp-queue { resource-range = <896 7296>; }; }; };

Keystone II RM - Policies • Specified in Device Tree Source (DTS) format • Open source, dual GPL/BSD-licensed LIBFDT used for parsing GRL • Assign resource permissions to RM instances • Available resource permissions: • Initialization (i) – Allocate resource for initialization • Usage (u) – Allocate resource for use • Exclusive (x) – Resource not shared, can be allocated to no more than one RM instance • Linux Shared (s) – Resource reserved by Linux kernel but can be shared with specified RM instances • Policies can be modified, created, and deleted runtime (Planned) • Policy APIs available to application • Linux CLI

Keystone II RM - Policy Example /dts-v1/; /* RM Server global policy */ / { /* All RM instances expected to be resource assignees within the policy. RM will fail at init if * it finds a resource assignee that is not in the user-instances list */ valid-instances = "RM_Server", "RM_Client_Delegate", "RM_Client"; /* RM will deny any resource requests for resources not defined in the policy. */ /* Format for assigning resources to specific RM instances */ qmss { gp-queue { assignments = <2000 1000>, "iu=(RM_Server RM_Client_Delegate RM_Client)", <3000 1>, "iux=(RM_Server) & iu=(RM_Client_Delegate RM_Client)"; <4000 1>, “s=(RM_Client_Delegate RM_Client)”; }; }; cppi { pass-rx-flow-id { assignments = <0 20>, "iux=(*)"; }; }; };

Keystone II RM - Name Synchronization Global Resource List (GRL) / { … myFirstResource { resource-range = <0 20> /* 20 myFirstResource’s in system */ }; mySecondResource { resource-range = <0 50>; /* Alias path must be path to value in Linux DTB */ linux-dtb-alias = “hw 2nd-res values”, <2 0 1>; }; }; RM Instance Policy RM Service Port from RM Instance “myRmInst” Resource Manager / { valid-instances = “myRmInst”, “myOtherRmInst”; … myFirstResource { /* All myFirstResource can be * used by anyone in system */ assignments = <0 20>, “iu=(*); }; mySecondResource { /* All mySecondResource is split * between the two valid RM * instances in system */ assignments = <0 25>, “iu=(myRmInst)”, <25 25>, “iu=(myOtherRmInst)”; }; }; - myFirstResource - mySecondResource Resource Names must be synchronized! char resName[] = “myFirstResource”; serviceInfo.resourceName = &resName[0]; serviceInfo.resourceIndex= 4; Services API (*rmService) (&serviceInfo); Linux DTB hw { 2nd-res { /* Kernel takes mySecondResource * in range 20 - 30 */ values = <20 10>; }; 3rd-res { … }; };

BACKUP

Keystone I Resource Manager (RM) LLD (cont’d) Core 0 Core 1 App App Queue/cppiChOpen/ … Queue/cppiChOpen/ … Rm_init QMSS LLD CPPI LLD PA LLD Rm_start QMSS LLD CPPI LLD PA LLD resource_table.h Init/usePermissionCheck Init/usePermissionCheck rmResourceTable{ {RES_ID, START, END}, { …, …, … }, … } RM RM Permission validation Permission validation Rm_init maps rmResourceTable to internal Permission Tables Shared Permissions …

Keystone II RM - Architecture Application Code Task IPC LLDs BIOS BIOS and LLD Internal RM Callout APIs Linux Kernel/XDC Runtime/BIOS Service API Resource Manager (RM) OS Abstraction Layer OSAL APIs Allocation Policy CLI (ARM Linux Only) OSAL Other OS Transport API DSP<->DSP Application Transport ARM<->DSP Application Transport ARM<->ARM Application Transport

Keystone II RM - Architecture • Key RM Interfaces • Service API • Service ports opened from the RM service API • Service ports provide resource services to system entities • Services will be labeled as originated from the RM instance from which the service port originated • Transport API • Generic transport interface over which RM communicates between instances • Application registers instances with one another over transport API • Application provide transport glue function API to RM instances at registration • RM calls registered transport glue functions when sending/receiving RM packets between RM instances

RM Server • One RM server for a given exclusive set of system resources • Maintains global view of allocator, policy, and NameServer states • Can satisfy pre-main resource service requests • OpenMP, BIOS, IPC, etc. • Maintains the global policy that defines the resource permissions for all RM instances • Policy synchronization with policies stored on CDs not yet implemented • Maintains all resource allocators • Who has which resources • Maintains RM NameServer

RM Client Delegate (CD) • Acts as a proxy for RM Server to RM Clients • Quick response mechanism to Clients for certain services (Planned) • Can be used as single path to Server when transport path to server is longer than transport path to Clients • Instead of all Clients connecting to Server over a high cycle transport path they can connect to CD • CD can allocate resources that have been provided it by the Server (Planned) • Will request new block of resources from Server when it runs out of resources to allocate to Client service ports and service ports hanging off itself (Planned) • Can satisfy pre-main allocation requests • Pre-main policy will be relayed to Server after initialization and transport registration (Planned) • Currently, the pre-main policy must be reflected in the global resource policy provided to the Server at initialization • Due to policy synchronization feature not being implemented

RM Client • “Dumb” RM instance • Sole purpose is to provide gateway to RM services for system components • Minimum permissible policy unit • All service requests offloaded to the registered RM CD or RM Server • No restriction on where Clients can exist

Keystone II RM – Instance Topologies • Minimal requirements for RM instances within a system • If there are multiple RM Servers exist within a system the resources they track must be mutually exclusive • RM CDs must be connected to at most one RM Server • RM Clients must be connected to at most one RM Server or RM CD • No restrictions on where RM instances can be created • Service ports can be opened from any RM instance type

GRL and Policy Compilation Device-specific Global Resource List (GRL) and any Application Policies DTC Executed on Host (Linux) Converted Binary GRL Converted Binary Application Policy Device Tree Compiler (DTC) Human Read-able Device Tree Source (DTS) File Compiled GRL Device Tree Blob (DTB) Compiled Policy Device Tree Blob (DTB) Application Code RM Library (compiled and linked with libfdt) Compile Full Application Rm_init(… *dtbPointer …) Data from GRL DTB used to initialize resource allocators Global Resource DTB libfdt APIs Full Application Binary RM Server Init Policy data is validated at init. Extracted later to check permissions Policy DTB

Porting RM to New Devices • Required to port RM to new device • Update GRL to exclude/include device-specific resources that are to be tracked by RM • Update Policy resources permissions to match new GRL resources • Update GRL with any new resource path alias to reserved resources in Linux DTB • No need to modify RM source • Application source will need to change

To Do • Fault management • Policy editing Linux CLI • Remove RM NameServer • Add NameServer interface so that an existing NameServer can be registered with RM • Resource and policy front-loading on RM CD

Linux DTB Resource Extraction • Cannot rely on Linux DTB to match GRL resource specification exactly and vice versa • Alias path is clean way of extracted Linux reserved resources from Linux DTB • Alias path specifies node path down to node containing value that should be reserved by RM for Linux kernel • Resources specified as reserved by the Linux kernel are not able to be used by any RM instance unless otherwise specified in the Policy

Keystone II RM - Name Synchronization • Key to abstraction of static resource definitions away from RM • Global Resource List (GRL) defines each resource that RM will track and how many of that resource exist • Policies define which RM instances are allowed to use the resources defined in the GRL and how the instances can use them • Utilize Linux Device Tree framework specification, ePAPR, to define easy to read, modify, and parse GRL and policy specifications • Facilitates extraction of resources used by Linux kernel from Linux DTB file • LIBFDT library included by RM to read, parse, and modify the GRL and Policy DTBs • Dual GPL/BSD licensed

Allocators • RM Allocators implemented using a open-source binary balance search tree algorithm • BSD-licensed • Resource allocator nodes specify a resource base and length as well as a list of RM instances the resource is allocated to • Wrapper written around the tree algorithm that expands and contracts the allocator tree nodes as resources are allocated and freed • Saves time when searching for resources to allocate • Saves space when RM can be potentially managing thousands to millions of resources

Allocators - Example 1 • Assumptions: • System has 1024 GP Queues • Policy specifies blocks of 256 2 -Allocate queues 335 – 345 -Allocate queue 767 qmss-gp-queues 1024 qmss-gp-queues 1024 • Not exact to tree algorthm 0-255 Free 256-511 Parent 512-767 Parent 768-1023 Free 0-255 Free 256-511 Free 512-767 Free 768-1023 Free 256-334 Free 335-345 Reserved 346-511 Free 512-766 Free 767 Reserved 3 -Free queues 335 – 345 -Allocate queues 384-400 qmss-gp-queues 1024 0-255 Free 256-511 Parent 512-767 Parent 768-1023 Free 256-345 Free 346-511 Parent 512-766 Free 767 Reserved 346-383 Free 384-400 Reserved 401-511 Free

RM NameServer • Internal, simple NameServer • Not intended for long term use • Allows association of resources with a user-specified name • System entities can use service ports to map and unmap resources to names, and allocate/free resources by name • Instance permissions will not be checked against the name given the resource but the resource itself • Planning for future revisions of RM to have external NameServer where RM would not have knowledge of names • Aware of growing of NameServers in SDK offering • Plan to remove RM NameServer and replace with API allowing for hookup to an external, existing NameServer

Pre-Main Resource Allocation • Available on Servers and CDs • Startup policy checked on pre-main allocations • After initialization startup policy will be synced with Server and along with any allocations • Startup policy must be in sync with global policy given to server at init time

“Resource Management on Keystone Devices" by Justin Sobota