Oracle Database Appliance - Technical deep dive

1. Oracle Database Appliance - Technical deep dive

2. Agenda • Hardware internals • Download appliance software kit • Get Core config file • Configuration and setup • Oracle 11g R2 • Backup strategies

3. Oracle Database Appliance – Hardware internals

4. Chassis • Sun Fire X4370 M2 • 4U Redundant Storage Server • 2 Server Nodes (SN) • Two internal boot disks per SN • 24 3.5” dual ported SAS/SATA/SSD disk slots • 20 x 600GB 15K RPM SAS(4 TB Usable for database files) • 4x 73GB STEC GEN3 SSD(for redo logs) • 2 Hot-swap redundant power supplies • Redundant 5V and 12V disk backplane power • Independent PWR, locate buttons and status per SN FRONT VIEW REAR VIEW

5. Chassis • Independent power buffer chips and individually wired between PSUs and SNs. • Chassis control and status is independently accessible on each Server Node • No centralized chassis control manager • Hot Swap SN modules, Fans, Disks, Power Supplies • All CRUs are front and rear accessible FRONT VIEW REAR VIEW

6. Chassis - Front View

7. Chassis - Rear View SN1 SN0

8. Server Node • Leveraged from Sun Fire X4170/X4270 M2 server platforms • 2 x Intel Xeon Processor X5675 (6C, 3.06 GHz, 95W) per Server Node • 2-12 CPU cores per SN enabled on demand • 6 x DDR3-1333 DIMMs per socket total of 96 GB of memory per SN • 12x 8GB DDR3 low-voltage DIMMs per SN • 1 Internal low profile 8-lane PCI-E GEN-2 HBA per SN • LSI SAS GEN2 Erie HBA • 3 x low profile 8-lane PCI-E GEN-2 Slots via PCI riser per SN • Slot 2: LSI SAS GEN2 Erie HBA • Slot 1: Intel quad port 1GbE Northstar • Slot 0: Intel dual port 10GbE Niantic

9. Server Node • 2 x 1-GbE RJ45 connectors for Host per Server Node • 2 x 1-GbE ports within chassis for SN to SN connectivity • 2 x rear accessed hot-plug SATA 2.5” drive per SN • Boot disks • 2 x Rear, 1 internal USB connector per SN • AST2100 Service Processor per SN • 1 SP Serial, 1 SP Network, 1 SP VGA per SN • 2 x Hot-swap 80MM counter-rotating fans per SN

10. CPU Cores on Demand • CPU cores on demand enables as few as 4 cores or as many as 24 cores • System ships with 12 cores enabled per server—no need to do anything if running at full capacity • To configure for subcapacity (see also “get Core config file” slides): • Log onto MyOracleSupport • Declare configuration for servers • Download encrypted key to reconfigure servers • Run CLI to reconfigure servers (as specifed in key) in BIOS (reboot) • Once configured for sub-capacity, core count can only increase in subsequent reconfigurations • Customer responsible to ensure proper number of licenses owned

11. OS Boot Disk Internal HBA Internal USB CPU1 CPU0 PCI Riser Server Node View

12. SAS, Chassis, FIM, +5V disk, PSU Signals Server Node View

13. DB Appliance CRU/FRU Assemblies

14. Component Identification

15. Chassis Cooling

16. Expandability • Technically iSCSI or NFS expansion via 1 or 10-GbE network connectivity is possible, but restrictions apply • FC-AL expansion not supported • Infiniband not supported • SAS expansion not supported

17. Hardware Benefits Review • Internal chassis wiring • Less potential of accidentally pulled cables • Redundant and Hot-serviceable components • Server Nodes • Fan Assembly • Power • Two redundant PSU • Individually power cabled from PSU to SN • Independent buffer chip to each SN • HDD and SSD • Triple-mirrored • Two internal networking to fuse cache across the clustered SNs • Two dual-SAS controllers per SN

18. Hardware Benefits Review • Improved Performance • SSDs used for redo logs • Networking throughput • CPU on demand • Allows for growth by enabling more CPU cores to meet customer demands

19. Oracle Database Appliance – Download appliance software kit

20. Go to My Oracle Support http://support.oracle.com 1 …and sign-in if needed

21. Select Oracle Database Appliance

22. Click download…

23. Click on appliance software…

24. …and save the file

25. Software kit download recommendations • Downloading the kit beforehand • This will reduce installation time at start of deployment • Put the kit on a local server or USB stick • Reasons to do so are: • Security • Speed (local copy from download server to ODA using high-speed LAN) • When USB stick is used, no network connectivity needed to copy the software kit to the appliance

26. Oracle Database Appliance – Get Core config file

27. My Oracle Support User Interface Overview for Generating Core Configuration Key • Only Customer User Admin of a Customer Support Identifier (CSI) is allowed to generate a key • Any one with an access to CSI and view “asset privilege” can view and download the key • Customer needs to perform following few simple steps to generate/download a key: • Add Support Identifier associated with ODA assets in their My Oracle Support profile • Specify # of CPU cores per server • Copy key and paste it into a file on the ODA server • Customer can view/download previously generated key anytime • My Oracle Support will allow user to re-generate a key ONLY for higher # of cores than last generated key • Oracle Store is going to store the keys and minimum audit history associated with the key generation activities.

28. MOS Asset Page: Manage Key

29. Manage Core Configuration Key Pop-up

30. Generating a Core Configuration Key

31. Copy-n-Paste the Key on the ODA server

32. Core Key Information Detail • Customer proposed core count • Customer Support Identifier # • Customer name (This is the customer name as in “Oracle Corp”) and not the Data Base Account name

33. Core Key Important Notes • All the Information Details are mandatory • Cores has to be one of the following numbers (2,4,8,12,16,20,24) • The program does not validate the cores. • Manually ensure that the core count is one of the numbers mentioned above • The generated key cannot be used to find the serial# or the cores so it is recommended that the out file attribute be some combination of serial# and core count.

34. Applying Core Key • Customer should not attempt editing the generated key using any tools • Customer should move the key to the Appliance using scp in binary mode. • Open a Terminal session or xterm and Login as root • Execute the following command as root #/opt/oracle/oak/bin/oakcli apply core_config_key /tmp/core08_1018FM005

35. Core Key Example

36. Verifiying the Core Configuration File • $oakcli show core_config_keyOptional core_config_key is not applied on this machine yet ! • # oakcli show core_config_key Host's serialnumber = 1132FMW003Configured Cores = 20 • cat /proc/cpuinfo | grep -i processor |wc –l 10

37. Oracle Database Appliance – Configuration and Setup

38. Factory Customer Site Deployment Overview Build System Image Firmware Install OS Provide Basic info to Configurator Connect System to Network & Power ON Download SW from MOS Deploy Grid, DB with best Practices Initiate HealthCheck Fully Configured Database

39. Oracle Appliance Manager Deployment Steps • Download standalone DB Appliance Manager configurator from MOS • Fill the configuration parameters and generate the configuration parameter file. • Do the first boot of DB Appliance server and configure the initial network – oakcli configure firstnet • Get core_key from MOS and apply core_key* • scp the user generated parameter file and end user clone zip file, on the first node ** • Unpack the user image – oakcli unpack –package <file_name> • Copy the user generated parameter file – oakcli copy –conf <params.file>** • Start the deployment by using the parameter file – oakcli deploy • Get some coffee and come back in 2hrs * optional ** oakcli will place them at the proper locations

40. Oracle Appliance Manager Planning Required for typical type deployment • Cluster Name • Region, Timezone • Domain • DNS Servers • Client Access Public Network Hostnames and its Ips • Client Access Public Network netmask • VIP Name (auto populated with –vip) and its Ips • Scan Name (auto populated with –scan) and two scan Ips • Database Type, Name, Class and Language

41. Oracle Appliance Manager PlanningOptional for custom type deployment • NTP Servers • ILOM network info • Additional network info • ACFS • Backup type, database block size, and characterset • ASR proxy and MOS username, password

42. Oracle Appliance ManagerPredefined Values • Private ip address: 192.168.16.24, 192.168.16.25 • HA ip address: 192.168.17.24, 192.168.17.25 • NIC names – eth0, eth1, bond0, bond1, bond2, xbond0 • Grid user – grid, oracle user – oracle • Grid home - /u01/app/11.2.0/grid • Grid base - /u01/app/grid • Oracle home - /u01/app/oracle/product/11.2.0/dbhome_1 • Oracle base - /u01/app/oracle • Inventory location - /u01/app/oraInventory • Scan port number 1521 • Diskgroup name - +DATA, +RECO, +REDO • Data disk name - HDD_Ex_Sxx_<wwid>p1 (S00-S19) • Reco disk name – HDD_Ex_Sxx_<wwid>p2 (S00-S19) • Redo disk name – SSD_Ex_Sxx_<wwid>p1 (S20-S23) • Oracle GI and RDBMS version – initial release is 11.2.0.2+psu3+patches. 12639177 (ASM) 12914151(MLR on top of GI PSU3) 12419331 (DB PSU3)

43. Network port bonding

44. Initial Hardware Setup • Machines arrive with factory pre-loaded 0923.1 iso, that includes OEL 5.5 and oak 0923.1-1 rpm. • Connect the two power cords • Connect the network cables • Connect the monitor, keyboard, mouse or serial console • Wait for Service Processor lights from blinking to solid • Power on the host and wait for host lights from blinking to solid • When rebooting is done, login at the console or via ILOM • If you prefer GUI mode, type ‘startx’ • * Alternately, customer can manually setup ilom network (covered later)

45. About oakcli • Command line interface to manage Oracle Database Appliance. • Needs to be run as root. • Invocations of CLI is logged to /opt/oracle/oak/log/<hostname>/client/oakcli.log oakcliUsage: oakcli <command> <object> [<options>]commands: show|locate|apply|deploy|update|validate|manage|unpack|copy|configure objects : disk|diskgroup|expander|controller|storage|core_config_key|version|firstnet For complete usage, use:oakcli –hFor detailed help on each command and object and its options use: • oakcli <command> -h oakcli <command> <object> -h

46. Oracle Appliance Manager Configuring first network • This to be done at remote console redirect (ILOM) • No network identity of the server when shipped from factory. • First network can be configured for ease of use. Only need to be done on first node # /opt/oracle/oak/bin/oakcli configure firstnet Select the interface to configure network on [bond0 bond1 bond2 xbond0]:bond0 Configure DHCP on bond0?(yes/no):no INFO: Static configuration selected Enter the IP address to configure:10.2.3.24 Enter the netmask address to configure:255.255.255.0 Enter the gateway address to configure:10.2.3.1

47. Oracle Appliance Manager Unpack end-user bundle and Copy Core keyr • Download the end-user bundle file from MOS • Download the core key from MOS • Transfer the core key file, if created earlier • Unpack the end-user bundle on first node • Copy the core key file Assuming the core_key_file is applied – covered in previous session # /opt/oracle/oak/bin/oakcli apply core_config_key <core_key_file> # /opt/oracle/oak/bin/oakcli unpack -package /tmp/grid112.zip # /opt/oracle/oak/bin/oakcli unpack -package /tmp/grid112.zip # /opt/oracle/oak/bin/oakcli copy –conf /tmp/<params.file>

48. Oracle Appliance Manager • GUI-based enquiry collects configuration information • Two modes • Configurator • Deployment • oakcli deploy • oakcli deploy config • java –jar onecommand.jar config

49. Oracle Appliance Manager Deployment overview • oakcli deploy runs the command in linear workflow order. • It is designed to be simple so that it can be used by users with basic experience with Grid Infrastructure and RAC • oakcli deploy does not provide complete control to the users during install but does standardized install that works for majority of the scenarios

50. Oracle Appliance Manager Config Type • Two types • Typical • Custom, including ntp, bond1, bond2, xbond1, ilom, asr setup on 1st node, and acfs • Can load the pre-created onecommand.params file at /opt/oracle/oak/onecmd