220 likes | 425 Views
Dolby Australia Internship Programme. Introduction. Dolby and the new Internship Programme Learn about building production software Share my experiences working as a software developer Please interrupt with questions. Dolby Voice. New initiative targeting multi-party audio communication
E N D
Dolby Australia Internship Programme CONFIDENTIAL INFORMATION
Introduction • Dolby and the new Internship Programme • Learn about building production software • Share my experiences working as a software developer • Please interrupt with questions CONFIDENTIAL INFORMATION
Dolby Voice • New initiative targeting multi-party audio communication • Over $4.5B spent annually on audio conferencing, with the user experiencing poor to extremely poor quality • We have the right people, technology and audio knowledge to make it better CONFIDENTIAL INFORMATION
History: Dense Immersive Communications Environment • We began as a CRC research project at the University of Wollongong • Funded by Telstra Research Labs and Motorola • Project was to create scalable immersive audio • MMO games were considered the hardest practical problem • Globally distributed user base, all IP connected • Real time voice for 5,000 player virtual worlds i.e. WoW • Cost effective to deliver for virtual worlds – even free to play • Integrated the technology into HL2:DM, CS:S and ET as demonstrators • Ran game trials with TRL Launceston CONFIDENTIAL INFORMATION
History: Spatial Voice and Axon • Spun off into a startup owned by the CRC and UOW in 2006 • Acquired by Dolby in September 2007 • Dolby created Axon • Voice platform targeting the Gaming segment • Technology from Spatial Voice • Enhanced with custom Echo and noise suppression, VAD, AGC, Codec and Renderer CONFIDENTIAL INFORMATION
Axon API Axon Desktop Client Dolby Axon Network • Consumer service • 100,000+ consumer accounts • Hybrid peer-to-peer and hosted • 10,000,000+ minutes of voice served by us each week • 99.99% uptime since launch • Wholesale Voice Service • GA Ready November 2010 • Infrastructure for 130,000 Peak Concurrent Users • Customers: • EA Need for Speed • Jump-gate Evolution • The platform for creation of other Axon Products • Middleware provided to game developers • 1,000,000 Peak concurrent users in China (Giant) CONFIDENTIAL INFORMATION
Building Production Software • Changing requirements • Security • Multiple developers • Long term support of code base • Diagnosing and fixing bugs in the field • Minimum downtime • Supporting legacy platforms • Time constraints CONFIDENTIAL INFORMATION
Design and Planning • Keep it simple • Make it modular • Think about the tradeoffs between different designs • Discuss with other team members • Document your invariants • Define object ownership and lifetimes CONFIDENTIAL INFORMATION
Development • Don’t trust you inputs • Especially from the network or user • Use asserts to check your state • Always handle the “Release build” case • Use logs and state dumps to provide a window into your code • Test as you develop • Pause in the debugger and check the state CONFIDENTIAL INFORMATION
Testing • Unit tests • System tests • Negative and Positive testing • Code reviews • Playing around and looking for trouble • Test release builds once you are getting close to shipping • Dynamic and static analysis • Valgrind • Visual Studio Analyze (2010 Premium or better) • Clang Static Analyzer • CPPCheck • Compiler warnings (Multiple compilers) CONFIDENTIAL INFORMATION
Deployment • Create procedures around production builds and deployment • Automate builds and deployment • If you can’t automate everything, automate what you can • Test after your deployment • Put in place mechanisms for informing customers about downtime and updates • Have a backup plan, deployment may fail CONFIDENTIAL INFORMATION
Production • Get feedback from users • Make it easy/automated • Include logs and state dumps • Crashes in the field • Keep debugging files • WER • OSX Crash files • Automated Uploads • Custom system • Breakpad + Socorro – Recommended • Automated handling of feedback and crashes CONFIDENTIAL INFORMATION
WER • Free* service • Provides access to minidumps • By default any application crash on Windows will be reported here CONFIDENTIAL INFORMATION
Mac OS X Crash reports Process: Dolby Axon [194] Path: /Applications/Dolby Axon.app/Contents/MacOS/Dolby Axon Version: 0.2.2.0 (1) Code Type: X86 (Native) Date/Time: 2012-05-01 23:30:54.496 -0700 OS Version: Mac OS X 10.6.8 (10K549) Exception Type: EXC_CRASH (SIGABRT) Crashed Thread: 0 Dispatch queue: com.apple.main-thread Assertion failed: (assert_buffsize(writec, readers[id].playc, readers[id].buffer_size)), function RegisterInputLane, file /DTC_branch/AxonCore/src/dvclient/EchoBuffer.cpp, line 278. Thread 0 Crashed: Dispatch queue: com.apple.main-thread 0 libSystem.B.dylib 0x9025ec5a __kill + 10 1 libSystem.B.dylib 0x9025ec4c kill$UNIX2003 + 32 2 libSystem.B.dylib 0x902f15a5 raise + 26 3 libSystem.B.dylib 0x903076e4 abort + 93 4 libSystem.B.dylib 0x902f420f __assert_rtn + 252 5 com.dolby.axon.dvclient 0x01191895 AxonClientPrivate::EchoBuffer::RegisterInputLane(unsigned int, unsigned int) + 485 6 com.dolby.axon.dvclient 0x011922e9 AxonClientPrivate::EchoBufferManager::RegisterInputLane(unsigned int, unsigned int) + 553 7 com.dolby.axon.dvclient 0x0118a637 AxonClientPrivate::InputLane::InputLane(AxonClientPrivate::EchoBufferManager*, Axon::Util::LogBackend*, unsigned int, AxonClientPrivate::InputDevice*) + 231 8 com.dolby.axon.dvclient 0x0118a53d AxonClientPrivate::InputLane::InputLane(AxonClientPrivate::EchoBufferManager*, Axon::Util::LogBackend*, unsigned int, AxonClientPrivate::InputDevice*) + 77 9 com.dolby.axon.dvclient 0x0119889d AxonClientPrivate::DeviceManager::AcquireInputLane(unsigned int, unsigned int, AxonClientPrivate::AxonEngine*, AxonClientPrivate::InputLane**) + 573 10 com.dolby.axon.dvclient 0x0112b8a8 AxonClientPrivate::AxonEngine::SetupInputLane(unsigned int, unsigned int, unsigned int*) + 104 11 com.dolby.axon.dvclient 0x012354a1 AxonClientPrivate::DVClientCommon::SetDevice(_DVCL_AUDIO_PATH_TYPE, DVCL_DeviceGuid_, unsigned int, unsigned int) + 737 • /Library/Logs/CrashReporter/ (10.5 and below) • /Library/Logs/DiagnosticReports/ (10.6 and above)
Custom System • DTC uses a launcher that acts a crash reporter • Any uncaught exceptions or crashes are automatically uploaded • Also detects potential deadlocks and uploads • Server collects reports and categorizes based on the call stack • Top 5 Windows crashes from a previous version • 1188 ./0xC0000005/iceclient_1/AxonClientPrivate__EchoBufferManager__DeregisterOutputLane/ • 342 ./0xC0000005/dsound/CThreadBase__StaticThreadProc/ • 230 ./0xE06D7363/kernel32/RaiseException/ • 190 ./0xC0000005/winmm/timeGetTime/ • 169 ./0xC0000005/wdmaud/CWorker___StaticThreadProc/ CONFIDENTIAL INFORMATION
Top Windows Crash iceclient_1!AxonClientPrivate::EchoBufferManager::DeregisterOutputLane Line 274 iceclient_1!AxonClientPrivate::DeviceManager::ReleaseOutputLane Line 299 + 0xF bytes iceclient_1!AxonClientPrivate::AxonEngine::Destroy Line 270 + 0xC bytes iceclient_1!AxonClientPrivate::AxonEngine::~AxonEngine Line 201 iceclient_1!AxonClientPrivate::TickManager::destroyhelper<AxonClientPrivate::AxonEngine> Line 94 + 0xE bytes iceclient_1!AxonClientPrivate::TickManager::~TickManager Line 50 + 0x1C bytes iceclient_1!AxonClientPrivate::DeviceManager::Shutdown Line 43 + 0x1B bytes Axon!_DVP_context::~_DVP_context Line 1319 Axon!DVP_destroy Line 89 + 0x6 bytes Axon!DVPModel::ModelManager::Release Line 42 + 0x1F bytes Axon!Run Line 92 Axon!wWinMain Line 155 + 0x5 bytes Axon!__tmainCRTStartup Line 324 + 0x1C bytes kernel32 + 0x1339A bytes [Frames below may be incorrect and/or missing, no symbols loaded for kernel32] ntdll!__RtlUserThreadStart + 0x70 bytes ntdll!_RtlUserThreadStart + 0x1B bytes CONFIDENTIAL INFORMATION
Support - DAN and DTC • Monitoring systems are in place • Key metrics are graphed • Checked twice daily by a developer on “support duty” • Automated scripts detect failures and look for problems • Any failures trigger alerts to the support device • Support diagnoses and handles it or escalates to module lead CONFIDENTIAL INFORMATION
Ensuring uptime • Redundancy is you friend • Dual redundant databases • Dual internet connections • Dual redundant switches • Dual power supplies • Multiple data centers* • Multiple servers, load based/round robin scheduling • Standby servers for single points of failure • Software that can automatically recover from a restart CONFIDENTIAL INFORMATION
General tips • Never ignore a crash or assert it will happen to a customer (or in a demo) • If you notice something unusual investigate, it is probably a bug • You will need diagnostics to help you solve issues in the field. Make life easier on yourself and write them early so you can use them in development and testing • Reread your code before committing • Something can always go wrong even with a “simple fix” • Prototypes become demos. Demos become products • Do what works for you and your team CONFIDENTIAL INFORMATION