Training 2014

1. Training2014 Session602 ROIinAction:AnIntegratedCaseStudy LearningObjectives:     Describeatleastfivewaystocollectdata Describeatleastthreewaystoisolatetheeffectsoftheprogram Identifyfourwaystoconvertdatatomoney ExplainthemisconceptionsofROI SpeakerName:PattiPhillips,Ph.D. Company:ROIInstitute,Inc. Phone:2056788101 Email:patti@roiinstitute.net Website:www.roiinstitute.net

2. Framework Measurementbeginswithaframework.Theframeworkservesasthebasisforansweringthe“So what?”questionandasthefoundationforansweringallotherquestions. LevelsofEvaluation Level0:InputsandIndicators Level1:Reaction,Satisfactionand PlannedAction Level2:Learning Level3:ApplicationandImplementation Level4:BusinessImpact Level5:ROI KeyQuestionsAnswered Howmanypeople? Howmanyhours? Howmanyofferings Whatarethecostsperperson? Istheprogramrelevanttoparticipants’jobsand mission? Istheprogramimportanttoparticipants’job/mission success? Doestheprogramprovidenewinformation? Doparticipantsintendtousewhattheylearned? Willparticipantsrecommendittoothers? Isthereroomforimprovementwithfacilitation, materials,andtheenvironment? Doparticipantsknowwhattheyaresupposedtodo withwhattheylearned? Doparticipantsknowhowtoapplywhatthey learned? Doparticipantsknownextsteps? Areparticipantsconfidentintheirabilitytoapply whattheylearned? Howeffectivelyareparticipantsapplyingwhatthey learned? Howfrequentlyaretheyapplyingwhattheylearned? Howmuchofwhattheylearndotheyapply? Iftheyareapplyingwhattheylearned,whatis supportingthem? Iftheyarenotapplyingwhattheylearned,whynot? Towhatextentdoesparticipantapplicationofwhat theylearnedimprovemeasuresofoutput,quality, cost,time,customersatisfaction,jobsatisfaction, workhabits,and/orinnovation? Howdoweknowitwastheprogramthatimproved thosemeasures? Dothemonetarybenefitsoutweighprogramcosts? BCR=ProgramBenefits ProgramCosts ROI=NetProgram Benefitsx100 ProgramCosts 2                      

3. DefiningBenefit/CostRatioandtheReturnonInvestment ProgramBenefits Benefits/CostRatio= ProgramCosts NetProgramBenefits ROI= x100 ProgramCosts ProgramBenefitsfromParticipants(1styear)=$750,000 CostsperProgram(25participants)=$425,000 ROICalculation Example BCR= = ROI= = x100= ROITarget—Options     Setthevalueaswithotherinvestments:i.e.,15% Setslightlyaboveotherinvestments:i.e.,25% Setatbreak-even:0% Setatclientexpectations Whatvaluewouldbeappropriate?WhatisanacceptableROIinyourorganization?________% 3

4. CaptureCosts OfProgram ROIMethodologyTM DataAnalysis Reporting EvaluationPlanning DataCollection LEVEL1: LEVEL3: Reactionand Application PlannedAction Develop/ Review Objectivesof Program Develop Evaluation Plansand BaselineData CollectData During Program Implementation CollectData After Program Implementation Calculate theReturnOn Investment Generate Impact Study Isolatethe Effectsof Program ConvertData toMonetary Value LEVEL2: Learning ROICalculation NetProjectBenefits ProjectCosts LEVEL4: BusinessImpact Level 1.Reaction&Planned LEVEL5:ROI Identify Intangibles IntangibleBenefits EvaluationFramework MeasurementFocus Measuresparticipantreactiontotheprogramand ROI= Action 2.Learning 3.Application capturesplannedaction. Measureschangesinknowledgeandskills. Measuresimplementation,actions,andchangesin behavioronthejob. Costofproject$230,000 Benefitsofproject(1styear)$430,000 $430,000-$230,000 ROI= =0.87x100=87% $230,000 4.BusinessImpact Measureschangesinbusinessimpactvariables. 5.Returnon Investment Comparesmonetarybenefitsoftheimpactofthe program. 4

5. Standards GuidingPrinciples 1.Whenconductingahigher-levelevaluation,collectdataatlowerlevels. 2.Whenplanningahigher-levelevaluation,thepreviouslevelofevaluationisnot requiredtobecomprehensive. 3.Whencollectingandanalyzingdata,useonlythemostcrediblesources. 4.Whenanalyzingdata,selectthemostconservativealternativesforcalculations. 5.Useatleastonemethodtoisolatetheeffectsoftheprogramorproject. 6.Ifnoimprovementdataareavailableforapopulationorfromaspecificsource,assume thatnoimprovementhasoccurred. 7.Adjustestimatesofimprovementsforthepotentialerroroftheestimates. 8.AvoiduseofextremedataitemsandunsupportedclaimswhencalculatingROI calculations. 9.UseonlythefirstyearofannualbenefitsintheROIanalysisofshort-termsolutions. 10.Fullyloadallcostsofthesolution,project,orprogramwhenanalyzingROI. 11.Intangiblemeasuresaredefinedasmeasuresthatarepurposelynotconvertedto monetaryvalues. 12.CommunicatetheresultsoftheROIMethodologytoallkeystakeholders. TheCompleteStory 5

6. Challenges CollectingData TypesofData Method 1 2 3 4               Surveys Questionnaires Observation Interviews FocusGroups     Tests/Quizzes Demonstrations Simulations      ActionPlanning/ImprovementPlans PerformanceContracting PerformanceMonitoring FactorstoConsider Whenselectingmethods,consider:        Accuracy Utility Culture/Philosophy Timerequiredforparticipants Timerequiredforsupervisors Costsofmethods Amountofdisruption Whenselectingsourcesofdata,consider:        Internalstaff Externalsources Organizationalrecords Participants Supervisors Directreports Peergroups Whendeterminingtiming,consider:      Availabilityofdata Idealtimeforbehavior change/application Idealtimeforbusinessimpact Convenienceofdatacollection Constraintsondatacollection 6

7. IsolatingtheEffectsoftheProgram Thesteptoisolatetheeffectsofaprogramorprojectistheultimatestepinaligningprograms withthebusiness.Itallowslearninganddevelopmentprofessionalstodemonstratehowmuch oftheimprovementinameasureisduetoyourprogram.Itanswersafundamentalquestion: Howdoyouknowitistheprogramorprojectthatisinfluencingimprovementinthemeasure? Or,anotherwaytoputit:Howmuchoftheimprovementisduetotheprogramandnotsome otherinfluence? Methods 60% 55% 1. 2. ControlGroups TrendLineAnalysis 50% 3. 4. 5. 6. ForecastingMethods ParticipantEstimates ManagerEstimates SeniorManagerEstimates 37% 40% 30% 32% 29% 7. 8. ExpertInput CustomerInput 20% 13% 11% 11% 10% 5% *SurveyofUsers,N=235 0% 1 2 3 4 5 6 7 8 ControlGroups ClassicDesignanswersthequestion:Whatisthedifferenceinthechangeinperformance? ExperimentalGroup Program PreMeasure ControlGroup PreMeasure PostMeasure PostMeasure Example Measure:Annualturnoverof1500technicalspecialists.Randomlyassign100toexperimental andcontrolgroup.ExperimentalgroupisinvolvedinaMastersDegreepilotrollout. Differenceinthechangeinperformanceis30–8=22.Thereductionof22turnoverstatistics canbeexplainedbyparticipationinthepilotprogram. 7

8. Post-ProgramOnlyDesignanswersthequestion:Whatisthedifference?Post-ProgramOnlyDesignanswersthequestion:Whatisthedifference? ExperimentalGroup Program ControlGroup PostMeasure PostMeasure Example Measure:Salesperweekpersalespersonfor46salespeoplefromthreestoresinvolvedina salestrainingprogram.Controlgroupwasuninvolved.Groupswerematchedbasedonstore salesperformance,customertraffic,storelocation,andstoresize. Differenceinsalesbetweengroupsis$1,626.Theincreaseinsalesof$1,626perpersonper weekcanbeexplainedbyparticipationintheprogram. TrendLineAnalysis Measureisunexpectedabsences.Asshowninthefigureonthenextpage,historicaldata indicateatrend–absencesaregoingdown.Thesix-monthaverageis18absencespermonth, basedonthistrend.HRimplementsaperformancemanagementprocess.Unexpected absencesaretrackedforsixmonths.Thepost-programaverageis7permonth. Whatisthereductioninabsencesduetotheperformancemanagementprocess?__________ 18PreProgramSix-Month PerformanceManagement ProjectedAverage— UsingPreDataasa Base14 7PostProgram Six-Month Average 20 Unexpected absences J F M A M J J A S O N D J MONTHS 8

9. Conditions forUsingTrendLineAnalysis     Pre-programdataavailable Dataitemsarestable Pre-programinfluencesexpectedtocontinue Nonewinfluencesenterthepost-programperiodexceptfortheprogram Estimations Measureisunexpectedabsences.Recordsshowthatoverasix-monthperiodunexpected absenceshavegonedownby11permonth.Managementwantstoknowwhatcausedthe reduction. Identifythemostcrediblesourcesofdata–thosewhoknowbestaboutwhyabsenceshave gonedown.Askthreequestions: 1.Giventhereductioninunexpectedabsences,whatarethefactorsthatcouldhave causedtheseabsencestogodown? 2.Givethosefactors,whatpercentageofthedecreaseinabsencesisduetoeachfactor? 3.Howconfidentareyouinyourestimates? Giventheresponsesinthetablebelow,howmanyabsenceshavegonedownasaresultofthe performancemanagementprocess?_____________________ AverageMonthlyDecreaseinAbsences:11 ContributingFactors PerformanceManagementProgram LeadershipChanges RecruitingStrategies JobEngagementInitiative Other AverageImpacton Results(percent) 32% 41% 14% 11% 2% 100% Average ConfidenceLevel (percent) 83% 87% 62% 75% 91% 9

10. ConvertingImpactDatatoMoney Ifyouwanttoshowtherealeconomicimpactofaprogramorprocess,youmustconvert businessmeasurestomoney.Thereareavarietyofwaystotakethisstep.Thesimplestis throughtheuseofstandardvalues(e.g.profitmarginonsales;costofquality;valueoftime). Butwhenastandardvaluedoesnotexist,thereareothertechniques.           Convertingoutputtocontribution–standardvalue Convertingthecostofquality–standardvalue Convertingemployee’stime–standardvalue Usinghistoricalcosts Usinginternalandexternalexperts Usingdatafromexternaldatabases Linkingwithothermeasures Usingparticipants’estimates Usingsupervisors’andmanagers’estimates Usingstaffestimates HistoricalCosts andExpertInput 35Complaints ActualCostsfrom Records LegalFees,Settlements, Losses,Material,Direct Expenses AdditionalEstimated CostsfromStaff EEO/AAStaffTime, ManagementTime $852,000Annually LinkingwithOther Measures ClassicRelationships Jobsatisfaction Jobsatisfaction Jobsatisfaction Organizationcommitment Engagement Customersatisfaction Turnover Absenteeism Customersatisfaction Productivity Productivity Revenue vs. vs. vs. vs. vs. vs. 10

11. ExampleofLinkagewithOtherMeasures ACompellingPlace toWork ACompellingPlace toShop ACompellingPlace toInvest Customer Recommendaons Customer Impression Customer Retenon Drives Atude About theJob Atude Aboutthe Company 5-Unit Increase inEmployee Attitude Service Helpfulness Merchandise Value 1.3-Unit Increasein Customer Impression ReturnonAssets OperangMargin RevenueGrowth 0.5Increase inRevenue Growth Employee Behavior Employee Retenon Drives 2 Source:Rucci,A.J.,Kirn,S.P.,andQuinn,R.T.(1998January-February).TheEmployee-Customer ProfitChainatSears.HarvardBusinessReview.ReprintNumber98109. FiveStepsto Convert Impact toMonetary Benefits 1. 2. 3. 4. 5. Focusonaunitofmeasure Determinethevalue(V)ofeachunit Calculatethechangeinperformance(P) Determinetheannualchangeinperformance(AP) Calculatethetotalannualvalueoftheimprovement(APxV) StandardValuesasSourcesofMonetaryValue 1. 2. 3. 4. 5. Onesale V=30%profitmargin P=$20,000revenuepermonth AnnualP= APxV= InternalExpertsasSourcesofMonetaryValue 1. 2. 3. 4. 5. Onecomplaint V=$6,500(fromDirectorofnursingandHRExperts) P=averageof3outof11complaintspreventedpermonthduetotheprogram AnnualP= APxV= 11

12. DatabasesasSourcesofMonetaryValue 1. 2. 3. 4. 5. Oneturnoverstatistic(MiddleManagers:Salary$150,000) V=125%ofannualsalary= P=10turnoverstatisticsperyear AnnualP= APxV= TabulatingFully-LoadedProgramCosts          AssessmentCosts(Prorated) DevelopmentCosts(Prorated) ProgramMaterials Instructor/FacilitatorCosts FacilitiesCosts Travel/Lodging/Meals ParticipantSalariesandBenefits Administrative/OverheadCosts EvaluationCosts IdentifyingtheIntangibleBenefits Inmanycases,programsdrivemorethanbusinessmeasuresweconverttomoney.Theydrive asixthtypeofdata–intangiblebenefits.Intangiblebenefitsarethosebusinessmeasureswe choosenottoconverttomoney.Suchas: • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Adaptability Awards Brandawareness Careerminded Caring Collaboration Communication Commitment Conflicts Cooperation Corporatesocialresponsibility Creativity Culture Customercomplaints Customerresponsetime Customersatisfaction Decisiveness Decisiveness Employeecomplaints Engagement Execution Image Innovation Jobsatisfaction Leadership Networking Organizationalclimate Organizationalcommitment Partnering Reputation Resilience Stress Talent Teamwork 12

13. RetailMerchandiseCompany Situation RetailMerchandiseCompany(RMC)isanationalchainof420stores,locatedinmostmajor USAmarkets.RMCsellssmallhouseholditems,giftsofalltypes,electronics,andjewelry,as wellaspersonalaccessories.Itdoesnotsellclothesormajorappliances.Theexecutivesat RMChavebeenconcernedabouttheslowsalesgrowthandwereexperimentingwithseveral programstoboostsales.Oneoftheconcernsfocusedontheinteractionwithcustomers.Sales associateswerenotactivelyinvolvedinthesalesprocess,usuallywaitingforacustomerto makeapurchasingdecisionandthenproceedwithprocessingthesale.Severalstore managershadanalyzedthesituationtodetermineifmorecommunicationwiththecustomer wouldboostsales.Theanalysisrevealedthattheuseofverysimpletechniquestoprobeand guidethecustomertoapurchaseshouldboostsalesineachstore. Theseniorexecutivesaskedthetraininganddevelopmentfunctiontoexperimentwithavery simplecustomerinteractiveskillsprogramforasmallgroupofsalesassociates.Aprogram producedbyanexternalsupplierwouldbepreferredtoavoidthecostofdevelopment, particularlyiftheprogramprovedtobeineffective.Thespecificchargefromthemanagement teamwastoimplementtheprograminthreestores,monitortheresults,andmake recommendations.IftheprogramincreasedsalesandpresentedasignificantpayoffforRMC,it shouldbeimplementedinotherstores. Thesalesassociatesaretypicaloftheretailstoreemployeeprofile.Theyareusuallynot collegegraduatesandmosthaveafewmonthsofretailstoreexperience.Turnoverisusually quitehighandformaltraininghasnotbeenamajorpartofprevioussalesdevelopmentefforts. TheSolution Thetraininganddevelopmentstaffconductedaverybriefinitialneedsassessmentand identifiedfivesimpleskillsthatwouldneedtobecoveredintheprogram.Fromtheiranalysis,it appearedthatthesalesassociatesdidnothavetheseskillsorwereveryuncomfortablewiththe useofthem.Aprogramcalled“InteractiveSellingSkills”wasselected–aprogramwith significantuseofskillpractices.Theprogramhadtwodaysoftraining,whereparticipantshad anopportunitytopracticeeachoftheskillswithafellowclassmate,followedbythreeweeksof on-the-jobapplication.Then,afinaldayoftrainingwasconductedthatincludedadiscussionof problems,issues,barriers,andconcernsaboutusingtheskills.Additionalpracticeandfine- tuningofskillswasapartofthatfinalsession.Theprogram,anexistingproductfroman externaltrainingsupplier,wouldbetriedintheelectronicsareaofthreestoresandsixteen peoplewouldbetrainedineachstore.Theprogramwasactuallytaughtbythestaffofthe trainingsupplierforapredeterminedfacilitationfee. Questionsfordiscussion: 1.Whichtypesofdatadoestheseniormanagementteamdesire? 2.Whichdatacollectionmethodsshouldbeutilizedforapplicationandbusinessimpact measurement?(Level3and4) 3.Whichmethodshouldbeutilizedtoisolatetheeffectsoftheprogram? 4.Howshouldthebusinessimpactdatabeconvertedtomonetaryvalues? 13

14. 14