1 / 9

mccoy_2018_oi_180183

Mccoy published document

Praful3
Download Presentation

mccoy_2018_oi_180183

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OriginalInvestigation | StatisticsandResearchMethods AssessmentofTime-SeriesMachineLearningMethods forForecastingHospitalDischargeVolume ThomasH.McCoyJr,MD;AmeliaM.Pellegrini,BA;RoyH.Perlis,MD,MSc Abstract KeyPoints Question Whatistheperformanceofa newtime-seriesmachinelearning methodforpredictinghospital dischargevolume? IMPORTANCE Forecastingthevolumeofhospitaldischargeshasimportantimplicationsfor resourceallocationandrepresentsanopportunitytoimprovepatientsafetyatperiodsof elevatedrisk. Findings Inthiscohortstudyofdaily hospital discharge volumes at 2 academicmedicalcenters(101867 patientdischarges),predictorsof dischargevolumewerewellcalibrated. These findings were achieved even with shortertrainingsetsandinfrequent retraining. OBJECTIVE Todeterminetheperformanceofanewtime-seriesmachinelearningmethodfor forecastinghospitaldischargevolumecomparedwithsimplermethods. DESIGN A retrospective cohort study of daily hospital discharge volumes at 2 large, New England academicmedicalcentersbetweenJanuary1,2005,andDecember31,2014(hospital1),orJanuary1, 2005,andDecember31,2010(hospital2),comparingtime-seriesforecastingmethodsfor predictionwasperformed.DataanalysiswasconductedfromFebruary28,2017,toAugust30,2018. Group-leveldataforalldischargesfrominpatientunitswereincluded.Inadditiontoconventional methods,atechniqueoriginallydevelopedforallocatingdatacenterresources,andcomparison strategies for incorporating prior data and frequency of model updates, was conducted to identify themodelapplicationthatoptimizedforecastaccuracy. Meaning Theseresultsappearto demonstratethefeasibilityofdeploying simpletime-seriesmethodstomore preciselyestimatehospitaldischarge volumesbasedonhistoricaldata,and may facilitate better matching of resourceswithclinicalvolume. MAINOUTCOMESANDMEASURES ModelcalibrationasmeasuredbyR2and,secondarily,number ofdayswitherrorsgreaterthan1SDofdailyvolume. RESULTS Duringtheforecastedyear,hospital1had54411discharges(dailymean,149)andhospital 2had47456discharges(dailymean,130).Themachinelearningmethodwaswellcalibratedatboth sites(R2,0.843and0.726,respectively)andmadeerrorsgreaterthan1SDofdailyvolumeononly 13 and 22 days, respectively, of the forecast year at the 2 sites. Last-value-carried-forward models performedsomewhatlesswell(calibrationR2,0.781and0.596,respectively)with13and46errorsof 1SDorgreater,respectively.Morefrequentretrainingandtrainingsetsoflongerthan1yearhad minimaleffectsonthemachinelearningmethod’sperformance. +Supplementalcontent Authoraffiliationsandarticleinformationare listedattheendofthisarticle. CONCLUSIONSANDRELEVANCE Volumeofhospitaldischargescanperhapsbereliablyforecasted usingsimplecarry-forwardmodelsaswellasmethodsdrawnfrommachinelearning.Thebenefitof thelatterdoesnotappeartobedependentonextensivetrainingdataandmayenableforecastsupto 1yearinadvancewithsuperiorabsoluteaccuracytocarry-forwardmodels. JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 Introduction Variationsindischargevolumescreateachallengeforhospitals.Adequatestaffingisessentialfor optimizingpatientoutcomes;however,thesestaffmembersareasignificantsourceoffixedhospital cost.1-3Assuch,volume-matchedstaffingisanimportantcomponentinthegoalofdeliveringhigh- value care. The biomedical literature includes many efforts to predict discharges at the level of OpenAccess.ThisisanopenaccessarticledistributedunderthetermsoftheCC-BYLicense. JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 1/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  2. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume hospitalunitorclinicaldomain.4-6Althoughtheseeffortsareinvaluabletoolsfordiscovery,the resourcedemandissuchthattheycannottypicallybeintegratedintoroutineoperationsasa monitoringtoolorscaledacrossallunits;thus,thereisaneedforhighlyscalableforecasting approachesthataresuitableforbroadapplicationandoperationalimplementation. Predictingtime-seriesdata—thatis,usingpastinformationtoforecastfuturevaluesofthe series—isanareaofinterestinthefieldofmachinelearningandstatisticsmorebroadly.Facebook recentlyreleasedsoftwareimplementingaBayesianforecastingapproachdevelopedforallocation ofcomputationalresources.7Thismethodrecognizesrepeatingpatternsoverweeks,months,years, andidentifiedholidays.Recognizingthattheseseculartrendsareimportantdriversofhospital volume,wehypothesizedthatthismethodwouldalsobewellsuitedtohospitalvolumeforecasting. Wefurtherhypothesizedthatminimaldependenceontuningofhyperparameters,achallenge with many standard methods in machine learning, would make implementation practical and generalizationpossible.WethereforeappliedtheFacebookforecastingmethodtopredictdischarge volumefrom2largeacademicmedicalcenters.Withaneyetowarddeploymentofthissystem,we examinedtheimportanceoflargetrainingdatasets(ie,consideringlongervsshorterperiodsoftime) andfrequenttraining(ie,regeneratingthemodelonaregularbasisvsinfrequently).8,9 Theoverallaimofthestudywastounderstandthistool’sperformancesufficientlytofacilitate broaderdisseminationandapplicationamonghospitalsystems.Tocontextualizethisunderstanding, wealsoappliedsimpleprevious-value-carried-forwardandautoregressiveapproachesthathave beenstudiedbyotherinvestigatorsinthecontextofhospitalvolumeforecasting.10-14 Methods OverviewandDataSetGeneration Hospitaldischargedataforeachcalendardatewereextractedfromthelongitudinalelectronichealth recordsof2large,NewEnglandacademicmedicalcenters.Datacoveringdifferentyearswere availablefromthe2sites.Athospital1,datafromJanuary1,2005,throughDecember31,2014,were available,whereasathospital2,datafromJanuary1,2005,throughDecember31,2010,were available.Weanalyzedtime-seriesdatainwhichtheunitofanalysiswascalendardate.Whilehospital shiftsdonotcorrespondsolelytosuchdates,theavailabledataallowedreliableestimatesof calendardatesonly.Nodataweremissingand,thus,noimputationstrategywasrequiredandall availabledatawereincluded.DataanalysiswasconductedfromFebruary28,2017,toAugust30, 2018.Adatamartcontainingthesedatawasgeneratedwiththei2b2,version1.6serversoftware (i2b2tranSMARTFoundation),acomputationalframeworkformanaginghumanhealthdata.15,16 ThePartnersHumanResearchCommitteeapprovedallaspectsofthisstudywithwaiverof informedconsent.ThestudywasconductedusingtheStrengtheningtheReportingofObservational StudiesinEpidemiology(STROBE)reportingguideline. StatisticalAnalysis Theprimarylearningtaskinthisstudywasaforecastofdailyhospitaldischargevolumeforthelast full year available for both hospitals (2010). This task was approached using 5 separate models for subsequent comparison: 3 simple variations on prior values carried forward, a seasonal autoregressive-integratedmovingaverage(SARIMA)model,andFacebook’sProphetmodel(FIS Corp).7Theprimaryoutcomeforcomparisonbetweenmodelswaspredictionaccuracy,measuredby correlationbetweenpredictedvalueandactualobservedvalueoverthe1-year(2010)prediction horizon.Thisoutcomewascalculatedasthelinearmodelobservedday= β0+β1forecastedday,1.As eachcomponentofthismodelisinterpretable,itisreportedinwholewithR2valuesandtheir95% CIs.17Tofurthercharacterizemodelperformanceinunitsofdischarges,errorwasoperationalizedas the difference of the predicted and the observed number of discharges over the forecast period (forecastedday− observedday).Becausetheerrorcanbenegativeandthuserrorsovertheforecasting horizon could cancel one another, which may or may not be desirable depending on intended use, JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 2/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  3. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume bothtotalandtotalabsoluteerrorarereported.18Exceptwherenotedinthesecondaryanalysis,the forecastinghorizonwas1year. Prophetisanopen-sourceimplementation(PythonandRinterfacesavailable)ofaBayesian forecaster with learned modeling of yearly and weekly seasonality, as well as prespecified holidays expectedtobeanomalous,whichautomaticallydetectschangepointsinagrowthcurve,releasedby FacebookResearchinearly2017.Conceptually,Prophetreframesforecastingasacurvefitting problemusingadecomposabletime-seriesmodelincludingholidays,seasonality,andoveralltrend thatmakesuseofnonlinearsmoothers.19 The3carry-forwardmodelswerethecorrespondingday,1yearearlier;thecorrespondingday,1 weekearlier;andthemeanofthese2.Forexample,fortheyearlycomparison,thesecondMonday of 2010 would be compared with the second Monday of 2009, representing a simple means of forecastingvolumethatstilltakesintoaccountdayofweekandseasonaleffects.Fortheweekly comparison, the second Monday of 2010 would be predicted to have the same volume as the first Mondayof2010.Thethirdcarry-forwardforecastforthesecondweekof2010wouldbethemeanof theprior2(secondMondayof2009andfirstMondayof2010). Fortheprimaryanalysis,forecasting2010volume,Prophetwastrainedonallprioryears (January1,2005,throughDecember31,2009)andthenusedtopredictthefull2010calendaryear. Hospitalcalendarswereusedtoidentifyobservedholidaysateachsiteandthesewereusedin trainingandforecastingofboththeProphetandSARIMAmodels.Inall5cases,eachhospitalwas modeledindependently.AllanalysiswasperformedusingR,version3.4withtheRinterfaceto Prophet,version0.1.1. ModelParameterInvestigation Wenextexamined2importantoperationalcharacteristicsofProphetrelevanttoclinical disseminationandoperationalizationofhospitaldischargesforecasting.First,weallowedthetraining datasettovarybetween1and5yearsforallyearsateithersitewithatleast5yearsofpriordata availablefortraining.Inotherwords,asbefore,2010wouldbepredictedbutthistimeusingfirstonly 2009,then2009and2008,then2009to2007,andsoonbackto2005.Inthisanalysis,theyears availableforonly1ofthe2hospitals(2011-2014)wereincludedasforecastingtargets,subjecttothe 5-yeartrainingdatalimitforcomparability.Thisvariablereflectstheamountoftrainingdatarequired to build a reliable prediction model, that is, whether a hospital with a single year of discharge data couldbenefitfromapplicationofthismodelandwhetherahospitalcouldreasonablyexpect accuracytoimprovewithadditionaldata.Thisassessmentoftheconsequenceofadditionaltraining datacomesfromthemachinelearningliteratureonlearningcurves.20 Second, we compared the forecast accuracy when run once a year vs rerunning on a monthly basis.Inotherwords,asbefore,2010wouldbepredictedbutthistimethefirstfitoftheyear(2005- 2009)wouldbeusedtoforecastJanuary2010;next,2005toJanuary2010wouldbeusedto predictFebruary2010,andsoonthroughtheendoftheyear.Thisvariableprovidesguidanceabout howfrequentlyamodelshouldberegeneratedandinsightintohowquicklyforecastaccuracy degradeswithdistancefromthelasttrueobservation.Thisiterativerefittingofamodelusinga shorterforecastinghorizonhasconceptualvalidationtotheideaofcrossvalidation.21These follow-upsecondaryexperimentswereperformedonlyfortheProphetmodel. Results Overthecourseoftheprimaryoutcomeyear,2010,hospital1had54411discharges(dailymean,149) andhospital2had47456discharges(dailymean,130).Fortheprimaryoutcome,accuracyofthe 2010 forecast based on all prior data, the Prophet model was the most accurate of the 5 models at both hospitals (Table 1 and Figure 1). The mean absolute error of the 1-year forecast by the Prophet modelathospital1was11.5dischargesperdayand11.7dischargesperdayathospital2.Amongthe 3 carry-forward models, the mean of the prior week and prior year’s value had the highest accuracy JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 3/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  4. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume (Table1).Themeanabsoluteerroroftheforecastbythemeanofthepriorweekandprioryearcarried forwardmodelathospital1was13.7dischargesperdayand14.3dischargesperdayathospital2.To further characterize the forecast accuracy, we selected 3 error thresholds (1 SD of daily volume, 25 discharges, and 10 discharges) and compared the total number of days for which the absolute forecasterrorwasabovethethresholdforthe2bestmodels(Prophetandthemeanoftheprior weekandyear).TheseperformancemetricsarepresentedinTable2,withProphetoutperforming the mean carry-forward model in 5 of 6 comparisons. Prophet was well calibrated at both sites (R2, 0.843 and 0.726, respectively) and made errors greater than 1 SD of daily volume on only 13 and 22 days,respectively,oftheforecastyearatthe2sites.Last-value-carried-forwardmodelsperformed somewhat less well (calibration R2, 0.781 and 0.596, respectively) with 13 and 46 errors of 1 SD or greater,respectively. Wecomparedthetotalabsoluteforecasterrorandthetotalforecasterrorforbothofthe top-performingmodels(Table3).Inthiscomparisonoftotalerror,themeancarry-forwardmodel Table1.CalibrationofTargetYearPredictionbyModelandHospitala Calibration (95% CI) Hospital 1 y = 64 + 0.58 × x; R2= 0.655 (0.598-0.711) y = 30 + 0.8 × x; R2= 0.644 (0.586-0.702) y = 21 + 0.86 × x; R2= 0.756 (0.713-0.799) y = 11 + 0.93 ×x; R2= 0.781 (0.742-0.820) y = −6.5 + 1 × x; R2= 0.843 (0.814-0.872) Model SARIMA Last week carried forward Last year carried forward Mean of last week and year Prophet Hospital 2 y = 57 + 0.63 × x; R2= 0.359 (0.281-0.437) y = 49 + 0.62 × x; R2= 0.384 (0.306-0.461) y = 25 + 0.8 × x; R2= 0.596 (0.532-0.659) y = 16 + 0.88 × x; R2= 0.596 (0.532-0.659) y = −13 + 1.1 × x; R2= 0.726 (0.678-0.773) Abbreviation:SARIMA,seasonalautoregressiveintegratedmovingaverage. aWithassociationcalibrationsshowninFigure1. Figure1.ComparisonofDischargePredictionAccuracyThroughCalibrationCurvesforProphet,MeanofLastYearandLastWeekCarriedForward, andSeasonalAutoregressiveIntegratedMovingAverage(SARIMA)Model Hospital 1 A Prophet Mean (prior week and prior year carried forward) SARIMA 250 250 250 Discharges Observed, No. Discharges Observed, No. Discharges Observed, No. 200 200 200 150 150 150 100 100 100 50 50 50 0 0 0 50 100 150 200 50 100 150 200 50 100 150 200 250 250 250 Discharges Predicted, No. Discharges Predicted, No. Discharges Predicted, No. Hospital 2 B Prophet Mean (prior week and prior year carried forward) SARIMA 250 250 250 Discharges Observed, No. Discharges Observed, No. Discharges Observed, No. 200 200 200 150 150 150 100 100 100 50 50 50 0 0 0 50 100 150 200 50 100 150 200 50 100 150 200 250 250 250 Discharges Predicted, No. Discharges Predicted, No. Discharges Predicted, No. JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 4/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  5. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume outperformedProphetontheneterroroverthecourseofthefullyearforecast,asthismodeltended tooverpredictandunderpredictinequalmeasureandthusnegativeandpositiveerrorscanceled eachotheroverthecourseoftheyear,whereasProphetconsistentlyoverpredictedhospitalvolume butdidsotoalesserextentthanthemeancarry-forwardmodelasindicatedbythetotalabsolute errorinTable3.Whetherintermsofcalibration(Table1),daysaboveerrorthreshold(Table2),or cumulativeerroroverthefullforecasthorizon(Table3andFigure2),theautoregressivemodel producedlargererrorsthantheProphetmodel. Inthesecondaryanalysis,weassessedtheconsequencesoftrainingdataandforecastwindow ontheaccuracyofProphetmodelpredictions.Additionaltrainingdata,added1yearatatime,slightly increasedtheaccuracyofProphetforecastsandaresummarizedineFigure1intheSupplement. Similarly, refitting the model monthly—using a shorter forecast horizon—had a minimal association withaccuracy(eFigure2andeTables1-3intheSupplement,whichmirrorTable1,Table2,andTable3 usingtheshorterpredictionwindow). Discussion Inthisefforttomodelvolumeofhospitaldischargefrom2largeacademicmedicalcentersspanning more than a decade, we found that an open-source tool intended to model server load reliably, if imprecisely,predictedvolume.Thepredictionswerebettercalibratedthanthosemadeby autoregressivemodelsandsimplecarryforwardofpriorvolumes.Moreover,themodestamountof training data required and the adequate performance for up to 365 days of follow-up suggest that this approach is feasible for essentially any hospital. It appears that the largest portion of forecast accuracycanberecognizedwithasingleannualforecastingeffortbasedononlytheprioryear’sdata. Unlike many methods in machine learning, the model training and forecasting reported herein can be replicated on an Intel i5-2400 system from 2011 in less than half an hour. In short, this method is neitherdatanorcomputeintensiveandthuscouldbewidelyadopted.Assuch,giventhatthe existingliteratureonforecastingusingcarry-forwardmodels,conventionalregression, autoregression,andmoreexoticmodelsismixedwithrespecttomostsuccessfulmodel,theProphet modelisofparticularappealasitbothperformswellandishighlyusableintermsofcomputational, data,andhumanresources.10-14,22 Istheabilitytoreliablypredictvolumeusefulforqualityandsafety?Certainlyattheextremes, matching patient load is important; studies suggest optimal patient to clinical staff ratios vary Table2.NumberofDaysOverForecastYearWithForecastErrorExceedingaGivenThresholda Days, No. (%) Hospital 1 Hospital 2 Mean of Last Week and Year 13 (3.56) 56 (15.34) 196 (53.7) Mean of Last Week and Year 46 (12.6) 59 (16.16) 208 (56.99) Error Threshold, Days >1 SDb >25 >10 Prophet Model 13 (3.56) 28 (7.67) 170 (46.58) SARIMA 81 (22.19) 173 (47.40) 303 (83.01) Prophet Model 22 (6.03) 32 (8.77) 184 (50.41) SARIMA 120 (32.89) 142 (38.90) 256 (70.14) bStandarddeviationofeachsite’sdailydischargevolume. Abbreviation:SARIMA,seasonalautoregressiveintegratedmovingaverage. aDenominator365days. Table3.AbsoluteTotalandTotalCumulativeErrorOvertheForecastYeara Hospital 1 (n = 54411) Hospital 2 (n = 47456) Mean of Last Week and Year 4997 −197 Mean of Last Week and Year 5220 32 Error Measure Total absolute error Total error Prophet Model 4189 1295 SARIMA 9699 −1525 Prophet Model 4262 968 SARIMA 8157 −5161 Abbreviation:SARIMA,seasonalautoregressiveintegratedmovingaverage. aDenominator365days. JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 5/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  6. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume substantiallybyspecialtybutareassociatedwitharangeofoutcomes,includingmortality.23 Differencesinriskandlengthofstayassociatedwithdischargeonweekendsoratnightfurther underscoretheimportanceofsuchstaffingdecisions,althoughnotallstudiesfindsuch variability.24-28Conversely,consistentlyerringonthesideofoverstaffingislikelytoentailadditional costs,consumingresourcesthatcouldbebetterspentonotherquality-improvementstrategies.As such, even coarse predictions may allow hospital administrators to better balance staffing and patientneeds.Wearenotthefirsttonotetheimportanceofholidaysinforecastinghospitalvolume asthesedaysareofparticularrelevanceinstaffing.13Furthermore,weareinterestedinthepossibility ofusingreal-timedeviationfromforecastedvolumeatthenursingunitandclinicalservicelevelasa means of gaining insight into health system performance; however, this application requires additionalworkbeyondthefoundationaleffortreportedhere. Limitations Wenoteseverallimitationsininterpretingtheseresults.First,whileonaverage,errorsaresmall,the absolute errors on any given day may be relatively large. At each of the 2 hospitals, the error exceeded25patientsonfewerthan10%ofthedays.Althoughtheseerrorsarestilllessthanthose arisingfromasimplerpredictionapproach,theynonethelessindicatethataflexiblestaffingmodelis likelytobenecessaryevenwithoptimalprediction.Inaddition,weemphasizethattheseestimates represent only a starting point. It is likely that further optimization, for example, taking into account weatherorlocalratesofinfluenzainfectioninwinter,ormodelingindividualunits,wouldallowmore precisenear-termpredictions.12Ontheotherhand,astrengthoftheapproachstudiedhereisthat Figure2.ComparisonofCumulativeTotalAbsoluteErrorOvertheCourseoftheForecastedYear byHospitalSiteandPredictiveModel Hospital 1 A 10000 Model SARIMA Mean (year, week) Prophet 7500 Total Absolute Error, No. 5000 2500 0 Jan 2010 Apr 2010 Jul 2010 Days Oct 2010 Jan 2011 Hospital 2 B 10000 7500 Total Absolute Error, No. 5000 2500 0 SARIMAindicatesseasonalautoregressiveintegrated movingaverage. Jan 2010 Apr 2010 Jul 2010 Days Oct 2010 Jan 2011 JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 6/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  7. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume itisreadilyimplementedatnearlyanysitewithoutrequiringotherdatastreamsortuningof hyperparameters.Theeaseoffittingisofparticularimportancegiventhevariabilityinmodel performanceseenbetweenthe2hospitalsites.Thisvariabilityisconsistentwiththeexisting literaturethatshowsvariableresults.10-14Assuch,thoselookingtoforecastvolumeshouldevaluate arangeofmodelsandconsideraddingadditionalvariablesbeyondhistoricalvolumeifforecastsare ofinsufficientaccuracy. Wenoteanimportantprincipleofforecastingingeneral:thesetoolsarebestapplied thoughtfully,withconsiderationoftheirstrengthsandlimitations.Forexample,computerscannot beexpectedtoincorporateexternalitiesunavailabletothem,suchaschangesinpatientflowrelated totheavailabilityofbedsatotherhospitalsortoreimbursement. Conclusions Foralltheenthusiasmaboutmachinelearninginmedicine,whichseemstorecurapproximately every30years,29impactonreal-worldclinicalpracticeremainmodest;arecentcommentarynoted the mismatch between promise and concrete accomplishment.30The present study suggests that straightforward application of existing software would allow reliable prediction of a critically importantmetricofhospitaloperationandthatsuchapplicationneednotuseprohibitivelylargedata sets,computationalresources,ortheoperationalcomplexityoffrequentupdates.Whilemore advancedmodelsaredeveloped,time-series–basedpredictionoffersthepossibilityofimproving clinicalplanninginthenearterm. ARTICLEINFORMATION AcceptedforPublication:September3,2018. Published:November2,2018.doi:10.1001/jamanetworkopen.2018.4087 OpenAccess:ThisisanopenaccessarticledistributedunderthetermsoftheCC-BYLicense.©2018McCoyTH Jretal.JAMANetworkOpen. CorrespondingAuthor:ThomasH.McCoyJr,MD,CenterforQuantitativeHealth,DepartmentofPsychiatry, MassachusettsGeneralHospital,HarvardMedicalSchool,185CambridgeSt,SimchesResearchBldg,SixthFloor, Boston,MA02114(thmccoy@partners.org). AuthorAffiliations:CenterforQuantitativeHealth,DepartmentofPsychiatry,MassachusettsGeneralHospital, HarvardMedicalSchool,Boston,Massachusetts. AuthorContributions:DrsMcCoyandPerlishadfullaccesstoallofthedatainthestudyandtakeresponsibility fortheintegrityofthedataandtheaccuracyofthedataanalysis. Conceptanddesign:McCoy,Perlis. Acquisition,analysis,orinterpretationofdata:Allauthors. Draftingofthemanuscript:Allauthors. Criticalrevisionofthemanuscriptforimportantintellectualcontent:Perlis. Statisticalanalysis:McCoy. Administrative,technical,ormaterialsupport:Pellegrini. Supervision:Perlis. ConflictofInterestDisclosures:DrMcCoyreportedreceivinggrantsupportfromTheStanleyCenterattheBroad Institute. Dr Perlis reported receiving grants from the National Human Genome Research Institute and from the National Institute of Mental Health and personal fees for service on scientific advisory boards or consulting to Genomind,PsyTherapeutics,andRIDVentures.Nootherdisclosureswerereported. Disclaimer:DrPerlisisanassociateeditorofJAMANetworkOpen,buthewasnotinvolvedinanyofthedecisions regardingreviewofthemanuscriptoritsacceptance. JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 7/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  8. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume REFERENCES 1. BeltempoM,BlaisR,LacroixG,CabotM,PiedboeufB.Associationofnursingovertime,nursestaffing,andunit occupancywithhealthcare–associatedinfectionsintheNICU.AmJPerinatol.2017;34(10):996-1002.doi:10. 1055/s-0037-1601459 2. GriffithsP,BallJ,DrennanJ,etal.Nursestaffingandpatientoutcomes:strengthsandlimitationsofthe evidencetoinformpolicyandpractice—areviewanddiscussionpaperbasedonevidencereviewedforthe NationalInstituteforHealthandCareExcellenceSafeStaffingguidelinedevelopment.IntJNursStud.2016;63: 213-225.doi:10.1016/j.ijnurstu.2016.03.012 3. KimY,KimS-H,KoY.Effectofnursestaffingvariationandhospitalresourceutilization.NursHealthSci.2016;18 (4):473-480.doi:10.1111/nhs.12294 4. DeRienzoCM,ShawRJ,MeanorP,LadaE,FerrantiJ,TanakaD.Adiscreteeventsimulationtooltosupportand predicthospitalandclinicstaffing.HealthInformaticsJ.2017;23(2):124-133.doi:10.1177/1460458216628314 5. SchreyerKE,MartinR.Theeconomicsofanadmissionsholdingunit.WestJEmergMed.2017;18(4):553-558. doi:10.5811/westjem.2017.4.32740 6. GardnerG,GardnerA,MiddletonS,etal.Mappingworkforceconfigurationandoperationalmodelsin Australianemergencydepartments:anationalsurvey.AustHealthRev.2018;42(3):340-347.doi:10.1071/AH16231 7. TaylorSJ,LethamB.Prophet:forecastingatscale.FacebookResearch.https://research.fb.com/prophet- forecasting-at-scale.PublishedFebruary23,2017.AccessedAugust30,2018. 8. FriedmanJH.Onbias,variance,0/1—loss,andthecurse-of-dimensionality.DataMinKnowlDiscov.1997;1 (1):55-77.doi:10.1023/A:1009778005914 9. GemanS,BienenstockE,DoursatR.Neuralnetworksandthebias/variancedilemma.NeuralComput.1992;4 (1):1-58.doi:10.1162/neco.1992.4.1.1 10. TandbergD,QuallsC.Timeseriesforecastsofemergencydepartmentpatientvolume,lengthofstay,and acuity.AnnEmergMed.1994;23(2):299-306.doi:10.1016/S0196-0644(94)70044-3 11. BergsJ,HeerinckxP,VerelstS.Knowingwhattoexpect,forecastingmonthlyemergencydepartmentvisits: Atime-seriesanalysis.IntEmergNurs.2014;22(2):112-115.doi:10.1016/j.ienj.2013.08.001 12. JonesSS,ThomasA,EvansRS,WelchSJ,HaugPJ,SnowGL.Forecastingdailypatientvolumesinthe emergencydepartment.AcadEmergMed.2008;15(2):159-170.doi:10.1111/j.1553-2712.2007.00032.x 13. LuoL,XuX,LiJ,ShenW.Short-termforecastingofhospitaldischargevolumebasedontimeseriesanalysis.In: 2017IEEE19thInternationalConferenceonE-HealthNetworking,ApplicationsandServices(Healthcom).Dalian, China:IEEE;2017:1-6.doi:10.1109/HealthCom.2017.8210801 14. TingZhu,LiLuo,XinliZhang,YingkangShi,WenwuShen.Time-seriesapproachesforforecastingthenumber of hospital daily discharged inpatients. IEEE J Biomed Health Inform. 2017;21(2):515-526. doi:10.1109/JBHI.2015. 2511820 15. MurphySN,MendisM,HackettK,etal.Architectureoftheopen-sourceclinicalresearchchartfrom InformaticsforIntegratingBiologyandtheBedside.AMIAAnnuSympProc.2007:548-552. 16. MurphySN,WeberG,MendisM,etal.Servingtheenterpriseandbeyondwithinformaticsforintegrating biologyandthebedside(i2b2).JAmMedInformAssoc.2010;17(2):124-130.doi:10.1136/jamia.2009.000893 17. CohenJ,CohenP,WestSG,AikenLS.AppliedMultipleRegression/CorrelationAnalysisfortheBehavioral Sciences.3rded.Mahwah,NH:LawrenceErlbaumAssociatesInc;2002. 18. DeGooijerJG,HyndmanRJ.25yearsoftimeseriesforecasting.IntJForecast.2006;22(3):443-473.doi:10.1016/ j.ijforecast.2006.01.001 19. TaylorSJ,LethamB.Forecastingatscale.PeerJPreprints.https://peerj.com/preprints/3190/.PublishedSeptember 27,2017.AccessedAugust30,2018.doi:10.7287/peerj.preprints.3190v2 20. PerlichC,ProvostF,SimonoffJS.Treeinductionvslogisticregression:alearning-curveanalysis.JMachLearn Res.2003;4:211-255.doi:10.1162/153244304322972694 21. BergmeirC,HyndmanRJ,KooB.Anoteonthevalidityofcross-validationforevaluatingautoregressivetime seriesprediction.ComputStatDataAnal.2018;120:70-83.doi:10.1016/j.csda.2017.11.003 22. KimK,LeeC,O’LearyKJ,RosenauerS,MehrotraS.PredictingPatientVolumesinHospitalMedicine: AComparativeStudyofDifferentTimeSeriesForecastingMethods.Evanston,IL:NorthwesternUniversity; 2014:1-13. 23. GershengornHB,HarrisonDA,GarlandA,WilcoxME,RowanKM,WunschH.Associationofintensivecareunit patient-to-intensivistratioswithhospitalmortality.JAMAInternMed.2017;177(3):388-396.doi:10.1001/ jamainternmed.2016.8457 JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 8/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

  9. JAMANetworkOpen | StatisticsandResearchMethods Time-SeriesMachineLearningMethodsforForecastingHospitalDischargeVolume 24. KeswaniA,BeckC,MeierKM,FieldsA,BronsonMJ,MouchaCS.Dayofsurgeryandsurgicalstarttimeaffect hospitallengthofstayaftertotalhiparthroplasty.JArthroplasty.2016;31(11):2426-2431.doi:10.1016/j.arth. 2016.04.013 25. deCordovaPB,JohansenML,MartinezME,CimiottiJP.Emergencydepartmentweekendpresentationand mortalityinpatientswithacutemyocardialinfarction.NursRes.2017;66(1):20-27.doi:10.1097/NNR. 0000000000000196 26. HindsN,BorahA,YooEJ.Outcomesofnighttimerefusalofadmissiontotheintensivecareunit:theroleofthe intensivistintriage.JCritCare.2017;39:214-219.doi:10.1016/j.jcrc.2016.12.024 27. MetcalfeD,PerryDC,BouamraO,etal.Istherea“weekendeffect”inmajortrauma?EmergMedJ.2016;33 (12):836-842.doi:10.1136/emermed-2016-206049 28. HamaguchiS,KinugawaS,Tsuchihashi-MakayaM,GotoD,TsutsuiH.Weekendversusweekdayhospital admissionandoutcomesduringhospitalizationforpatientsduetoworseningheartfailure:areportfromJapanese CardiacRegistryofHeartFailureinCardiology(JCARE-CARD).HeartVessels.2014;29(3):328-335.doi:10.1007/ s00380-013-0359-5 29. SchwartzWB,PatilRS,SzolovitsP.Artificialintelligenceinmedicine:wheredowestand?NEnglJMed.1987; 316(11):685-688.doi:10.1056/NEJM198703123161109 30. ChenJH,AschSM.Machinelearningandpredictioninmedicine—beyondthepeakofinflatedexpectations. NEnglJMed.2017;376(26):2507-2509.doi:10.1056/NEJMp1702071 SUPPLEMENT. eFigure1.LearningCurveforEachofSixTargetYearsShowingImpactofAdditionalYearsofTrainingDataonMean ForecastAccuracyandMinandMaxForecastAccuracy eFigure2.PredictionCalibrationbyHospitalSitefortheProphetModelUsingaFullOne-YearForecastingHorizon vsaMonthlyForecastingHorizon eTable1.ModelCalibrationsforFullYearForecastHorizonvsIterativeRefittingWithaMonthlyForecastHorizon (FullYearIsReproducedFromTable1forEaseofComparison) eTable 2. Number of Days Over Forecast Year by Error Sizes for Prophet Model Using Full Year Forecast Horizon (ReproducedFromTable2)vsMonthlyStepwiseForecastHorizon eTable3.AbsoluteTotalandTotalCumulativeErrorOvertheForecastYear(Denominator365Days)Comparing FullYearForecastHorizonandIterativeMonthlyForecastHorizon JAMANetworkOpen.2018;1(7):e184087.doi:10.1001/jamanetworkopen.2018.4087 November 2,2018 9/9 Downloaded From: https://jamanetwork.com/ on 12/06/2021

More Related