230 likes | 397 Views
Modeling Long Term Care and Supportive Housing. Marisela Mainegra Hing Telfer School of Management University of Ottawa. Canadian Operational Research Society, May 18, 2011. Outline. Long Term Care and Supportive Housing Queueing Models Dynamic Programming Model
E N D
Modeling Long Term Care and Supportive Housing Marisela Mainegra Hing Telfer School of Management University of Ottawa Canadian Operational Research Society, May 18, 2011
Outline • Long Term Care and Supportive Housing • Queueing Models • Dynamic Programming Model • Approximate Dynamic Programming
LTC problem λRC λC Community LTC μLTC , CLTC λH Hospital λRH Goal: Hospital level below a given threshold Community waiting times below 90 days
LTC previous results • MDP model determined a threshold policy for the Hospital but it did not take into account community demands • Simulation Model determined that current capacity is insufficient to achieve the goal
λRC λC Community λC-LTC λLTC LTC λH Hospital μLTC , CLTC λH-LTC λRH H_renege μRH, Queueing Model Station LTC: M/M/CLTC Station H_renege: M/M/∞
steady state: <1 Queueing Model Station LTC: M/M/CLTC The probability that no patients are in the system: The average number of patients in the waiting line: The average time a client spends in the waiting line: The number of patients from the Hospital that are in the queue for LTC (LqH-LTC).
Queueing Model Station H_renege: M/M/∞ • The average number of patients in the system is
Queueing Model Data analysis • Data on all hospital demand arriving to the CCAC from April 1st, 2006 to May 15th, 2009. • ρLTC = 1.6269 for current capacity CLTC= 4530 • To have ρLTC < 1 we need CLTC> 7370.08, 2841 (62.71%) more beds than the current capacity. • With CLTC > 7370 we apply the formulas. • Given a threshold T for the hospital patients and the number LqLTC of total patients waiting to go to LTC, what we want is to determine the capacity CLTC in LTC such as:
Queueing Model Results • 19 iterations of capacity values • Goal achieved with capacity 7389, the average waiting time is 31 days and the average amount of Hospital patients waiting in the queue is 130 ( T=134) . • This required capacity is 2859 (63.1%) more than the current capacity.
λH Hospital λRH λH-SH λH-LTC μRH, LTC λSH-LTC SH μLTC , CLTC μSH , CSH λC-LTC λC-SH H_renege Community λC λRC Queueing Model with SH
Queueing Model with SHResults • Required capacity in LTC is 6835, 2305 (50.883%) more beds than the current capacity (4530). • Required capacity in SH is 1169. • With capacity values at LTC: 6835 and at SH: 1169 there are 133.9943 (T= 134) Hospital Patients waiting for care (for LTC: 110.3546, reneging: 22.7475, for SH: 0.89229), and Community Patients wait for care in average (days) at LTC: 34.8799, and at SH: 3.2433.
Semi-MDP Model S = {(DH_LTC, DH_SH, DC_LTC, DC_SH, DSH_LTC, CLTC, CSH, p) } A = {0,..,max(TCLTC,TCSH)} d(s,a) = Pr(s,a,s’) = r(s,a) = State space: Action space: Transition time: Transition probabilities: Immediate reward: Optimal Criterion: Total expected discounted reward
Goal find π: S A that maximizes the state-action value function γ: discount factor Bellman: there exists Q* optimal: Q* =maxQ(s,a) and the optimal policy π* Approximate Dynamic programming
Reinforcement action state Reinforcement Learning
RL: environment ENVIROMMENT action state transition probabilities reward function next state, immediate reward
RL: Agent Knowledge: Q(s,a) exploratory Learning: update Q-values state action Behavior reward Knowledge representation (FA) Learning method • Backup table • Neural network • ... • Watkins QL • Sarsa () • ...
QL: parameters • θ: number of hidden neurons. • T: number of iterations of the learning process. • 0: initial value of the learning rate. • 0: initial value of the exploration rate. • Learning-rate decreasing function. • Exploration-rate decreasing function.
explorationvs/ exploitation Learning and exploration rates QL: algorithm (T, θ T
QL: tuning parameters(observed regularities) • (θ, )-scheme: T= 104, 0= 10-3, 0 =1, T= 103, T=v103 , v[1,.. ]. PR(θ, ): best performance with (θ, )-scheme • PR(θ, ) monotically increase respect until certain value (θ) • PR(θ, ) monotically increase respect θ until certain value θ() • (θ) and θ() depend on the problem instance
QL: tuning parameters(methodology: learning schedule given PRHeu) • ∆θ =50, θ =0, PRθ =0,=0, vbest=1, • While PRθ <PRheu or no-stop • θ = θ + ∆θ, PRbest=0 • While PRbest ≥ PR • = +1, T= 104, T= 103 • PR=PRbest, • For v= vbest to • T=v103 • PR[v]=Q-Learning(T, θ, 10-3, 1, T, T) • [PRbest,vbest]=max(PR) • PRθ= PR
Discussion • For given capacities solve the SMDP with QL • Model other LTC complexities: • different facilities and room accommodations, • client choice and • level of care
Thank you for your attention • Questions?