Adaptive Stochastic Control for the Smart grid

1. Adaptive Stochastic Control for the Smart grid Qinghua Shen Smart grid meeting

2. Outline • Introduction to the control of smart grid adaptive stochastic control, smart grid • Adaptive Stochastic Control basis of stochastic system, policy search and approximation, convergence • Example: distributed generation despatch with storage ADP for resource allocation, value function approximations • Challenges

3. Introduction to the control of smart grid • Control of the smart grid • goal for control: instantly, corrective and dynamically • Self-healing: auto repair or removal of potentially faulty equipment • Flexible: rapid and safe interconnection of distributed generation and storage • Predictive: statistics, machine learning, predictive models • interactive: appropriate information is provided transparently in near real time • Optimal : operators and customers efficiently and economically • Secure : cyber- and physical-security

4. Introduction to the control of smart grid • Major Components

5. Outline • Introduction to the control of smart grid adaptive stochastic control, smart grid • Adaptive Stochastic Control basis of stochastic system, policy search and approximation, convergence • Example: distributed generation despatch with storage ADP for resource allocation, value function approximations • Challenges

6. Adaptive Stochastic Control • Stochastic system • State variables • physical state: energy amount, status of a generator • information state: current and historical demand, price and weather • belief state: probability distributions • The decisions • whether or charge/discharge, use backup • The exogenous Information • all the dimensions of uncertainty • The Transition Function • given the state, decisions and exogenous information, determines next state • The Objective Function • metrics that governs how we make those decisions and evaluate the performance of policies of the controller designs

7. Adaptive Stochastic Control • Policies • maps the information in state S to a decision x. • , which is the state variables, capturing energy resources Rt, exogenous information pt, and belief state Kt. • The problem • is known variously as the value of the a policy or the cost to go function. • can be a cost function if we minimize, or contribution function if we maximize. • Cost include generating electricity, purchasing fuel, losses due to energy conversion, cost of repair, and penalties for curtailing loads • policy for what • includes whether to charge/discharge, when to run a distributed generator, how much energy draw from grid for every customer in every networks and the utility

8. Adaptive Stochastic Control • Design a robust policy: four classes • Myopic Policies • minimize next-period cost without decisions for future( special structure good) • Look-ahead Policies • Optimize over some time horizon using a forecast of the possible variability of exogenous events such as weather. Forecast can either be deterministic forecasts or stochastic forecasts • Policy Function Approximations • Functions return an action given a state, without solving any form of optimization, including: rule-based lookup table; Parameterized rules(threshold hold); statistical functions • Policy based on Value Function Approximations • Optimal policy obtained from HJB equation, to avoid curses of dimensionality • a) Approximate to eliminate the expectation; b) replace the value function with a computationally tractable approximation; c) solve the resulting deterministic maximization problem using a commercial solver

9. Adaptive Stochastic Control • ADP and the Post–Decision State • Value function approximation • when structure of a policy is not obvious, estimates the value of being in a state • When x is a vector, solve the maximization problem is problematic(expectation hard to compute exactly)---refer to stochastic search • Post-Decision state • Post decision state determined through current state and action

10. Adaptive Stochastic Control • Design policy • look up tables • Parametric models • With this strategy, we face the challenge of first identifying the basis functions, and then tuning the parameters • Nonparametric models • handle high-dimensional, asymptotically unbiased • • Kernel regression; • • Support Vector regression; • • Neural networks; • • Dirichlet process mixtures.

11. Adaptive Stochastic Control • Policy search • Direct policy search • Depend on Monte Carlo sampling ----stochastic search • Methods: sequential kriging • Using the knowledge gradient • Applied when the policy structure is apparent • Bellman residual minimization for value function approximations • This is the most widely used strategy for optimizing policies, and encompasses a variety of algorithmic approaches that include approximate value iteration (including temporal difference learning) and approximate policy iteration

12. ASC for distributed generation despatch • Approximate Dynamic Programming for resource allocation • Resource allocation • how much energy to store in a battery, whether a diesel generator should be turned on, and whether a mobile storage device (and/or generator) should be moved to a congested location. • A general model • Rta is the number of resources with attribute vector a • xtad is the number of resrouces we act on with a decision of type d. • a decision d can be (-1,0,1) to discharge, hold, or recharge a battergy • (0,1) to turn a distributed generator off or on

13. Adaptive Stochastic Control • Value Function Approximations for resource allocation • Approximate the value function • resource allocation utility function: concavity property • Approximate value function by the post decision resource vector Separable piece-wise linear function • Estimate piecewise linear concave functions by iteratively stepping forward through time and updating value functions

14. Adaptive Stochastic Control • Experimental work • evaluate the results • Resource determine the quality of the resulting policy is a major challenge • fit the value functions for a deterministic problem, and compare the resulting solution to the optimal solution for the deterministic problem, obtained by using a commercial solver • limited by the size

15. Challenges • Convergence • Only some structure can be proofed to be convergence with approximation • Concavity is an important category • For smart grid • Beneficial to both utility and end users– enough incentive • The track of key performance metrics