Policy gradient

Given a policy $\pi$ parameterized by $\theta$ having performance $\rho$, policy gradient methods perform iterative updates of the parameters by:

$\Delta \theta \propto \frac{\partial \rho}{\partial \theta}$

TODO:

• Sutton et al. (1999): use a value-approximation-function to reduce variance

Direct gradient-based methods

(The name comes from Baxter & Barlett (2000).)

These methods estimate the gradient of expected reward without the help of value function, action-value function, TD, etc. The first representative is REINFORCE (Williams, 1992) which "learns much more slowly than RL methods using value functions and has received relatively little attention" (Sutton et al., 1999).

TODO:

• Infinite horizon: Bartlett & Baxter (2002)
• VAPS (value and policy search) (Baird & Moore, 1999): generalize policy search and value-based algorithms, guaranteed to converge, work with episodic tasks.
• Cao & Wan (1998)

Actor-critic

Konda & Tsitsiklis (1999)

Parameter space exploring

Rückstieß et al. (2010) reviews some techniques that perturb parameters instead of actions. The main idea was to replace parameters by a distribution (says, Gausian) of parameters. Different from REINFORCE, the policy is deterministic given parameters. The paper report experiments in which parameter-based exploration is better than action-based. However, they only consider continuous action cases. I think the approach is unlikely to work for discrete action cases because a change if parameters may make no difference in the trajectory or changing the trajectory in an unpredictable way.