Autonomously driving is really hard. How do we integrate planning + learning in a close-loop style. We’ll start from the current belief, and construct a tree of all reachable belief state.
Recall DESPOT.
Approach
- learning (b): a neural network which maps the driving history into a policy and value
- planning (a): we will use the neural network’s derived policy and value to run MCTS
- execution (e): execute the actions in a simulator
The data which is obtained using the simulator is used to train the neural network.
learning
The learning component is a supervised policy where a CNN takes a situation and map
planning
its a AR-DESPOT. We select actions by:
\begin{equation} a^{*} = \arg\max_{a \in A} \left\{u(b,a) + c \pi_{\theta}(a|x_{b}) \sqrt{ \frac{N(b)}{N(b,a)+1}}\right\} \end{equation}
where \(\pi_{\theta}\) is our policy network.
Every time we encounter a new node, use the learned value function as a lower bound.
Needed less depth in the DESPOT than using it naively.