Learning an Action-Value Function

• Assigns value to action-state pairs, not just states
• These values are called Q-values $Q(i,a)$
  • $Q(i,a)$ = value of doing action $a$ in state $i$
  • Do not need transition model
  • Learned directly from explicit reward feedback
  • Provide condition action rules
• Relation between utility values and Q values
• $U(i) = \max_a Q(i,a)$
• TD-based Q learning
  • When transition from state $i$ to state $j$
  • $Q(i,a) = Q(i,a) + \alpha * (R(i) + max_{a'} Q(j,a') - Q(i,a))$