Introducing a new approach to multiagent reinforcement learning and distributed artificial intelligence, this guide shows how classical game theory can be used to compose basic learning units. This approach to creating agents has the advantage of leading to powerful, yet intuitively simple, algorithms that can be analyzed. The setup is demonstrated here in a number of different settings, with a detailed analysis of agent learning behaviors provided for each. A review of required background materials from game theory and reinforcement learning is also provided, along with an overview of related multiagent learning methods.