Which RL Environment Should You Start With?

October 2026

The most common beginner mistake in reinforcement learning isn't picking the wrong algorithm — it's picking the wrong environment. Start on full StarCraft and you'll wait hours to learn that your reward function is broken. Start on CartPole and you'll know in ninety seconds. Here's the ladder we'd recommend, in order.

Step 1

CartPole-v1 — learn the loop

The canonical first environment: balance a pole on a cart with two discrete actions. Episodes last seconds, and Gymnasium's 200-point solve threshold tells you unambiguously whether your agent learned anything. Perfect for tabular methods and your first DQN.

Step 2

LunarLander-v3 — learn reward shaping

Land the module between the flags. Shaped rewards, a real 200-point solve threshold, and just enough complexity to punish sloppy exploration. The step where most people first feel what "credit assignment" means.

Step 3

MiniGrid — learn to debug fast

Minimalist gridworlds where experiments finish in minutes, not hours. This is where you develop the most valuable RL skill of all: iterating quickly. Partial observability also introduces you to memory and recurrence gently.

Step 4

Atari (Pong, then Breakout) — learn pixels

Your first visual environments. Start with Pong — it's nearly trivial for modern methods — then Breakout, where the agent must discover long-horizon strategies like tunneling. This is the DQN rite of passage.

Step 5

MuJoCo HalfCheetah — learn continuous control

Your first continuous-action environment: teach a cheetah to run. Everything changes here — policy gradients, action noise, and reward scales all behave differently. Budget GPU time; this is where compute starts to matter.

Rules of thumb

  1. Discrete before continuous, fast before slow. Every hour your environment wastes is an hour you're not learning.
  2. One new difficulty at a time. Don't learn pixels and continuous control simultaneously — isolate what you're studying.
  3. Use maintained suites. Gymnasium, MiniGrid, and the Arcade Learning Environment are actively maintained by the Farama Foundation. You'll get working installs instead of dependency archaeology.
What to avoid (for now): full StarCraft II, NetHack, Isaac Lab, and Google Research Football. They're magnificent and they will eat your month. Come back when the ladder above feels easy.

Want a personalized answer? Our env picker matches environments to your goal, experience, and compute in four questions — or browse all 51 in the directory.