DQN

DQN(Deep Q-Network)은 2015년 DeepMind가 개발한 심층 강화 학습 알고리즘으로, Q-러닝과 심층 신경망을 결합하여 Atari 2600 게임에서 인간 수준의 성능을 달성했다. 훈련 안정화를 위해 경험 재생과 타깃 네트워크를 도입했으며, 이는 AI 연구의 돌파구를 마련했다.

Deep Q-Network (DQN) is a deep reinforcement learning algorithm developed by researchers at DeepMind (now part of Google DeepMind) and first published in a 2015 paper in the journal Nature. It combines the classic Q-learning algorithm with deep neural networks to approximate the action-value function, enabling an agent to learn directly from high-dimensional sensory inputs, such as raw pixel frames.

Architecture and Training

DQN uses a convolutional neural network (CNN) to process raw game frames. The input is a stack of the most recent grayscale frames, resized to 84x84 pixels. The network architecture typically consists of three convolutional layers followed by two fully connected layers, with the final output layer having one neuron per possible action. The network is trained to minimize the mean squared error between the predicted Q-values and the target values, computed using the Bellman equation.

Training proceeds in an epsilon-greedy manner, where the agent explores the environment with probability epsilon and exploits the current policy otherwise. The epsilon value is annealed over time, starting from 1.0 and decreasing to 0.1.

Key Innovations

Experience Replay

DQN introduces the concept of experience replay, where the agent stores its experiences (state, action, reward, next state) in a replay buffer. During training, the agent samples mini-batches randomly from this buffer, breaking the temporal correlations in the data. This stabilizes training and reduces the risk of feedback loops.

Target Network

A periodically updated copy of the main network, called the target network, is used to compute the target values. This prevents the network from chasing a moving target, which can cause instability. The target network's parameters are updated every 10,000 steps.

Performance on Atari Games

DQN was evaluated on 49 Atari 2600 games from the Arcade Learning Environment. The algorithm achieved superhuman performance on a range of games, including Breakout, Enduro, and Pong, and demonstrated that a single architecture could learn to play multiple games without any task-specific information.

Subsequent Developments

DQN's success sparked a wave of research in deep reinforcement learning. Subsequent extensions include:

Double DQN, which separates the action selection and value estimation to reduce overestimation bias
Dueling DQN, which separates the state value and advantage functions
Prioritized Experience Replay, which samples important transitions more frequently
Rainbow, which combines several of these enhancements into a single agent

These developments have led to significant improvements in sample efficiency and final performance across various benchmarks.

Impact and Legacy

DQN is widely regarded as a foundational contribution to modern artificial intelligence. It demonstrated that deep learning could be effectively combined with reinforcement learning to solve complex sequential decision-making problems. The algorithm's success on Atari games also established the Arcade Learning Environment as a benchmark for evaluating reinforcement learning agents, a practice that continues in subsequent research.

DQN's influence extends into the development of generative AI and large-scale models, as the principles of stable training and function approximation are shared across these domains. Its legacy is evident in the ongoing work at institutions like Google DeepMind, OpenAI, and Anthropic, where reinforcement learning remains a key area of exploration.