Curiosity-Driven Exploration via Temporal Contrastive Learning

Abstract

Exploration remains a key challenge in reinforcement learning (RL), especially in long-horizon tasks and environments with high-dimensional observations. A common strategy for effective exploration is to promote state coverage or novelty, w hich often involves estimating the agent's state visitation distribution. In this paper, we propose Curiosity-Driven Exploration via Temporal Contrastive Learning Learning (C-TeC), an exploration method based on temporal contrastive learning that rewards agents for reaching states with unexpected futures. This incentivizes uncovering meaningful less-visited states. \methodName is simple and does not require explicit density or uncertainty estimation, while learning representations aligned with the RL objective. It consistently outperforms standard baselines in complex mazes using different embodiments (Ant and Humanoid) and robotic manipulation tasks, while also yielding more diverse behaviors in Craftax without requiring task-specific information.

Overview

A state-action pair $(s_t, a_t)$ is passed through the encoder $\phi$, and the corresponding future state is passed through the encoder $\psi$. The outputs of the encoders are used to compute the similarity score and the intrinsic reward.

Reward Visualization

Evolution of the C-TeC reward during training. This figure shows how the intrinsic reward changes over the course of training based on future state visitation. The black circle in the lower-left corner represents the starting state. Early in training (3M steps), higher rewards are assigned to nearby states. As training progresses, the agent explores farther, and the reward increases for more distant regions. All reward values are normalized for visualization.