The value of initiating a pursuit in temporal decision-making.

Abstract

Reward-rate maximization is a prominent normative principle in behavioral ecology, neuroscience, economics, and AI. Here, we identify, compare, and analyze equations to maximize reward rate when assessing whether to initiate a pursuit. In deriving expressions for the value of a pursuit, we show that time's cost consists of both apportionment and opportunity cost. Reformulating value as a discounting function, we show precisely how a reward-rate-optimal agent's discounting function (1) combines hyperbolic and linear components reflecting apportionment and opportunity costs, and (2) is dependent not only on the considered pursuit's properties but also on time spent and rewards obtained outside the pursuit. This analysis reveals how purported signs of suboptimal behavior (hyperbolic discounting, and the Delay, Magnitude, and Sign effects) are in fact consistent with reward-rate maximization. To better account for observed decision-making errors in humans and animals, we then analyze the impact of misestimating reward-rate-maximizing parameters and find that suboptimal decisions likely stem from errors in assessing time's apportionment-specifically, underweighting time spent outside versus inside a pursuit-which we term the 'Malapportionment Hypothesis'. This understanding of the true pattern of temporal decision-making errors is essential to deducing the learning algorithms and representational architectures actually used by humans and animals.