Cognition is dynamic and involves both the maintenance of and transitions between neurocognitive states. While recent research has identified some of the neural systems involved in sustaining task states, it is less well understood how intrinsic influences on cognition emerge over time. The current study uses fMRI and Multi-Dimensional Experience Sampling (MDES) to chart how cognition changes over time from moments in time when external attention was established. We found that the passage of time was associated with brain regions associated with external attention decreasing in activity over time. Comparing this pattern of activity to defined functional hierarchies of brain organization, we found that it could be best understood as movement away from systems involved in task performance. Moments where the participants described their thoughts as off-task showed a significant similarity to the task-negative end of the same hierarchy. Finally, the greater the similarity of a participant's neural dynamics to this hierarchy the faster their rate of increasing off-task thought over time. These findings suggest topographical changes in neural processing that emerge over time and those seen during off-task thought can both be understood as a common shift away from neural motifs seen during complex task performance.