Control processes allow us to constrain the retrieval of semantic information from long-term memory so that it is appropriate for the task or context. Control demands are influenced by the strength of the target information itself and by the circumstances in which it is retrieved, with more control needed when relatively weak aspects of knowledge are required and after the sustained retrieval of related concepts. To investigate the neurocognitive basis of individual differences in these aspects of semantic control, we used resting-state fMRI to characterise the intrinsic connectivity of left ventrolateral prefrontal cortex (VLPFC), implicated in controlled retrieval, and examined associations on a paced serial semantic task, in which participants were asked to detect category members among distractors. This task manipulated both the strength of target associations and the requirement to sustain retrieval within a narrow semantic category over time. We found that individuals with stronger connectivity between VLPFC and medial prefrontal cortex within the default mode network (DMN) showed better retrieval of strong associations (which are thought to be recalled more automatically). Stronger connectivity between the same VLPFC seed and another DMN region in medial parietal cortex was associated with larger declines in retrieval over the course of the category. In contrast, participants with stronger connectivity between VLPFC and cognitive control regions within the ventral attention network (VAN) had better controlled retrieval of weak associations and were better able to sustain their comprehension throughout the category. These effects overlapped in left insular cortex within the VAN, indicating that a common pattern of connectivity is associated with different aspects of controlled semantic retrieval induced by both the structure of long-term knowledge and the sustained retrieval of related information.