Inhibitors of the Hsp90 molecular chaperone are showing promise as anti-cancer agents. Here we describe a series of 4-aryl-5-cyanopyrrolo[2,3-d]pyrimidine ATP competitive Hsp90 inhibitors that were identified following structure-driven optimization of purine hits revealed by NMR based screening of a proprietary fragment library. Ligand-Hsp90 X-ray structures combined with molecular modeling led to the rational displacement of a conserved water molecule leading to enhanced affinity for Hsp90 as measured by fluorescence polarization, isothermal titration calorimetry and surface plasmon resonance assays. This displacement was achieved with a nitrile group, presenting an example of efficient gain in binding affinity with minimal increase in molecular weight. Some compounds in this chemical series inhibit the proliferation of human cancer cell lines in vitro and cause depletion of oncogenic Hsp90 client proteins and concomitant elevation of the co-chaperone Hsp70. In addition, one compound was demonstrated to be orally bioavailable in the mouse. This work demonstrates the power of structure-based design for the rapid evolution of potent Hsp90 inhibitors and the importance of considering conserved water molecules in drug design.