Brettell was born in Rochester, New York.  When he was eight his family moved to Denver, Colorado, and spent his formative years there. He entered Yale University, intent on studying molecular biophysics until hearing professor George Kubler speak, changing his mind to study art history. Brettell received his Bachelor's and Master's, degrees from Yale. At Yale, Brettell met Zoe Caroline Bieler (b. 1950), a graduate student in cultural anthropology, who he married in 1973. He mounted his first art installation, a photography exhibition at Yale, the same year.