Brady also had detractors who accused him of cheating