Inhibitors of the enzyme farnesyltransferase show potential as novel anticancer agents. There are many known inhibitors, but efforts to build predictive SAR models have been hampered by the structural diversity and flexibility of inhibitors. We have undertaken for the first time a QSAR study of the potency and selectivity of a large, diverse data set of farnesyltransferase inhibitors. We used novel molecular descriptors based on binned atomic properties and invariants of molecular matrices and a robust, nonlinear QSAR mapping paradigm, the Bayesian regularized neural network. We have built robust QSAR models of farnesyltransferase inhibition, geranylgeranyltransferase inhibition, and in vivo data. We have derived a novel selectivity index that allows us to model potency and selectivity simultaneously and have built robust QSAR models using this index that have the potential to discover new potent and selective inhibitors.