A quantitative determination method for the diagnosis of hyperlipidemia was developed using Fourier transform infrared (FTIR) spectroscopy. Random forest (RF) was demonstrated as a potential multivariate algorithm for the FTIR analysis of low-density lipoprotein cholesterol (LDL-C) and tri-glycerides (TG) in human serum samples. The informative wavebands for LDL-C and TG were selected based on the Gini importance. The selected wavebands were mainly within the fingerprint region. The RF modeling results were better than those derived using PLS in validation process, because the chance for over-fitting was possibly eliminated in RF algorithm. ARF also demonstrated favorable results in the test process. The prospective model exhibited a higher than 90% true prediction in negative/positive properties for male and female samples. These clinical statistical results indicated the optimization of RF algorithm performed accurately in the FTIR determination of LDL-C and TG. RF is evaluated as a promising tool for diagnosing and controlling hyperlipidemia in populations. The parameter optimization methodology is useful in the improving model accuracy using FTIR spectroscopic technology.