Abstract: Medical imaging and image-based modeling have made considerable progress in recent years in identifying atherosclerotic plaque morphological and mechanical risk factors which may be used in developing improved patient screening strategies. However, a clear understanding is needed about what we have achieved and what is really needed to translate research to actual clinical practices and bring benefits to public health. Histopathological analysis has served as the gold standard for validation and verification of vulnerable plaque studies. Lack of in vivo data and clinical events to serve as gold standard to validate model predictions is a severe limitation. A review of the key steps and findings of our group in image-based models for human carotid and coronary plaques will be provided, including image and data acquisition, model construction, data analysis and identification of mechanisms and risk factors for plaque progression and rupture. Predictive methods to quantify prediction accuracy of risk factors leading to prediction of acute clinical events will be presented.