To study the quality of stellar spectra of the Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) and the correctness of the corresponding stellar parameters derived by the LASP (LAMOST Stellar Parameter Pipeline), the outlier analysis method is applied to the archived AFGK stars in the fifth data release (DR5) of LAMOST. The outlier factor is defined in order to sort more than 3 million stellar spectra selected from the DR5 Stellar Parameter catalog. We propose an improved Local Outlier Factor (LOF) method based on Principal Component Analysis and Monte Carlo to enable the computation of the LOF rankings for randomly picked sub-samples that are computed in parallel by multiple computers, and finally to obtain the outlier ranking of each spectrum in the entire dataset. Totally 3,627 most outlier ranked spectra, around one-thousandth of all spectra, are selected and clustered into 10 groups, and the parameter density distribution of them conforms to the parameter distribution of LAMOST DR5, which suggests that in the whole parameter space the probability of bad spectra is uniformly distributed. By cross-matching the 3,627 spectra with APOGEE, we obtain 122 common ones. The published parameters calculated from LASP agree with APOGEE for the 122 spectra although there are bad pixels or bad flux calibrations in them. On the other hand, some outlier spectra show strong nebular contamination warning the corresponding parameters should be carefully used. A catalog and a spectral atlas of all the 3,627 outliers can be found at the link http://paperdata.china-vo.org/LY_paper/dr5Outlier/dr5Outlier_resource.zip.