For Smith, the dangers that natural liberty faces are not a result of the system of free markets itself, but of mankind’s flawed human nature, particularly the desire of those he called "merchants and manufacturers" (among others) to "rig the system."
Source