Google have a number of penalties and most are algorithmic. The 3 different penalties are listed below:
Panda: This is all about on page, content and duplicate content. When you have this penalty you will see your website slowly loose its ranking
Penguin: This is all about links and bad neighbourhoods. This will affect your site with massive drops in rankings all of a sudden.
Manual: This is where a human will asses your website and determine if you are manipulating the algaristhms or not following Googles guidlines. This will show up in the Google webmaster tools.