David Smith's 1851 publication of The Covenanter of the Reformed Presbyterian Church describes the Jesuits of Baden as suffering "a veritable pogonophobia at the sight of a democratic chin."
The term is generally meant to be taken in a jocular vein. In the 1920s, psychologist John B. Watson was able to condition this fear in a young boy by means of classical conditioning methods.
In August 2013 Christopher Oldstone-Moore, history lecturer at Wright State University in Ohio, and author of The Beard Movement in Victorian Britain commented, "Facial hair for the past century has been thought to reflect a suspicious streak of individuality and defiance... Politicians, public servants and businessmen – and apparently journalists – risk their reputations if they abandon the razor."
Some relationship to "beardism" — discrimination based on facial hair — is claimed, and a difference in cultures is noted. Some association with claims of unhygienic beards (e.g., among homeless men) and fashion preferences of women. That various religious groups treat beards more or less reverently is also a factor, for example in Judaism and in Islam. Similarly, some groups require beards and forbid shaving, which has an effect on that society's norms and perceptions.
I say: "Let your beard fly!!"