A hatter is a person who makes hats. Hatters used to use mercury in making hats. Mercury is a very dangerous substance. It tended to drive the hatters crazy. Hence, if you are mad as a hatter, you are really crazy. Lewis Carroll makes an indirect reference to this adage is his novel Alice in Wonderland. In the novel he has a Hatter character who is mad and who attends a Mad Tea Party.