It isn't a great mystery what is meant when someone calls you "mad as a hatter".
What is more of a mystery is the origin of this insult. It is widely accepted that the phrase has its origin from the fact that as far back at the 18th century, mercury was used in the production of hats. It is well established that prolonged exposure to mercury causes significant damage to the nervous system resulting in tremors and twitching.