Why are Mango's considered Drugs?

How do you know it was the mango that did it and not something you ate/drank earlier? Or that the mood change occurred coincidentally?