The annoying people in your life don't just ruin your day. They actually take years off your life, a new study says.