It happens to me once or twice a week, and always on my favorite maps, it seems. I'm always on the phone and in a group with my girlfriend while I play, and she says she sees me doing things ingame up until it removes me for "inactivity". So I don't even know