Anyone ever see people panning for gold in your streams?

I ran into two separate guys panning for gold today. I live in new england and I didnt think there was many places for that here. It's just weird that it happened twice.