I am reading several papers on methodology used for the evaluation of interactive artworks (this is where my thesis has gone, I will elaborate more soon but not now). The works that I am reading have been fairly good with recruiting users who are not in the interactive arts or the HCI field, and are of varying ages and occupations.
It’s common to see HCI studies (and this probably happens in other fields as well) where subjects are recruited from a University’s student body. Many times, those students are even in the same department and sometimes they also study HCI. I’ve been a user several times. I’ve also had fellow HCI students participate in my studies and I can see that happening again in the future.
So, has anybody studied the biases involved when your user is an HCI researcher him or herself? For example, if I am led through some interaction task, I often find that I’m thinking less about the interface and what I should be doing and more about how they have structured their study and what tools are being used for data collection. During interviews I feel I have more to say about my experience being a subject than my experience with the interface or application in question. If you know of anyone who has studied this – I would be interested in reading that.