I would like to share some experience about these processes using the agencies. There are many firms specializing in these job. For example, there is WatchLab in Chicago. Typically they are located in the prime central place with convenience access and has luxurious interior. For it is basically real-estate business like hotel business, location is very important. As you can imagine, the reason is that many people has to visit this place for the first time, and the time & cost to visit this place should be optimal. So it makes sense to have an office at the down-town prime space.
So you know the price, how can make it valuable process? In my opinion recruiting( and analysis for next time) is very important. For example, if you are conducting UX evaluation about the smartphone. You would like to include two groups of people, say novice users and advanced users. It will be very tempting to say I would like to recruit novice and advanced users in the meeting with the professional research recruiter. However it is very subjective what constitutes novice and advanced user group. So the busy research recruiter may have an entirely different point of view about the each group, and prepares not adequate samples for the each groups. So it is designer’s role to prepare more numeric and quantitative guideline for the recruiting, which characteristic each group has. Say novice group has less than one year of smartphone usage experience, and uses less than 5 apps daily, with the age group of 40~50, with the equal distribution of the male/female, and does not use social media apps using their smartphone,… and so on. And during the recruiting process, involve with the recruiter actively during the process, and requesting participant survey for confirmation is good idea.
Also in my opinion, determining the right number of participants are not so important than it sounds. More often than not, it is determined by the budget and resources to conduct the research. Remember more energy is required in the analysis of the result. So it is very practical that having a number of the users, designers can personally identify and “View” the entire session (which can be really boring after say 3 person), find the design improvement point, and do an analysis over the session. I would like to also remark that it is physically demanding task. Part of the reason is that it may start after say 6pm due to the working participant.
Finally I would like to briefly mention 2 services, which can help during the process.
The first is eye tracking. For the software review, knowing the pattern of the user’s eye movement can help detect the problematic area, which maybe not available without the use. It shows the inner working logic of the user mind by showing which area got attention and didn’t. It is possible to track eyes for the desktop app and mobile app. Also there is a rental firm of equipment and they also provide total service including the tech personel during the process and analysis report.
The second service is the streaming service. Sometimes designers or stakeholders are far-away. By streaming data real-time and providing video archive for the later reference, it is possible to participate during the process. It also provides stereo channel where left side can be original sound, and right side can be real time translation.