Chris Parnin

Here's the study: http://chrisparnin.me/pdf/stress_FSE_20.pdf They asked students to find the "Longest Substring Without Repeating Characters". For the formats: "We created two interview settings: public setting and private setting (see Figure 1). In the public setting, wearing an eye-tracker, participants solved the technical interview problem in the presence of an experimenter. The participant was instructed to talk-aloud while solving the task, which is a standard practice in technical interviews [28, 47]. The lab space contained a whiteboard at the front of the room. An experimenter was situated near the participant for the entire session without interrupting participants’ thought process. If a participant asked for clarifica- tion, the response was brief. For example, if a participant asked, “Does this look correct”, the experimenter was instructed to reply, “Complete to the best of your ability”. In the private setting, participants solved the technical interview problem in isolation. Participant were provided a private room with a whiteboard. Participants were informed that the experimenter will step out the room and will not return while they are solving the problem. Thus, they had to make sure that there is no questions left for them. When the participant was ready to begin the task, they wear the eye-trackers, the door was closed, and the participant worked in privacy, until the task was completed within allotted time. After the experiment, we clarified with…

All great examples! Here are some other great researchers/groups that do research in this area: - Brad Myer's Natural Programming Project at CMU (tons of novel tools!): http://www.cs.cmu.edu/~NatProg/index.html - Andy Ko's work on debugging and learning to code: http://faculty.washington.edu/ajko/publications - Emerson Murphy-Hill's work on the usability of dev tools, especially refactoring tools: https://people.engr.ncsu.edu/ermurph3/pubs.html - Chris Parnin's studies on the cognitive aspects of coding: http://chrisparnin.me/ - Chrsitian Bird's empirical studies of developers: http://www.cabird.com/wp/publications/ - Thomas LaToza's work on the collaborative aspects of development: https://cs.gmu.edu/~tlatoza/papers.html - Bjorn Hartmann's help support in code editors and tutorial generation: https://people.eecs.berkeley.edu/~bjoern/ - Philip Guo's tools to support novices with coding: http://www.pgbovine.net/publications.htm - My dissertation "Human-Centric Tools for Navigating Code": http://web.eecs.utk.edu/~azh/publications.html - Margaret Burnett's work on how people forage for information in code: http://web.engr.oregonstate.edu/~burnett/reprints.html

It also refers to https://chrisparnin.me/pdf/Debugging-Hiring.pdf

Referenced research paper: https://chrisparnin.me/pdf/interviews-HN.pdf Only read it partially so far, but it seems interesting.

But it has its flaws - major flaws, some would say. http://chrisparnin.me/pdf/stress_FSE_20.pdf "To understand if coding interviews—as administered today—can induce stress that significantly hinders performance, we conducted a randomized controlled trial with 48 Computer Science students,comparing them in private and public whiteboard settings. We found that performance is reduced by more than half, by simply being watched by an interviewer. We also observed that stress and cognitive load were significantly higher in a traditional technical interview when compared with our private interview. Consequently, interviewers may be filtering out qualified candidates by confounding assessment of problem-solving ability with unnecessary stress."

Chris: Thanks for including the link to your paper[1] in the README. I saved it in my Zotero library and am looking forward to reading it. Do you have any thoughts or experience with literate programming? What about the role of Docable, versus choosing PDF as the medium for your own paper, versus something like this[2] sort of thing? 1. http://chrisparnin.me/pdf/docable_FSE_20.pdf 2. https://csarven.ca/this-paper-is-a-demo

There is statistical data saying the opposite: http://chrisparnin.me/pdf/stress_FSE_20.pdf That whiteboard interviews are actually testing how you perform under stress rather than how good you are at your job. Additionally, in my opinion, it measures how good the interviewer is. If you get a bad interviewer or even just one in a bad mood, just pack your bags because you aren't getting it. Whiteboards are incredibly arbitrary in my opinion, if taken alone. If it is part of a complete assessment, ehhhh.

Direct link to the paper: http://chrisparnin.me/pdf/stress_FSE_20.pdf

Link to actual study: http://chrisparnin.me/pdf/stress_FSE_20.pdf The study participants were split into two groups. One group solved the problem alone, in silence. The other group was instructed to narrate their thought process out loud to an interviewer who stood over their shoulder (literally, see the photo on page 3) while they solved it. The flaw in this study is that the two groups weren't really assigned the same task. Solving a problem isn't the same as verbally narrating your solution to a problem. The authors attributed the differences not to the extra work narration work assigned to the second group, but to anxiety levels. The participants even explained that their difficulty came from having to talk while solving: > Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”. There are other concerns with the study format such as many of a significant number of participants simply giving up well before the clock ran out, in both the private and public interview groups. They were given 30 minutes to solve 3 problems, but some of the participants (including in the private group) were giving up around the 10 minute mark without solving anything at all.…

First a note on how to read the relevant study. The link to the study in the WSJ article is paywalled. This is a non-paywalled link [1] is the results [2]. 22 applicants solved a problem alone, 26 with a proctor present. Without the proctor present about 2/3rds passed, with the proctor present 1/2 passed. Using score >= 2 as "passing", 12 out of 16 men passed in private and 4 out of 4 women. 11 our of 20 men passed with the proctor, and 0 out of 6 women. The methodology looks robust, but especially with the claims with respect to gender I'd want to see a sample size larger than the single digits before making any generalizations. With that aside, my broader thoughts on tech interview processes: Companies want an interview process that are, - Successfully distinguishes between people that have the knowledge and abilities required to perform the job. - Has systems on accountability, consistency. - Is relatively easy to train employees to administer the interview. - Has a relatively low time-commitment for everyone involved, both interviewer and candidate. In reality, though, there are tradeoffs between each of these points. For instance, using a set question bank improves consistency and accountability. Rubrics can be more explicitly defined, and bias limited. But it means candidates can google for questions beforehand. This was a salient issue when I worked at Dropbox, there were only about a dozen technical. interview questions for a 2,500+ person company. Having…