For the third year in a row, we’ve made it through the speaker selection process for Ignite UX Michigan. As before, it’s kind of a heartbreaking process. Having been rejected from every conference I’ve pitched a talk to for the last 2 years, and having been accepted by two out of dozens, I can sympathize with the people that we can’t fit into our lineup. Being involved in the selection process also really drives home how good all of these proposals are.
We’ve used the same basic selection process three times. It’s designed to be as fair as possible, but the concept of fairness, unfortunately, depends a lot on how you define it. Lately, I’ve spent a lot of time thinking about how we can make it more fair next time, and how to further the goals of the event.
What Are We Trying to Accomplish?
The two highest priorities for Ignite UX Michigan are:
- Being a free event
- Having diverse speakers
The speaker selection process doesn’t have a lot to do with the first priority, freeness, but it obviously is highly relevant to the goal of speaker diversity. Diversity pretty much has the dimensions that you’d expect—gender and race. Closely related and of great importance to me is diversity in terms of “do you see this person speaking at conferences all the time?†The UX field sure has an in crowd. On the plus side, if you’re disappointed to miss one of the headline speakers at a conference, you can just catch them at another conference later on in the year. You always see the same people speaking. On the minus side there may be some other voices in our field that would be really good to hear from.
Of course, there are more aspects of diversity than what I’ve just discussed, and they’re important ones, too. Something dear to my heart that we can’t address in Ignite UX Michigan is class diversity
The scariest part of taking a stand and saying that diversity is important is that, once you start worrying about it, you see the ways that you can and do fail. The work is ongoing. I’d love to talk about what we can do to get better.
The Current Selection Process
Well, how can our process let us down when it comes to diversity? To get people to propose talks, we make public announcements about the submission process, and we try to encourage specific people to propose talks, but it’s obviously very likely that there are plenty of people that just never get the message in time. That’s already a limiting factor—we’re more likely to get proposals from people that are plugged into the community. To a large extent, we get what we get, and the most practical idea I can think of for increasing diversity at this stage would be to just keep engaging with the community to try to spread the word about our event.
Then, we feed all those proposals into the selection process. I would characterize the process that we have used for the past three years as not specifically promoting or detracting from diversity. The process is:
- The 12 highest-rated talks are selected, with the following adjustments:
- We average the ratings for each talk
- The reviewers fill out the survey. For each proposal, they rate it from 1 to 10, with 10 being the strongest vote in favor of a talk. Each proposal is rated separately (as opposed to ranked against each other). The reviewers are volunteers who do not know who proposed talks, insofar as trust people to be honorable about this.
- The facilitator creates a survey that lists the titles and descriptions of all of the talks. This survey doesn’t include the names of the people that proposed talks and, in a perfect world, has no way of identifying who submitted the proposal
- Although people can submit more than one proposal, we only take one talk per person
- In 2014, a talk I proposed was selected, but I didn’t want to take the space of another speaker, so we chose the top 13 instead
- In cases of a tie, the proposal with the lowest variance wins
Why choose this approach? Well, we want a diverse set of speakers, true, but we also don’t want to have a speaker selection process where we use our judgment to hand-pick our lineup. Using judgment risks us choosing who we know, and that’s how you end up with the same people talking at every conference. So we use the blind-review-with-math approach, and hope that 1) our pool of proposals is diverse to begin with and 2) that an impartial selection process will produce acceptable results.
Possible Improvements
I’m interested in improving the process, I’ve thought about it a lot, and I’m not sure exactly what the best approach would be. Some ideas that have come up are:
- Apply weighting to proposals:
- Like, if you spoke at the previous year’s event, your proposals get a not-insurmountable penalty—the logic being that good proposals will still get accepted, but it’ll be a bit less likely.
- Apply a bonus to people who have never spoken at a professional event before (or never spoken at a national conference)
- I am strongly reluctant to start asking people things about themselves, like gender, but we could:
- Set aside a certain number of speaking slots for students
- Have people classify their talk into one of X categories, and then have a certain number of slots for each category
- Of course, neither of these things necessarily help with diversity, but the student thing is perhaps promising
- Instead of using low variance as a tie-breaker, instead use high variance—this means that we’d favor the talks that people disagreed on
- Along those lines, we could fundamentally reimagine the review process so that diverging opinions is a primary factor
This is tricky but important stuff. When we started Ignite UX Michigan, we didn’t even know if people would submit talks. Our process was something we came up with after we got more proposals than room for speakers and an attempt to come up with something relatively fair. How can the selection process promote the goal of diversity? I’d love to get more ideas and, perhaps even more importantly, find people that are interested in helping us get better.
