Conference Deadlines: Is there a better way?

n the day to day life of a grad student, there are few things more important than meeting conference submission deadlines. These immovable objects tower over the academic landscape, casting their long shadows on other responsibilities and priorities, deepening as they approach. They are unavoidable and all-important: fellowships, scholarships, and applications for further study are distinguished by the student’s success in publishing at respected conferences and in journals. These venues are how we are best able to take part in and influence the intellectual discussions occurring in our field.

Despite the importance of these dates, however, the majority of us are waiting until the last minute to submit our work. This graph shows logins over time for the CHI 2011 deadline, which recently passed. This is shown on the login page of the Precision Conference submission system. From my experience, this trend is typical across other conferences and times of year.

CHI 2011: Submission system logins over time

CHI 2011: Submission system logins over time, deadline Sept. 24th

Am I misreading the data? Is the spike in logins representative of authors taking as much time as is available to polish their completed manuscripts? Anecdotal evidence and personal experience tells me this isn’t the case. What’s in fact happening is a panicked push for completion and submission, often down to the final minutes. [1]

I am guilty of this behaviour. It happens every time; my best intentions are swamped by the practical reality that writing high-quality papers with accurately represented data and thoroughly considered conclusions is extremely time-consuming. I continually underestimate the time required, which is exacerbated by the parallel challenge of effectively collaborating on these documents with multiple authors. A particularly stressful bout of this experience has prompted me to ask: is there a better way?

Considerations

After discussing this idea with several fellow researchers at a recent conference, the most common response was straightforward: the individual needs to take responsibility for their own behaviour. If you can’t acknowledge and plan for deadlines accordingly, maybe you’re in the wrong game. It is up to the individual student to map out their time to accommodate a proposal, research design, data collection and analysis, drafting, and polishing well in advance of the deadline. This cycle takes in the range of 3 months to 1 or more years to do properly, depending on the topic of study.

If the individual can do this without supervision, then they will almost certainly be successful. For the rest of us mere mortals, the overlapping responsibilities of research, teaching, coursework, and life make such a disciplined approach a bit of a pipe dream. I believe that the graph above indicates that this structured approach does not match the reality of the situation for the great majority of us.

Supervisors and research groups have a role in guiding students along this path. Undergraduate classes often build in this kind of guidance in the form of iterative submissions: an outline followed by a draft, and finally the completed work. This enables opportunities for revision and feedback, and ensures that the majority of the work is done before the final deadline is in sight. However, this kind of hand-holding disappears fairly quickly after a graduate student begins to take on their own research.

I am not laying blame. If grad students have a lot of responsibilities, it’s clear that their supervisors often have far greater demands on their time: grant-writing, supervision, teaching, and their own research, to name a few. Given this, different supervisors will be able to hold their students to a well-paced schedule to differing degrees, but it is unrealistic to expect them to micro-manage graduate research at this level. It truly is an individual responsibility, and one which each of us grapples with in his own way.

However, the apparently widespread last minute nature of conference submissions often leads to reduced clarity and increased errors in the submitted papers, and subsequent rejection. [2] It also places immense stress on one area of performance at the expense of a more balanced approach. I think in any intellectual undertaking the process is as important as the outcome, and that the current structure often reduces the quality of both. Given this, I believe the monolithic deadline system is not as obvious a solution as it seems. The question isn’t who’s to blame, but how we as a community can do better.

Alternatives

I propose two alternatives for consideration. The first is concerned with a restructuring of the emphasis which is placed on different publication venues. The second suggests that we may learn a more sane alternative to publication methods from agile approaches to software development.

Conferences vs. Journals

One senior researcher I spoke with believes our particular community (HCI specifically, along with wider applied computing fields) places far too much emphasis on conferences, with journals running second. His perspective was that conferences should be a place to present work in progress with early results, and to network and “have a chat.” In contrast, journals should be where the “real research” is published, in long form and of archival quality. Journal submissions take place over a longer time period and offer the opportunity for researchers to work alongside editors to position the work and make revisions based on their feedback. Journal articles also enjoy the input of layout and graphics professionals to assist with production.

I do not have enough experience with journal publications to comment on this very thoroughly, but when I compare my own experience publishing an article in IEEE Computer Graphics & Applications with my experience submitting to conferences, the difference is significant. The former was far more enjoyable and less stressful than the latter, and I believe the work was accordingly superior.

A few open questions:

  • How did conference publications take on such significance?
  • How does this compare to other fields?
  • Is our community best served by this emphasis?

Agile Development: Iteration and Incremental Improvement

Before beginning grad school, I worked in web development for several years. In that world, outcomes were more important than deadlines, and iterative, incremental improvements were valued more highly than large feature releases. This model of development, enabled by the network and the model of software-as-a-service, broke from traditional commercial software practices, where large feature releases were tailored to specific economically-imposed deadlines, and were inevitably followed by numerous bug patches.

If we consider research to be like software development, conference submissions are more like the old, broken way of building software. One deadline a year demands researchers corral their findings into a tightly defined format and submit on time, whether the bugs are ironed out or not. Fortunately, the review system finds and eliminates these bugs, but it doesn’t offer a way for the researcher to correct and re-submit. Iterations of the research may find their way into alternative conferences, but such opportunities are often limited. [3]

What would research modeled on agile development processes look like? It would be pragmatic, adaptive, flexible to shifting requirements, and performed in small iterative increments. I think this is actually a pretty good fit for a lot of research that happens in HCI, design and technology, where researchers are constantly asking: How can we answer these questions? What are the requirements? Does it work? Which evaluation method suits best? To my mind, however, our conference publication model doesn’t match this dynamic process, imposing a static deadline and a single filter on a diverse and unpredictable activity.

  • How would an iterative and flexible approach to research be reflected in submission and review processes?
  • What types of research is software development a good model for? Which do not fit?
  • Would a more open model of research, where the iterations are made as evident and accessible as the final product, improve our results? How would this be moderated?

Conclusions

In many ways, the current system works. The proceedings for high-level conferences are full of insightful papers that advance their respective fields, and these contributions offer a way of evaluating a given researcher’s place in the community. However, the current structure takes a significant toll on student quality of life, and limits the potential for a wider range of contributions. Each publication window missed represents a precious lost opportunity for graduate students to share their research. I believe we can do better, and that this is a conversation worth having.

Footnotes

1: It would be interesting to see the data from the submission system of last login for each submission, indicating the point at which all work was completed. My hunch is that it wouldn’t look much different.

2: The hard deadline may also have the unintended side-effect of limiting creativity and insight in published work. If you’re anything like me, the day or week after a deadline is when your most lucid insights into the problem crystallize. When you’re bent over a keyboard under the cruel tyranny of the clock, the data seems unyielding and your conception of the argument rigid.

3: This is at least in part due to the review cycle, which doesn’t indicate acceptance or rejection until months after submission. In the meantime, the author is unable to submit to alternative venues due to the requirement made by conference organizers that the work not be under simultaneous review.