OCT2 (Online Conference Toolkit v.2)
Program Chair's Guide to OCT2
This guide gives the overall organization of the conference reviewing
procedure implemented by OCT2 and used for CVPR 2003.
Installation instructions for OCT2 can be found here
|
1. User Groups
The following user groups are supported by OCT2:
- Registered Users: Anyone successfully registering on the
system is put into this group. Registered users can submit papers
during the paper submission period. Users that do not complete their
registration successfully are put in the deadbeats group.
- Reviewers:This is the group that does the actual paper
reviewing. They can have one or more papers assigned to them for
review by users in the admins group. In addition to the reviews
themselves, reviewers can submit a rank ordering of their papers. The
reviewers may give a paper to someone else to review that is not a
member of the 'official' reviewer pool (eg., a student). All CVPR
2003 Program Committee Members were classified as reviewers. Each
paper is supposed to be reviewed by several users in this group.
- Area Chairs (aka Chairs): Each chair is responsible for
overseeing the review process for a subset of the papers submitted to
the conference. Each paper can be assigned to one chair only. The
chair is responsible for recommending reviewers for each of their
papers but the actual assignments are finalized by users in the
admins group. During the committee meeting phase, where final
paper decisions are made, chairs can access the review information for
ALL papers submitted to the conference. The only exception are papers
with which the chairs have a conflict of interest. All CVPR 2003 Area
Chairs were put in the chairs group. For conferences with a
flatter reviewer hierarchy (e.g. with just program chairs and program
committee members) this group would consist of the conference's
program committee members.
- Admins: This group should contain only the Program Chair(s)
and anyone else providing systems support to the local OCT2
installation.
A group lower in this list subsumes the group above it. For instance,
program chairs can assign a paper to an area chair for review. Area
chairs may even be assigned papers by other area chairs if the program
chairs decide to allow this in OCT2.
top
2. Reviewing Phases
OCT2 implements the following reviewing phases. Follow the links for
more details on each phase and for phase-specific instructions.
- Abstract submission phase: Authors can register a paper by
submitting an abstract and/or full papers. To facilitate the review
assignment process, authors can provide up to four 'research
categories' that best match their paper, along with a set of weights
describing the relative importance.
- Final paper submission phase:
Authors can submit full papers for papers that were already registered
but cannot register a new paper.
- Area Chair assignment phase: Program chairs assign each
paper to an area chair. This process can be semi-automated using a set
of Matlab scripts that rely on the weights supplied by the authors to
obtain an initial assignment and identify 'assignment
bottlenecks'.
- Reviewer assignment phase: Area chairs recommend three
registered reviewers for each of their papers and one alternate
reviewer (eg. in case of contention or conflict of interest of the
other reviewers). In addition to the reviewers, area chairs give the
program chairs some additional information about the rationale behind
their particular choice, in the form or a drop-down menu of possible
reasons. Based on these recommendations, program chairs create the
final review assignments.
- Review phase: Reviewers get to see the papers they are
reviewing, and area chairs can keep track of the progress of their
reviewers.
- Discussion phase: Reviewers get to see what the other
reviewers said about papers they were reviewing and can discuss these
reviews. Discussions are mediated by the Area Chair who may decide to
withhold reviewer identities from each other.
- Author response phase: Authors get to see the reviews for
their papers and have the opportunity to submit a brief response to
the reviewer comments. These responses are visible only to the program
chairs and the relevant area chairs.
- Pre-Meeting phase: Based on all the information submitted
thus far, area chairs make their recommendations and write a short
summary statement and/or a response to the comments sent by authors.
- Meeting phase: OCT2 was designed to facilitate a meeting
conducted electronically. For CVPR 2003, area chairs came to the
meeting with their laptops and used them to access the system
directly. During this phase OCT2 allowed area chairs to view all the
papers submitted to the conference, along with the information
collected for them (except for conflict papers).
- Post-Meeting phase: OCT2 assumes that some papers may be
left on the borderline at the end of the committee meeting and that
some area chairs may not have completed their summaries. This phase
is for tying all these loose ends.
- Results phase: Authors see the final decisions along with
the complete set of reviews received for their paper and the area
chair comments.
OCT2 can only switch from one phase to another in the above order. For
instance, to switch OCT2 to the Meeting phase it will have to be
initialized in the abstract submission phase and then successively
switched from one phase to the next until the Meeting phase is
reached.
If one or more of these phases does not apply to your conference, it
can be effectively skipped by moving OCT2 out of that phase and into
the next phase.
top
3. Customizing OCT2
Information communicated to authors (e.g., research categories of
papers, emails to authors, messages displayed on the website) can be
customized with minimal effort and with no knowledge of the internal
workings of OCT2.
OCT2 offers some limited capabilities for customizing the review
form with no prior knowledge of OCT2 internals. This includes
modifying the text for each question on the form and modifying the
text associated with multiple-choice questions.
Further customization of the review form, such as increasing the
number of questions asked or the number of choices in a
multiple-choice question, are possible by directly modifying the tcl
scripts that define the review form. See the OCT2 User's Guide for more information.
top
4. Paper Assignments
Assigning Papers to Area Chairs
CVPR is a moderately large conference, having about 900 submitted
papers in 2003 and requiring more than 2,700 reviews. While most of a
program chair's functions can be done interatively on the website,
OCT2 allows the relevant information to be exported to text files for
further processing by the program chairs (either manually or
automatically). Similarly, program chairs can provide all the relevant
information (e.g., the table assigning papers to area chairs) into a
text file that can be loaded by OCT2.
To facilitate the assignment of these papers to Area Chairs, OCT2
allows authors to choose 4 research categories that best describe
their work among a list of possibilities, along with weights. OCT2 can
export this information to a text file for further processing.
For CVPR 2003, we relied on a set of Matlab scripts that used the weight information to semi-automate the assignment process. The scripts were written by David Fleet and
implement an assignment method designed by Peter Meer.
The scripts assume that program chairs assign weights and categories
to area chairs, just like authors assign weights to papers. This leads
to a matching process whereby vectors of paper weights are matched
against vectors of area chair weights. The implementation also
incorporates constraints on the number of papers that can be assigned
to any one area chair.
Once completed, final area chair assignments can be loaded back into
the system.
Assigning Papers to Reviewers
OCT2 relies on a two-stage process for assigning papers to reviewers:
(1) area chairs recommend reviewers for their papers and (2)
program chairs export this information to a file, process it, and
upload final decisions.
OCT2 implements four functionalities designed to reduce reviewer
contention and to allow program chairs to make more informed
assignments:
- To help guide their choices, OCT2 shows each area chair the total
number of papers for which a reviewer has already been recommended
by the area chairs.
- Area chairs specify a fourth (i.e., alternate) reviewer in addition to
the three main reviewer recommendations.
-
Area chairs can choose one of several options describing the main
rationale behind their choice.
-
To reduce reviewer contention and to make sure that reviewers get
papers only a certain category of papers, OCT2 allows program chairs
to restrict access to reviewers: before this phase begins, the program
chairs specify, by uploading a text file, which reviewers an area
chair will have access to. Area chairs can therefore recommend
reviewers only from their pool of accessible reviewers. This file can
be modified and reloaded by the program chairs if the need arises to
grant access to a particular reviewer during the review assignment
phase.
top
5. Conflicts of Interest
Double-blind reviewing
OCT2 implements a double-blind review process in which the identities
and institutions of authors and reviewers are hidden from each other.
Area Chairs know the identity of the reviewers for the papers they are
handling but they do not have access to author names and institutions.
Only the Admin group has global access to this information. Therefore
conflicts of interest can only be handled by the program
chairs themselves.
Identifying & Marking Conflicts of Interest
OCT2 provides simple tools for the Program Chairs to quickly identify
conflicts of interest between papers and reviewers or between papers
and area chairs. If desired, OCT2 allows program chairs to mark a
paper as having a conflict of interest with a particular area chair.
During the final committee meeting, the area chairs will not have
access to those papers.
top