Table of Contents
1. Preparation and General Questions
2. Overview of the Review Process
3. Detailed Steps and Options
4. Main Configuration Options
5. Reviewers Select Priority Topics
6. Bidding Phase for Reviewers
7. Rebuttals for Authors
8. Ratings of Reviews by Authors
9. Online Forum for Review Discussion
10. Making Results Accessible to Authors
11. Configuration of the Reviewing Options
12. Configuration of the Review Forms
13. Acceptance Status
Configuration of the Reviewing Functions
The quality of the presentations is essential for every academic conference and decides about acceptance and reputation of the event. Therefore, it is important to ensure that the submission and reviewing process that you want to employ is correctly configured in ConfTool Pro.
Each conference organizer has specific ideas how to perform the reviewing process. Some of the questions that arise are: How many reviewers are required per submission? Are open, single, or double blind reviews required? Will abstracts be reviewed, full papers, or both? What criteria are used to evaluate papers, and will there be review scores or only written comments or both?
The following manual will explain the configuration options for the reviewing module of ConfTool Pro. In case this is your first conference, you can follow the manual step by step. Experienced organizers can feel free to read specific sections only for detailed explanations.
Please note that several of the settings for the reviewing module are closely connected to the submission module. Thus, make sure that you have understood and configured the submission process appropriately. You can find all relevant information here: Submission Module Configuration.
This manual starts with a number of questions that can help you to identify the strategy behind your reviewing process. It continues with a short overview of the review process from the reviewer's point of view, to illustrate the concepts behind ConfTool Pro. Furthermore, you will find explanations on the most important features and options in a survey of the review module settings as well as the review form configuration. Finally, we have compiled some hints on the final step of the reviewing process: How to assign acceptance statuses to submissions. Please note that all options and settings are also explained online on the settings pages of the system. For less frequently requested options, take a look at the online forum, where you can find a lot of additional information.
1. Preparation and General Questions about the Requirements
- How do you want to recruit reviewers? Did they already confirm their participation or do you want to invite them first?
- How many reviewers are required per submission?
- What phases does your reviewing process have and how long is each of them?
- Shall anonymity be unilateral or reciprocal (single- or double-blinded reviewing)?
- Shall there be no anonymity at all (open review process)?
- How many submissions are you going to accept?
- What are the evaluation criteria?
- How are the criteria weighted?
- What are reasons for rejection?
- What are the rules for conflict of interest (COI)?
- Can reviewers refuse to review a paper?
- Shall only abstracts or full paper uploads be reviewed?
- Do you require a two-phase peer reviewing process? (You can find more information in the support forum.)
- What details of the review results do you want to pass on to the authors? Will it include narrative feedback or scores?
- Shall authors be able to respond to the review (as narrative only or including a review rating)?
- Is there a final upload phase, where authors can submit a revised version of their papers that considers the feedback of the reviewers?
- Do authors get the chance to rate or comment reviews?
2. Overview of the Reviewing Process
Short Overview
The reviewing process in ConfTool is straightforward.
First of all, each reviewer will require a user account, so they can take part in the reviewing process. These accounts allow organizers to assign reviews to reviewers, and reviewers to access the submissions assigned to them.
These user accounts with the user role "reviewer" can be established manually in the back end of the system. Another option is to import users via the ConfTool import function. Simply create an Excel sheet with the first and last name and e-mail addresses of all reviewers and follow the instructions for user import. If the reviewers have not confirmed their participation, you can also invite reviewers by using the invitation function of ConfTool Pro.
Secondly, the received submissions have to be assigned to reviewers. This can be done automatically or manually. After this step, you can enable the reviewing phase.
As a last step, you can inform reviewers that they can start the review process by using the bulk e-mail function.
Reviewer Account
When reviewers log in to their account, they will find a screen which contains at least two main boxes. All reviewing functionalities can be reached in the box with the heading "You are reviewer or member of the program committee (PC member). You currently have the following options" (see image 2).
3. Detailed Steps and Options
The review process can also include further steps for reviewers. All these operations are performed online via ConfTool Pro:
- Reviewers can use "Select Priority Topics" to select their area of expertise. This information will support conference chairs to assign contributions to reviewers with matching interests. It is therefore important for the quality of the review process.
- During this phase, reviewers can also be allowed to state the maximum number of reviews which they are willing to do (see image 2).
- In a bidding phase (after the submission deadline has passed), reviewers can "Bid for Contributions". Reviewers will find the list of all submitted contributions of the conference shown in their account. Contributions are listed by matching topics and reviewers can state their preference (much wanted, would like, uncertain, would dislike) or a conflict of interest (see image 2).
- After these phases are over, the chairs will assign submissions to reviewers, e.g. based on matching topics and biddings.
- After enabling the reviewing phase, reviewers will then find the submission assigned to them for reviewing in their user accounts. Now, they can "Enter and Edit Reviews". They will find details of the proposals (including uploads, if available), which they can read online or download. Reviewers will be able to enter their evaluation online. Depending on the review form, reviews will be based on review factors and / or textual feedback.
- It is possible to let reviewers have access to evaluations from other reviewers. Reviewers can furthermore be given access to the online forum after they have submitted their reviews, in order to leave comments and discuss about their own submissions.
- Furthermore, you have the option to define Meta-Reviewer and Senior Reviewer who always have access to all reviews of those submissions they were assigned to. Additionally they can use the ConfTool system to send messages to all other Reviewers.
- Members of the Program Committee can have an online discussion via the online forum and vote for or against acceptance for all submissions.
- You can include a feedback loop for authors. When you enable the rebuttal function, authors will have the chance to reply to all reviews that their submission received.
- In consequence, reviewers might be asked to revise and refine their evaluations based on the comments of the authors.
- Authors can also be allowed to rate each review separately on a scale in order to judge the quality of reviews and the work of reviewers.
- Subsequently, the chairs will decide about acceptance based on all reviews that were received (taking into account the results of discussions and recommendations in the online forum and rebuttals and review ratings, if available).
- When review results are published, authors of accepted papers might be asked to incorporate the hints and comments from the evaluations and the final result of the program committee in a final upload or camera-ready copy.
4. Survey of the Configuration Options
You will find most configuration options in the box "Parameters for the Reviewing Functions of ConfTool". Additional options can be found when you configure each submission type / track, as they are specific for each submission type / track (see image 3).
Let us start with the main options for the review process, which you find here:
Overview => Settings => Main Settings for the Review Process
Here, you can access general settings, such as deadlines, access rights and display options. Many of them are explained in the software itself. Therefore - in the following - we will only take a closer look at the most important of them.
Display Options
In the section "Display Options", you can enter a short text used as "Introductory text for the page 'Enter and Edit Reviews'". This information will be displayed to reviewers when they access the system and will appear above the list of proposals to be evaluated. Often, short texts are more practical. If you have detailed instructions, it could be a good idea to include a link to an external source where they are explained in full.
In the following section "Reviewing phase: PC members and reviewers can access the submitted contributions assigned to them for review" you can activate the reviewing module by selecting the check box for the option "Enable module and select start and end date". Once activated, you can use the option "Set start and end date" to let the software automatically start and close the reviewing phase (see image 4).
Review Options
Moving on to the section "Review Options", you can first set a "Method for 'Conflict of Interest' Identification". The software will cross-check the details of reviewers and the details given in the submissions. Choose one of the two options. Later on, when you assign reviewers to submissions, a conflict of interest (COI) warning will appear wherever the software has detected similarities according to the COI rule you have chosen (see image 5).
The next two options "Enable Meta-Reviews?" and "Enable Senior Reviews?" are almost identical. When you activate one or both of them, you will get the additional roles of Meta-Reviewer and Senior Reviewer. Users with either of these roles have read access to reviews of other experts assigned to the same submission and can contact them via e-mail and the ConfTool messaging function, e.g. to remind them of deadlines. Furthermore, they can summarize the review results in their own review.
If you want to assign a larger number of reviews to each reviewer, please use the option "Enable Filter for Reviewers", which will enable a function to filter the proposals by submission type / track or open / completed reviews on a reviewer's "Enter and Edit Reviews" page.
In some cases, the option "Finalize Reviews" can come in handy. When you activate it, reviewers will have to finalize their review to submit it, i.e. they cannot update their evaluations after submission.
If you want your reviewers to be able to refuse to work on an evaluation that has been assigned to them, please activate the option "Do Reviewers have the Option to Refuse Reviewing a Submission". The reviewer will have to state a reason why he / she cannot write the review and the chairs / track chairs will be informed about the refusal via e-mail.
You will then find an additional option to manage the reasons for reviewers to refuse an evaluation here:
Overview => Settings => Manage Reasons to Refuse an Evaluation
In case your reviewers will also have access to PDF files uploaded by the authors, the option "Add Watermark to PDF Downloads" - when activated - will automatically create a watermark on the PDF for reviewers that reads "For Peer Review Only".
You can also allow reviewers to upload a file for each submission they evaluate if you enable the next option “File Upload for Reviewers”. This function is helpful if you want reviewers to provide a file with comments and feedback for the authors as supplement to their review in ConfTool. (You can find more information in the support forum.)
You can allow reviewers / PC members to select their maximum number of reviews with the option "Selection of No. of Reviews". Define a span for the minimum and maximum number of reviews and also set a default in the option "No. of Reviews per Reviewer". These numbers will be considered during the automatic review assign process (if possible) and be shown to chairs for the manual assignment process. Please activate the phase and set a start and end date for "Reviewers Can Select Their Priority Topics" to enable the option for the selection of the number of reviews (see image 6).
5. Reviewers Can Select Their Priority Topics
In the next section "Reviewers Can Select Their Priority Topics", you can let reviewers / PC members or all users select their priority topics. This is helpful for matching (potential) reviewers with submissions. Please activate the phase and set a start and end date. Choose an appropriate setting for the option "Priority Topic Selection". Furthermore, you can activate an automatic redirection to the priority topic selection page after user login if you activate the option "Test Topic Selection at Login" (see image 7).
6. Bidding: Reviewers Can Bid for Submitted Contributions
For some conferences, it is convenient to enable a phase during which reviewers indicate their preferred papers and conflicts of interest (The options available are: much wanted, would like, uncertain, would dislike, and conflict). You can activate this option in the section "Bidding: Reviewers Can Bid for Submitted Contributions". Again, activate the phase and set a start and end date. Choose the maximum number of submissions that a reviewer can select as "much wanted" to review in the option "Biddings for "Much Wanted" Reviews" (see image 8). The bidding function can also be enabled or disabled for each submission type / track separately. Biddings further simplify the review assignment process and reduce the risks of a conflict of interest.
7. Enable Rebuttals for Authors
Some conferences allow authors to write a rebuttal as a reply to the evaluation of the reviewers. The rebuttal can refer to all reviews and there are no separate rebuttal forms for each single review. A rebuttal to a submission will appear in the account of all reviewers who were assigned to the submission. The reviewers then might refine their original review. Use the option "Enable Rebuttals for Authors" to activate this feature. Set a maximum number of words using the option "Maximum length of rebuttals". Don't forget to activate the phase and set a start and end date (see image 9). Reviewers and members of the program committee will find the rebuttals in the online forum (if enabled, see below) and in the details of the submissions; admins can also export all contributions with review details including rebuttals as XLS.
8. Enable Ratings of Reviews by Authors
A similar feature is "Enable Ratings of Reviews by Authors". When you enable it, the authors can rate each review separately. Enable the module and set a start and end date. Reviewers / PC Members will find the review ratings in the online forum (if enabled, see below) and in the details of the submissions; these hints might be helpful to assess the quality of the reviews and the work of the reviewers (see image 10).
Please specify the elements of the review rating form for authors here:
Overview => Settings => Manage Elements of the Review Rating Form for Authors
Create categories and enable or disable them. Then define Min and Max scores in the option "Range of Scores for all Ratings". In the section "Text Fields of the Review Rating Form" you can define if one part of the message created by the authors should be directed to the reviewers and another to the program committee. Finally, you can choose an appropriate setting for the option "Visibility of the Review Rating Results" to decide what part of the reviews rating the reviewer will get to see.
9. Settings for Online Forum for Review Discussion
The section "Settings for Online Forum for Review Discussion" deals with options of an online discussion tool, with which members of the program committee can read all submissions and reviews and create messages for other members with their opinion whether the submission should be accepted, including a personal message (see image 12). To use this module, activate it and set a start and end date. You can choose for each submission type / track separately if the submissions of the track shall be available in the forum. With the option "Access to reviews of co-reviewers? / Access of Reviewers to Online Forum?" you can allow reviewers to get access to reviews from other reviewers and the online forum to exchange messages. You can choose from several display options regarding anonymity using "Show names of message authors". If the uploads are of importance also during the discussions in the online forum, enable the option "Uploaded files available in program committee online forum". You can also let PC members have access to rebuttals and review ratings in the forum when you enable the option "Rebuttals and/or Review Ratings available in program committee online forum". This additional information could help the committee to make the best possible decision (see image 11).
You can also use the online forum for a slightly different purpose: the discussion about the presentations that have already been assigned to sessions in the program via the ConfTool agenda. PC Members can use it to write internal comments about the presentations and make these available to all other PC Members to see. To use this function, please activate the option "Link Conference Agenda with Online Forum". NB: When you use the online forum for this purpose, please disable the options "Enter Votes with Forum Messages" and "Close Discussion after Decision" (set both to "No").
If you activate the option "Enter Votes with Forum Messages", PC members have to state their vote: "Accept", "Maybe accept", "No vote / undetermined", "Maybe reject", "Reject" (see image 12). All votes will be aggregated and visualized here: Overview => Submissions & Reviews => Results of the Reviewing Procedure & Decision About Acceptance. This can facilitate the decision making process (see image 13).
The option "Close Discussion after Decision" can be used to no longer allow any discussions in the online forum on a submission when an acceptance status other than "On Hold" has been chosen. (More specifically: acceptance statuses for which the option "Show Results" is enabled).
10. Results of the Reviewing Procedures Are Accessible to Authors
Finally, in the section "Results of the Reviewing Procedures Are Accessible to Authors" you can determine when authors will have access to the review results of submissions for which the acceptance status has been changed from "On Hold" to another status. Enable the module and set a start and end date (see image 14).
11. Configuration of the Reviewing Options in ConfTool Pro
You can set further review options and the preferred review form per conference track / submission type (see image 15). Please go to:
Overview => Settings => Manage Submission Types and Conference Tracks
... to edit your submission type(s).
Scroll down to the section "Review Options". The option "Enter and Edit Reviews" allows you to disable the reviewing function completely for a specific submission type / track. You can choose from three different options:
- Disable Reviewing: This setting will remove the reviewing option for this track and hide all existing reviews from authors as well as reviewers.
- Close Reviewing: Reviewers can be assigned. Assignments will not (yet) be shown to reviewers. Only completed reviews will be shown to them.
- Enable Reviewing: Reviewers can be assigned. They can access the assigned submissions and submit reviews if the reviewing phase is enabled.
With the option "Double Blind Reviews" you can hide authors' names from reviewers, but please note that this will (of course) not remove any author information from the content of the submitted files. Only the file names are anonymized.
With the option "Review Results", you can choose for each submission type / track, if the full review results will be shown to the authors (the corresponding phase must also be enabled and an acceptance status be set) or only the acceptance status (for which the option "Show Results" is enabled) without the comments from the reviewers will appear in the account of the authors.
"Bid for Contributions" can be enabled or disabled for each submission type / track separately. The bidding module has to be enabled and the phase set in (as described above in this manual):
Overview => Settings => Main Settings for the Review Process
The "Online Forum for the Program Committee" can also be enabled or disabled for each submission type / track separately. The module has to be enabled and the phase set in (as described above in this manual):
Overview => Settings => Main Settings for the Review Process
In the section "Further Settings ", you can define for the option "Class Name of Review Form" which review form shall be used for the corresponding submission type / track. The system provides four standard review forms you can choose from. Usually you will keep the “Default_Form”. You can find more information here: Configuration of the review forms in ConfTool Pro. In case we have developed a customized review form for you, you can find the according "Class Name" for the review form here, too (see image 15).
12. Configuration of the Review Forms in ConfTool Pro
For the two review form classes "Default_Form" and "Simple_Form" there are numerous options for customizing the forms.
Please go to:
Overview => Settings => Review Form Options and Factors
... to enable or disable and set the weighting factors for each review criterion (review category). The weight will be used to calculate the total score of each review (see image 16). Please note that we do not recommend setting the "Overall recommendation" to 0%, as this might be confusing or lead to (logically) inconsistent review results.
If you click on "Edit Labels and Ratings", you can edit the wordings of each criterion. If you enable the expert options on this page, you can also change the style of the review ratings and add a text field for each criterium (see image 16).
You can choose for the option "Range of Total Scores" one of two settings: 0-100 will add all weighted scores of the criteria to a maximum of 100, the setting 0-10 will show the average and has therefore a maximum of 10.
You can enable the option "Familiarity of Expert". With this option enabled, reviewers have to state their familiarity with the topic of the submission during the review process. The value entered here (from 0 "Completely new to me" to 10 "Very familiar") will be considered when the overall score is calculated (“Default_Form” only).
If you disable the option "Show Review Categories and Scores to Authors", none of the review criteria (including the scores) will appear in the author's review results. The author will only get to see the acceptance status and any comments for the authors entered by the reviewers (if enabled).
The option "Comments for the Authors" is activated and set to "Yes, required" per default. This option will show a field for comments from reviewers. These comments will be made available to the authors when the review results are published. This option is recommended, as it is the direct feedback with explanatory notes, hints, criticism and praise from reviewers to authors.
For the "Default" review form only, you can find extra choices in the section “Additional Options for "Default" Review Form”.
You can let reviewers enter the "Contribution of the Submission", i.e. a statement about the impact of the submission on scientific progress. The contribution of the submission as entered by the reviewers will also be made available for the author to read.
When you activate any of the four options "Best Paper Award", "Recommended as Poster", "Major Revision Required" and "Veto Against Acceptance", reviewers will see checkboxes which they can simply check if they think that the statement is true. If a reviewer has checked the corresponding box, an icon will appear for the submission on the list "Results of the Reviewing Procedure & Decision About Acceptance" (see image 17):
If you want to, you can also update the review form options for each of submission type / track separately. First, please be advised that you can choose one of the available review forms classes for each track. Second, if you enable the expert options, you can also create alternative labels for review categories / factors, create new factors (and disable others) or change the weighting for the factors for each submission type / track separately.
It is important that you always test your review form! You can easily test your forms when you go to “Manage Submission Types / Conference Tracks”, select the Track / Type you want to check and then select in the "Related Functions" menu on the right the option “Show review form for track …” (see image 18):
Overview => Settings => Manage Submission Types / Conference Tracks
Finally, please do not change the review form settings after reviews have been submitted, as these will also have effect on the already submitted evaluations.
13. Acceptance Status
The acceptance status is mainly needed as final part of the reviewing process. It is usually set by the chair(s) and is based on the recommendations made by the reviewers. The acceptance status defines for instance if a contribution is rejected, accepted or if it needs revision.
However, in a conference with several review phases, the status can also define that an abstract was first accepted ("Accepted - Abstract") but then rejected as a full paper ("Rejected - Full Paper"). Any "Accepted" status might be further divided into submission types, e.g. "Accepted - Oral Presentation", "Accepted - Full Paper" and "Accepted - Poster Presentation". You can also use the acceptance status to mark certain papers: For instance, distinguish between preliminary accepted papers for which authors have not yet paid their registration fees ("Accepted - Unpaid") and papers for which at least one of the authors has registered for participation and paid the fees ("Accepted - Paid").
Please go to:
Overview => Settings => Manage Acceptance Status
Here, you can define different statuses and the options of each status. The system provides several frequently used examples such as "Accepted", "Rejected" and "Reservations". You might have to add more acceptance statuses, e.g. "Accepted as Oral Presentation" or "Abstract Accepted - Upload Full Paper". To define such new statuses, we recommend using an existing status with similar parameters and copying it, so you just need to update the wording.
Hints:
- "On Hold" is the default status and will not be shown to authors. Choose a status other than "On Hold" to show the review results to authors.
- More options are available in the expert settings (click on the cogwheel in the bottom navigation bar to activate them)
- You find more information about setting the acceptance status for submissions here:
Evaluating the Review Results and Setting the Acceptance Status of Submissions.
