Your assigned papers are now available at https://cmt3.research.microsoft.com/BMVC2019/.
Please check them and notify us at email@example.com of any suspected conflicts, violations, or other concerns ASAP by Friday, May 17 latest.
The deadline for completion of the reviews (for BMVC Reviewers) is Monday, June 10th.
So your main Area Chairing tasks do not begin in earnest for a few weeks yet.
Note the deadline for Area Chair meta-reviews is Monday, June 17. For full dates see: https://bmvc2019.org/dates/
Most ACs have been assigned 16–20 papers. (Most reviewers were assigned 3-5 papers each.)
Role of the AC:
- The AC must ensure that all reviewers are submitted on time by Monday June 10, or if necessary request additional reviewers/reviews.
- The AC leads the discussion with Reviewers, from Monday, June 10 – Monday June 17 at the very latest.
- The AC must write a meta-review for each paper by Monday June 17. See below for advice on how to write a meta-review.
You must ensure that you have 3 good reviews for each paper for which you are AC. Instructions sent to Reviewers is available at: https://bmvc2019.org/authors/instructions-for-reviewers/
In the case of very weak papers, it is ok for some of these reviews to be provided by the ACs.
Reviewers have been instructed to submit their reviews by Monday June 10.
We will be sending reminders to Reviewers as the deadline approaches.
However it is often helpful if the ACs send a more personal reminder.
If you suspect that a review will not be provided in time, you should seek an alternate reviewer by liaising with the Programme Chairs.
Final recommendation and meta-reviews:
Area chairs need to write a meta-review (See below) for each paper for which they are the AC, due June 17. The meta-review must make a clear recommendation for accept/reject, and outline the reasons for this recommendation.
With regards reviewing policy:
We have done our best to desk reject our any obvious dud papers but given the large number of submissions some may have crept through.
Papers that do not adhere to the paper submission policy (https://bmvc2019.org/authors/submit-your-paper/) should be rejected. This is mainly page length and formatting style.
We were quite strict on over length papers but we did let papers that exceeded the 9 page limit by a few lines (up to 5-6) through.
IMPORTANT NOTE: The 9 page limit is only for main text. References and appendices do not count. In case of a few papers an appendix (or appendices) of a few pages is present. Due to ambiguous wording on the BMVC2019 submissions page, we felt we had to let these papers through. Reviewers and Area Chairs may wish to make suggestions how the final paper, if accept, may look with or without the appendices.
With regards ArXiv submissions, whilst discouraging publication on ArXiv prior to the review process, we adopt the same policy as CVPR:
We realize that with the increase in popularity of publishing technical reports and ArXiv papers, sometimes the authors of a paper may be known to the reviewer. As per CVPR (2015 PAMI-TC meeting), ArXiv papers are NOT considered prior work since they have not been peer reviewed. Therefore, you should review your BMVC papers independently as if the ArXiv papers didn’t exist. Citations to these papers are not required and failing to cite or beat performance of arXiv papers are NOT grounds for rejection. Please see above for guidelines on handling ArXiv papers. An important general guideline is to make every effort to treat papers fairly whether or not they know (or suspect) who wrote them. There are some specific examples in the Reviewer FAQs below. Reviewers should not search for the authors of a paper, and complain that the paper is not anonymous if they happen to find them.
Citations to these papers are not required and failing to cite or beat performance of arXiv papers are not grounds for rejection.
Please note that there will a prize – sponsored by Dyson: A Robot Vacuum Cleaner – for the best two paper reviewers. Other top reviewers will be acknowledged on the BMVC2019 web site.
So we request that area chairs note any exceptional reviews either via CMT or by a general email to BMVC2019 program chairs (firstname.lastname@example.org).
Summary of AC Actions:
- To do by Friday May 17:
- Download your assignments from CMT and check for any potential conflicts or other problems.
- Go to https://cmt3.research.microsoft.com/BMVC2019/ and open your Reviewer Console. You should see all your assigned papers. To download the PDFs, click on the “Actions” button in the top right and select “Download Files”. Or you can download individual papers by clicking the links underneath the respective titles.
- Check your assignments for any conflicts that may have slipped through the institutional domain system in CMP. If you recognise the source/authors of a paper (e.g., from seeing it on arXiv), it is still fine for you to review it, unless you feel that you absolutely cannot judge the work fairly.
- Notify the Program Chairs (email@example.com) if you think you are conflicted with a paper or find other potential violations (anonymity, plagiarism, double submission). Programme Chairs checked the papers for obvious violations, but there is still a non-zero chance you may find problems. You may also notify us if you feel that you are not qualified to review a paper due to area mismatch, although we trust that most reviewers have the breadth and experience to do the right thing for papers a little bit outside their main areas of expertise.
- Collecting reviews Monday May 27 – Monday June 10
- Ramping up to June 10, monitor status of reviews for assigned papers and politely chase up reviewers to submit on time.
- Solicit new reviewers/reviews if reviewers are not responding or will be late.
- To do Monday June 10 – Monday June 17:
- Lead the discussion with Reviewers via CMT.
- Please note following a successful trial introduced at BMVC2018, we are NOT operating a paper rebuttal period again this year.
- To do by Monday June 17
- Complete the meta-reviews for all your assigned papers. See below for advice on how to write a meta-review.
- We hope that you will be able to manage your time so as to complete your meta-reviews on time.
- We have a quick turnaround of one week for the final paper decision, failure to turn in your assignments on time will potentially delay the entire BMVC review process.
Once again, we thank you in advance for your help in writing thoughtful and well-founded reviews for all your assigned papers. If you have any questions or concerns, please contact us at firstname.lastname@example.org.
David Marshall, Majid Mirmehdi, Bernie Tiddeman and Xianghua Xie
BMVC 2019 Program Chairs
How to write a meta-review
How to Write a Meta-Review (suitable adapted from http://iccv2019.thecvf.com/files/reviewer/HowtoBeanAreaChairforICCV.pptx)
A meta-review is a well-founded argument that a paper does/doesn’t meet the criteria for acceptance. It is not simply a summary of the individual reviews. Especially in the case of difficult “reject” decisions, you must reconcile the reviews and give explicit justification for the final recommendation.
Key principle: would a reasonable author object to the meta-review as the basis for the decision?
Use impersonal language: AC, ACs, authors — not “I”, “you”.
Take care when reviewers don’t agree and especially if your decision goes against the majority.
Meta-reviews for easy cases:
- When all reviewers agree, a couple of sentences will suffice.
- In cases of acceptance, do not mention oral vs. poster recommendation in your meta-review since this status will not be finalized until later by the PCs.
- In cases of rejection, briefly mention the main reasons for rejection and acknowledge the rebuttal (if only to say that it was missing or did not change anybody’s mind).
- “Three reviewers agree that the paper contains technical errors and is not clearly written. There is no rebuttal.”
- “Three reviewers find that the paper is below ICCV standards due to a lack of novelty and insufficient experimental validation. The rebuttal did not assuage these concerns.”
Meta-reviews for difficult cases:
If referees disagree or make borderline recommendations:
- Summarize the main arguments of the reviewers.
- Indicate whether the rebuttal removed any initial concerns (if there was no rebuttal, explicitly state “there was no rebuttal”).
- Summarize any substantive points from the reviewer discussion (authors will not see the discussion).
- Reconcile all the above information and explain the main factors for your decision.
Example of a good meta-review
- Summarize the main arguments of the reviewers
“Initially, R1 and R2 liked the main idea of the paper and gave it a positive recommendation, but R3, who wrote the most in-depth review, found that the experimental results were insufficient. In particular, all the experiments in the paper are on one small dataset [anon]. “
- Summarize any substantive points from the reviewer discussion
“In the subsequent reviewer discussion, R3 pointed out that other datasets provide ground truth, e.g. [anon] and [anon], and also that the comparisons given in the paper contradict the reported results in [anon]. R1 and R2 acknowledged the validity of these concerns, but kept the same rating.”
3.Reconcile the above information and explain the main factors for your decision
“The AC finds the arguments of R3 to be the most persuasive. To properly validate their approach, the authors need to address the discrepancies with prior work and evaluate on the datasets pointed out by R3.”
Examples of a poor meta-review:
- “R1 liked the paper for reason X. R2 disliked aspect Y. R3 disliked aspect Z. Therefore the paper was rejected.”
Good start, but you need to explicitly reconcile the three reviews.
- “Reviewers all say that they are OK if the paper is rejected, and we have many good papers.”
Really bad, reads as though the decision on this paper depended on decisions made on other papers, or as though the AC pressured the reviewers to reject.
- If all reviewers agree that a paper should be rejected, we expect the paper to be accepted ONLY if there are unusual circumstances, e.g., three exceptionally poor or misguided reviews.
- If all reviewers agree that a paper should be accepted we expect the paper to be rejected ONLY if there are unusual circumstances, e.g., a major technical error, fraud, or plagiarism that are not detected by reviewers.
- If you wish to overrule a unanimous reviewer recommendation:
*You must contact the Programme Chairs for approval.
Main Points Summary
- Make a well-founded decision for each paper, considering the substantive points raised by the reviewers and authors. Don’t rely solely on reviewer ratings.
- Write a meta-review that fully justifies the decision to the authors. They should understand the reasons for the decision even if they don’t like it.
- Always be respectful to the authors and reviewers, who put much time and trust into the process.
- Complete your tasks on time!