How to Review Scientific Papers

Peer review is a critical pillar of scientific progress. This page outlines essential guidance for writing respectful, professional, and constructive reviews, largely based on the excellent ICML 2022 Reviewer Tutorial and other community resources. Reviewing is both a responsibility and a learning opportunity for early-career researchers.

Reviewer Philosophy

"Review the papers of others as you would wish your own to be reviewed."
— Mihir Bellare [Video]

• Reviews should create value for both authors (actionable feedback) and the community (quality control, idea shaping).
• Always maintain professionalism and follow publication ethics.

The Review Process

1. Accept invitation, declare conflicts, fill in metadata (CMT).
2. Bid and confirm assignments. Flag conflicts or papers outside your expertise.
3. Phase 1 Review: Read carefully and critically. Fill the rubric. Submit on time.
4. Phase 2 Review: Similar to Phase 1 for second batch of papers.
5. Rebuttal: Read authors' responses. Update your review if necessary.
6. Discussion: Coordinate with reviewers and meta-reviewer to resolve contradictions.

Review Structure

• Start by reading the review form and note the evaluation criteria: Novelty, Soundness, Presentation, Literature.
• Write a clear summary of the contributions in your own words.
• Evaluate each aspect individually:
• Novelty & Significance: Does it advance the field? Is there a real benefit to the community?
• Soundness: Are the claims supported? Are the experiments/designs valid?
• Writing & Presentation: Are the ideas communicated effectively?
• Related Work: Is prior work properly cited and discussed?

Classifying Feedback

• Major issues: Those that question validity or need extensive revision.
• Minor issues: Easily fixable, won’t affect the paper’s conclusions.

Review Checklist

• Constructive, relevant, and detailed comments.
• Clearly separate major and minor concerns.
• Use numbered comments for easy referencing in rebuttals.
• Reference page/line numbers when discussing specific points.
• Summarize both strengths and weaknesses clearly.
• Be respectful: “The work fails to…” instead of “The authors fail to…”
• Be aware of and actively avoid personal or subfield biases.

Daniel Dennett’s Constructive Criticism Rules

1. Re-express your target’s position so clearly that they say: “Thanks, I wish I’d said it that way.”
2. List points of agreement, even if uncommon.
3. Mention what you’ve learned from the target.
4. Only then, raise criticisms.

Further Resources

• ICML Reviewer Tutorial – with slide deck
• ICLR Reviewer Guide + Mistakes Reviewers Make
• CVPR 2020 Reviewer Guidelines + slides
• How to Write Good Reviews - CVPR Mentoring
• Writing Reviews as a Newcomer
• Novelty is in the Eye of the Beholder – Konrad Schindler
• Daniel Dennett's Criticism Rules
• JMLR Reviewer Resources
• Crafting Papers in ML - Pat Langley
• Patrik Reizger's Reviewer Checklist

Last updated: August 2025