Registration and project submissions will be handled by our OpenReview Portal.

You can choose to reproduce papers published in the calendar year of 2022 for the following venues: NeurIPS 2022, ICML 2022, ICLR 2022, ACL 2022, EMNLP 2022, CVPR 2022, ECCV 2022, AAAI 2022, IJCAI-ECAI 2022, ACM FAccT 2022, SIGIR 2022, and also for papers published in top ML journals in 2022, including JMLR, TACL and TMLR.

You have to use the Reproducibility Summary template for writing your reports. The submission is strictly double blind, any violation of this will lead to desk reject. The first page of your report must incorporate this summary template, which should consist of the following sections:

• Scope of reproducibility: Briefly define the claim of the paper that you intend to tackle in this report
• Methodology: Briefly state the overall methodology used in your approach
• Results: Concisely state the results / findings of the reproducibility study
• What was easy: State which part of the reproducibility study was easy to do.
• What was difficult: State the difficulties that you encountered while performing the reproducibility study.

You can find the Reproducibility Summary template here. The first page of the template is mandatory and you can organize the rest of the report as you deem fit. We have also released a recommended outline of the report which you can use to incorporate your findings.

We will use the same OpenReview Portal to submit your Reproducibility projects. Click on “Add Reproducibility Challenge Report” to submit your work. In the field of “Paper URL”, paste the paper URL of the paper (from the appropriate conference proceedings). In the Abstract, copy paste the Reproducibility Summary from your report.

Please check Important Dates for the deadline for submission of your report. Your report should be of maximum 9 pages excluding references, in ReScience style format as specified in Reproducibility Summary template. You can also submit supplementary materials. Code submission is mandatory along with your supplementary materials.

We will be selecting reports which present a clear, well motivated, and thorough reproducibility effort. The report should very briefly discuss the main ideas of the paper and not simply reiterate the main work. Your report should clearly mention the key findings of your effort, and if necessary include discussion / commentary / suggestions to the readers re-implementing this work on its replicability / complexity. Your report must contain the Reproducibility Summary in the first page as defined in the previous section.

Reports will be reviewed on the following criteria. You can also refer to the reviewing criteria used in MLRC 2021 for reference.

• Reproducibility Summary
• Concise Problem Statement
• Reproducible code
• Communication with authors
• Hyperparameter Search
• Ablation Study
• Discussion on results
• Recommendations for reproducibility
• Results beyond the paper
• Clarity of writing