Unfortunately no, as of now we are not accepting reports on workshop papers from the listed conferences. Please work on any accepted papers from the conference proceedings.

Before starting to work on it, please contact the organizers to confirm, as your request will be evaluated on a case-by-case basis.

Typically we do allow it if the paper is published within a couple of years prior to the current scope of conferences. Although, before starting to work on it, please contact the organizers to confirm, as your request will be evaluated on a case-by-case basis.

Any number of team members are allowed.

Not at all, we encourage cross-institute participation! In that case, add the institutions of your team members in a comma separated format in the claims and report submission forms in OpenReview.

No, author order will depend on your final report.

No, you can participate independently on your own. Participation from industry is especially welcome!

Many thanks for your participation! You can just drop us a mail (reproducibility.challenge@gmail.com) with details of your course, and we will list it on our website!

Yes! If your course ends earlier than the submission deadline and you have already graded the assignments, you can directly send us the evaluations! Have your students submit their report in our OpenReview portal, and drop us a mail at reproducibility.challenge@gmail.com with the evaluations paired with the submission links.

Please contact your course instructor or TA to send us a mail (reproducibility.challenge@gmail.com) to register your course. We will update the website periodically and add new courses.

Yes you are more than welcome to! Please consider sharing the word about the challenge to your peers in your company too!

You will first add to the knowledge of the original paper. Peer reviewed reports will be showcased on PapersWithCode and published in the ReScience Journal. Starting this year, we are partnering with NeurIPS conference to showcase the accepted papers as a poster session in the main conference. Please check the following announcement for details.

Check the Resources tab for more information.

Yes! It is highly recommended to contact the authors of the paper you are reproducing, to clarify doubts and implementation details.

You can send the authors mail directly to initiate a discussion. The contact details can be found on the paper, which is linked in the pdf of the paper which is available for each paper.

You can either search the pdf of the paper for the code, or find the link to PapersWithCode page of the paper, which is usually updated with the publicly released code of the authors.

There is no restriction on the extent of the original code you can use for the reproducibility effort.

Yes, the report to be submitted should be double blind. When submitting code for review, include your codebase in the Supplementary Materials, or link to an Anonymous Github URL.

You can find the style files of the report in this template.

Yes! You should use our style files and add the “Reproducibility Summary” in the first page of your report. Make sure this summary does not exceed the first page. Failure of adding this summary will result in desk rejection.

Yes, the Reproducibility Summary counts towards the total page limit (nine pages).

You should copy your Reproducibility Summary in the abstract field. You should add a line separator between sections, and ensure the summary is properly formatted.

Once your report is accepted, you will be required to submit the final draft by camera-ready deadline. Details of this process will be communicated to you after acceptance notification.

Thanks for your interest in our challenge! You can help out by spreading the news. If you are a course-instructor you can help by enrolling your course in the challenge. You can also sign up to be a reviewer when we share the call for reviewing for the challenge! If you are a company you can help sponsor by providing compute resources. Please contact us at reproducibility.challenge@gmail.com to list your generous offer in the Resources section.