Paper reviewing needs improvements

[jbenito]

Hi,

This is an attempt to summarize the experience of the ACAT reviewers and referees with using Indico's proceedings interface.

* While the "CC to referees" flag is supposedly set for us (since some time mid February), I am not aware of anyone ever receiving any CC from Indico on any email sent to the authors. This flag should anyway be the default as it is the *only* way to know whether Indico has indeed sent out a notification.
* Indico "forgets" the state of the referees judgment and the reviewers comments. This happens without user intervention.
* Many, maybe even the majority of authors, have not received a notification from Indico about the final judgment.
* "Final judgment" sounds final; the naming has caused considerable confusion between referees as to whether this is final final or just an intermediary step to send the reviewer's comments.
* The notification emails that were sent do apparently not contain the reviewer's nor the referees comments; it would be good to include them, also to verify that the reviewer, the referee, Indico and the author agree on what those comments are.
* Formatting of comments (newlines) seems to get lost; itemized lists (like this one) will appear rewrapped to the reviewer (and probably also to the authors, though I have no way to verify that)
* The referee should get notified by email when the reviewer as submitted his judgment.
* The filter criteria "assign status" in "Assign Papers" are misleading: if all are marked it still shows all submissions; I would expect that only those fulfilling *all* criteria are shown.
* The same filter criteria should contain a filter "with material submitted" - that's the absolute prerequisite for deciding what needs action for reviewing.
* We have had a case (id 97) where Indico claimed that the judgment was withdrawn. The notification email has a spelling mistake ("widthdrawn")

So in the end we were unable to use Indico because fundamental ingredients (which authors got notified? did they find the comments?) were unreliable. Instead we had to send private email to all authors, to ensure that they get informed about the status of their proceeding and the reviewers' comments. I.e. as it is I cannot recommend the Indico proceedings system for production use.

Thanks for improving the system - I really hope that it will become usable at some point!

Best Regards,
Axel Naumann

[pferreir]

Some more feedback from Axel:

* Sometimes authors were unable to submit a new revision. I cannot tell whether this was a user issue (e.g. due to multiple Indico identities) or an Indico one; the proceedings in question were id 66 and id 83.
* These new revisions were instead sent by email, outside Indico. It is currently not possible for referees to upload new revisions. It would be extremely useful to add that feature, as without it, Indico claims that it is not aware of a new revision, and thus forbids further "final" judgments. Which means that we now have two proceedings that have their material under "Material" / "Paper" instead of under "Material to review", and that are still in the state "no material has been submitted" instead of "Accepted".
* We have received a few new revisions even after our final judgment. It seemed to be impossible for authors and referees to upload a new revision after the final judgment; this should be enabled (at least for referees), possibly starting a new review round. This means for us that we have several proceedings under "Material" / "Paper" instead of under "Material to review", because we were unable to replace the material for review.
* There is no easy browser path between the paper reviewing section (showing e.g. "Judge as Referee") and the submission reviewing part ("Material to review") etc. I.e. once a submission is opened one has to go back into the conference / paper reviewing, re-enter the filters etc. It would be nice if there is less of a context switch between the reviewing overview and the paper review pages.

[jbenito]

commit: b59e91092e260c35f451cef4b5a035b4ef1d374b
commit: e54d8ebd462daf57730983ac51934ca546d81b9c
commit: 450bb011dc10da0dc0f0f427652393405fea87b9
commit: d86c34800b3915701de949b9295b48c728e4da49
commit: 8b226cd2c12937e97485a2d1852ac74d28ff6856
commit: 0251ef1c72dea7726801913415c6035c2931974a
commit: d9da537c956cdab22e20a123469bc8d43fabedec
commit: 611f81e66e4ab220387602bd346d5628252f0ae6
commit: 68527f6c65171eb62365cdcf94d297698fc7139e
commit: ea509dfb5a26bbce11d3736fe7f9c5a40c191f66
commit: f5f4796ca512a1d792e85ae1c8f5d104b887e453