TL;DR
Other researchers directly criticized a recent publication of ours in a “research article”. Although they raised valid points, they outright lied about the availability of our results. In addition, they did not provide access to their own results. We have published new work supporting our original results, and a direct rebuttal of their critique in a perspective article. The peer reviewers of their “research article” must have been asleep at the wheel to allow the major point, lack of access to our results, to stand.
Original Publication
Back in the summer of 2015, I was second author on a publication (Yao et al., 2015, hereafter YS2015) describing an automated method to characterize zinc ion coordination geometries (CGs). Applying our automated method to all zinc sites in the worldwide Protein Data Bank (wwPDB), we found abberrant zinc CGs that don’t fit the canonical CGs. We were pretty sure that these aberrant CGs are real, and they have always existed, but had not been previously characterized because methods assumed that only the canonical geometries should be observed in biological systems, and were excluding the abberrant ones because they didn’t have good methods to detect and characterize them.
Also of note, the proteins with aberrant zinc geometries showed enrichment for different types of enzyme classifications than those with canonical zinc geometries.
For this publication, we made all of our code and results available in a tarball that could be downloaded from our website. This data went up while the paper was in review, on Dec 7, 2015 (with a correction on Dec 15). Recently, we’ve also put a copy of the tarball on FigShare. Every draft of the publication, from initial submission through to accepted publication, included the link to the tarball on the website.
Critique
Less than a year later, Raczynska, Wlodawer, and Jaskolski (RJW2016) published a critique of YS2015 as a “research article”. In their publication, they questioned the existence of the abberrant sites completely, based on the examination and remodeling of four aberrant structures highlighted in YS2015. To be fair, they did have some valid criticisms of the methods, and Sen Yao did a lot of work in our latest paper to address them.
As part of the critique, however, they claimed that they could only evaluate the four structures listed in two figures because we didn’t provide all of our results. However, we had previously made our full results available as a tarball from our website. As you can see in the below figure, all of the results were really available in that tarball.
<img src = “/img/ys2017_figure1.png”, width = “600”>
In addition, although RWJ2016 went to all the trouble to actually remodel those four structures by going back to the original X-ray density, they didn’t make any of their models available.
Finally, no one from RWJ2016 ever contacted our research group to see if the results might be available.
Response
Follow-Up Paper on 5 Metals
By the time the critiques appeared in RJW2016, Sen was already hard at work showing that the previously developed methods could be modified and then applied to other metal ion CGs, and that they also contained aberrant CGs (see YS2017-1).
Critique Direct Response
In addition to YS2017-1, we felt that the critique deserved separate response (YS2017-2). To that end, we began drafting a response, wherein we pointed out some of the problems with RJW2016, the first being that we did indeed provide the full set of results from YS2015, and therefore it was possible to evaluate our full work. We also addressed each of their other criticisms of YS2015, in many cases going beyond the original criticism, and explaining how it was being addressed in YS2017-1.
Open Results and Code
A major part of the conclusions in YS2017-2 was also devoted to the idea that code and results in science need to be shared, highlighting the fact that RJW2016 did not share their models they used to try and discredit our work, lied about the fact that we did not share our own results, and pointing out some other projects in this research area that have shared well and others that have shared badly, and that the previous attitude of competition among research groups does not move science forward.
Peer Review
Let’s just say that the peer-review of both of the papers was interesting. Both manuscripts had the same set of reviewers. YS2017-1, the five metal paper, had some rather rigorous peer review, and was definitely improved by the reviewer’s comments. YS2017-2, our perspective, in contrast, was attacked by one peer reviewer right from submission, and was questioned almost continually as to whether it should even be published. I am thankful that one reviewer saw the need for it to be published, and that the Editor ultimately decided that it should be published, and that we were able to rebut each of the reviewer’s criticisms.
Finally, I really don’t know what happened in the peer review of RWJ2016. The first major claim was that our data wasn’t available, it should have taken a reviewer 10 minutes to verify and debunk that claim. I would have expected a much different critique from the authors had they actually examined our full data set. But, because of traditional closed peer review, that record is closed to us.
Overall though, I’m very happy both of our publications are now out, and we can move on to new stages of our analyses. Looking forward to continuing to work with my co-authors to move the work forward.
Papers Discussed
- Original Zinc CGs: Yao et al 2015
- Critique of Zinc CGs: Raczynska, Wlodawer & Jaskolski 2016, publisher, sci-hub
- 5 Metal CGs: Yao et al 2017
- Response to critique: Yao et al 2017, publisher, copy on figshare
Reuse
Citation
@online{mflight2017,
author = {Robert M Flight},
title = {Criticizing a {Publication,} and {Lying} {About} {It}},
date = {2017-03-29},
url = {https://rmflight.github.io/posts/2017-03-29-criticizing-a-publication-and-lying-about-it},
langid = {en}
}