Unfortunately, I had a very small time window to review this article, so my comments are fairly high-level, not very thorough, and some may seem overly harsh.
The article highlights very important issues on scientific workflows and reproducibility of results. It also covers an array of relevant topics and technologies, with an aim for openness, linked and shared data/results. The idea to put together an ontology for decenralized workflows with linked data and a robust license scheme is excellent and I can see the merit in such an approach. The article forms a very good basis for discussion for such topics and brings forward important questions about reproducibility, reusability, licensing, and the contrast between what the peer review process ought to be and what it really is today.
The Not So Good:
I felt the paper lacked real focus and results. For me it serves more as a review and an interesting discussion of relevant topics and technologies. It is not clear how the described tool links to everything else and what is the real contribution of this work.
For example, the design decisions are not clearly explained. Why was OPMW selected vs. the myriad of other workflow languages? What was gained/lost from this (and similar) decision(s)?
The questionnaires and evaluation are also not thoroughly discussed. How does the pre-questionnaire determine the next task exactly? Is there a scoring mechanism or do the authors eyeball the results? The link to the post-questionnaire is broken. It is also unclear what the end results of this survey was. How many researchers were approached? What were the results and how do we know they are statistically significant (slight irony here that the authors’ results are not linked unless I have missed the link)? How would this work scale to other areas beyond NLP and ML? If this is just preliminary work, this should be stated more clearly, but also the goals of the work should be more prominently described in the paper.
More importantly, I think some fundamental issues are not touched upon. For example, healthcare research data is much more difficult to share due to privacy concerns. In contrast, other sciences such as astronomy have data and workflows at their core, so it should be easier (in principle) to apply these practices there.
Moreover, usability concerns are not discussed. Even if the data is properly annotated, licensed, and integrated into workflows, an interface that is not very usable may prohibit effective adoption in practice. It would be interesting to see the authors’ plans for future work in terms of further evaluating the tool in more realistic settings and in terms of both functionality and usability/UI.
Relevant publications that might be of interest to the authors:
Y. Gil, P. A. Gonzalez-Calero, J. Kim, J. Moody, V. Ratnakar, A semantic framework for automatic generation of computational workflows using distributed data and component catalogues, J. Exp. Theor. Artif. Intell. 23 (4) (2011) 389-467.