The Scientific Paper is Obsolete (2018)(theatlantic.com) |
The Scientific Paper is Obsolete (2018)(theatlantic.com) |
Currently there is way too much trivial shit being published in the ever-expanding number of scientific journals.
Our current model for career advancement in academia is partially to blame for this. Landing a tenure track job requires having a prolific publication record. And once you've landed that coveted ladder-rank position, the pressure to publish only heats up, with tenure on the line. And then even after you've secured tenure, career advancement still depends heavily on the ol' publication record. Pre-tenure the pressure to publish in high-impact journals is immense; it's the kind of pressure that drives otherwise honest people to consider partaking in fraud (and unfortunately those who stick to their morals often lose out to those who fabricate results to some degree). Post-tenure the pressure changes from securing high-impact papers to just getting on whatever papers you can. Review boards for associate professors would more readily give a promotion to someone with 20 meaningless papers over the last three years than someone with 2 papers in CNS journals over the same timeframe, even though a single paper in Cell/Nature/Science is typically more impactful than 100 papers in Frontiers or other similarly dogshit journals. So post-tenure pay raises are based on getting as many words in print as possible, using the least amount of effort to do so.
I almost forgot where I was going with this... right, so, in my opinion we need to switch from a publication-based mindset to a discovery-based mindset. We (the public) provide the NIH with $30 billion dollars per year, with the idea that such an investment will lead to medical breakthroughs, discoveries, and other innovations that can concretely improve our health and wellbeing. However, so much of that is wasted on the idiosyncrasies of career advancement inside the ivory tower.
If I were the director of the NIH my sole purpose would be to end this nonsense. And my first order of action would be no longer accepting grant applications from individual PIs. I would only entertain grant applications from a small force of scientists (4-8 lab equivalents) with thorough and cogent plans for making breakthroughs on cancer or heart disease etc. I would change the minimum R01 funding amount from <$500k to >$10 million dollars. I would change the grant renewal timelines from every year to every 5 years but require yearly progress updates to ensure the proposed experiment were soundly conducted. I would encourage the equal reporting of both positive and null findings. Performance would not in any way be based on positive findings, only that the experiments were carefully run. I would require that all raw data be deposited into publicly accessible repos (not at the end of the study, not yearly, but whenever data is generated it should be made accessible asap). I would encourage that research groups bother with drafting/submitting interim manuscripts (interim meaning prior to completion of the full 5-10 year study) only if they find something important, otherwise just provide a comprehensive writeup at the end of the study. This final writeup would not be submitted to a 3rd party journal. It would be posted directly on the NIH website I'd have created for such results reporting. Naturally this would also be publicly accessible. That would be a start...
"I would encourage the equal reporting of both positive and null findings. Performance would not in any way be based on positive findings, only that the experiments were carefully run."
Would you have considered the LHC experiments "failures" had we not found evidence of the Higgs Boson? I certainly wouldn't have. What matters most is that important questions backed by sound theoretical reasoning are addressed carefully, collaboratively, openly, and with plenty of long-term support. If we do that, we get genuine answers - such answers represent important information whether or not they support our original hypotheses.
As the article mentions, scientific fields are gigantic nowadays, and skimming papers is critical when you're citing 100+ references in your paper.
IMHO, the biggest problem with papers is politics and reviews. In many top journals like Nature there's no double-blind review (actually in Nature it's now optional but big groups never use it). And even if there was double-blind review, referees have no skin in the game. So the usual outcome is to get reviewed by a big name in your field, who is actually interested in controlling research trends and killing "competitors".
This is hindering progress and hurting new ideas. For example, proponents of Alzheimer's disease being caused by an infection or dysbiosis have had a hard time to do research, get grants and publish articles during the last 2 decades. Despite their theory is able to explain the etiology quite well, unlike competing alternatives.
Another problem is that to publish in good journals you need cool results. Cool results are rare, but Nature, Science, Cell et al. are full of articles every month. So, most groups are overselling and misreporting things. Research fraud, p-value hacking and data manipulation are really common.
That's a big problem. I just got a paper rejected. Reviewer 1 was just focalized on a single detail I mentioned somewhere, not central at all in the paper yet is basing most on this criticism around that. Reviewer 2 has difficulty understanding a table containing 2 columns and 3 rows, and what means N, V, ADJ and ADV in a paper about dictionary (not to mention the same abbreviation was used just before, and used a plain words numerous times in multiple paragraphs). Reviewer 3 is the only one saying a remotely nice thing and who seems to have grasped what the paper is about. There is of course some valid criticism raised in the reviews, but half of it is bullshit that would be dispelled in a more interactive process or/and if reviewers had incentives to actually put a minimal effort to understand the paper.
If you do it right, the code should in no way interfere with your ability to read abstracts.
If I publish a paper saying I have an algorithm which can factor large composites, and in the paper publish the factors to all of the RSA numbers listed at https://en.wikipedia.org/wiki/RSA_Factoring_Challenge , then I think people will take it seriously, and not consider it at the superficial level.
Even if I don't publish the algorithm. ("Because of the security implications of this work, I have decided to withhold publication for a year.")
Furthermore, some things are worth publishing even if the methods was "it came to me in a dream" à la Kekulé's snake. If you can demonstrate a sorting network of size 47 for n=14 input (which is the known lowest bound) then you can publish that exemplar, even without publishing the method used to generate it.
(If you used computer assistance then that method would likely also be publishable, but that's a different point. Newton famously used the calculus to solve problems, but published their proofs using more traditional approaches.)
If you can come up with a protein model that is a significantly better fit to the X-ray diffraction data, then that's publishable too, no matter how you came up with that model.
In all of these cases, there are ways to verify the validity of the results without reproducing the methods used to come up with the result.
Nothing prevents you from having two-column notebooks, if you find that advantageous, as well as abstract and conclusions sections. The part that you don't get with static paper is that of navigating the abstraction ladder[1] up and down with direct manipulation aids, instead of having to work it all in your head or by following dense detailed paragraphs.
[1] As also explained by Bret Victor in http://worrydream.com/LadderOfAbstraction/
I learned a lot and it was definitly worth it. The next paper will be easier with this knowledge. Nonetheless, there is an overhead and I feel that this overhead is not valued with the current makeup of journals, where you really need to dig deep to find any supplementary materials.
My supervisor manually copied all of the text from my PDF into a word document on his first revision ...
Readability and scalability is about making all this data available in the publication record, but easy to navigate for whoever is looking for whatever.
And not every paper has a lot of code or data associated with it. If you do experiments on organisms etc. then there is so much happening in the actual lab work - where would that go? Endless hours of video documentation?
Isn't that the practical part about digital technology? That you are not limited to one view?
To make matters worse, this is an evolved trait: researchers whose papers are intimidating are more likely to succeed, which means they're more likely to have future PhD students, which means that the style of writing is more likely to get passed on.
I think the main way to address this is to change the incentives. In particular, by creating publication venues that value simplicity and clarity (one such conference is SOSA, which has had a lot of impact on theoretical computer science in the last few years).
Ah, a fellow economist lol. Lack of clarity is a strategic advantage because (1) (as you said) it looks impressive and (2) it's hard to validate that it's correct.
So many papers contain such elementary statistics mistakes such as survivorship bias, e.g. 'returns to education' is almost exclusively measured by asking individuals who graduated (on average 50% of enrolled students don't) and respond back to surveys (good chance of bias).
Pubs are how you get jobs. It's not about science anymore, it's about navigating bureaucracy for an elite job.
Reminds me of the old "there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies."
I'm thinking about the success of Freakonomics, Thinking Fast and Slow, The Elegant Universe, etc. These are all academics, who've "translated" their research for the masses. That translation ended up being much more impactful - and prestigious - than an intimidating paper.
I hope this becomes more of a trend, and an incentive structure, in the future.
It’s been done and it is amazing. This is the best journal in the world imho
It's not hard to imaging a better UX - for the field what are the top 5 questions you want to answer before you start reading?
Eg: Sample size, Funding, etc. Put those at the top of the paper with symbols.
None the less I wonder if the goal should be to make it so any person could understand a complex paper? Should all people strain to understand every study? There are experts in certain fields for that very reason. It is not always possible to accurately explain higher level concepts to people who lack foundational knowledge that can take years to accrue. I am not certain if changing papers to be more interactive is going to bridge the gap as this author hopes, or if it is even the goal that should be pursued.
And he has all his articles listed on his site- https://jsomers.net/
We're trying to figure out how to facilitate taxonomists publishing their own taxon pages, i.e. species descriptions, from a science 250+ years old. Our MVP use case is ~20k pages, one per species, for one project. There will be many of these projects, though maybe not many with 20k pages, and some with much greater than 20k pages. Updates are needed with as little latency as possible with data from from multiple sources. There has to be basically zero cost to serve these (I know, nothing is free). Sites must be trivially configurable (e.g. clone a GH template repo and edit a YAML file and some markdown). Even if we can get this in infrastructure in place we then have to figure out how to get the social structure in place to have this type of product recognized as equivalent to traditional on paper publishing, i.e. advance people's careers because they "published".
In my field, until we give people the power to publish on their own, I don't see traditional publishing go away. Many in the past have indeed published (traditionally) their own species descriptions on their own dime, meeting the rules within the various international codes of nomenclature. I also don't have a problem with dead wood- if we go digital too fast we will loose so much for any number of reasons associated with the ephemeral nature of electron-based infrastructures.
At least after I transfer them to a single column, 14/16pt tall font with real headers, that is.
The graphic format itself is dated and annoying, yes, but I find the expositional tone and immediately searchable references pretty cool.
Redesigning the Scientific Paper - https://news.ycombinator.com/item?id=16764321 - April 2018 (107 comments)
There is little effort in making your results understandable and easy to replicate. Academia values paper production, which requires convincing peer reviewers that your results are not trivial and are worth publishing. Contrary to what the essay states, I don't think many scientists today think their research is "incremental". In fact, this word is used in many places as a derogatory term to indicate certain result doesn't contain enough novelty to deserve publication. Researchers are more incentivized to make their constructions and results as complicated and less accessible as possible.
This is not just a theory, this is something I've seen over and over throughout the years.
The thought clusters emerging from the recent “replication crisis” are a fascinating rabbit hole to crawl into. If you stay near the surface, you will find mostly young scholars cheerleading open science as the obvious solution to replication difficulties. The concepts of pre-registering your study, committing to sharing data, and publishing online are all various components of this idea, varying in their necessity by the author’s devotion to their cause.
But there are several downsides to such a system that aren’t immediately obvious. For example, does the skill set of the successful scientist broaden to include how skilled they are at poaching ideas from public data that wasn’t immediately seen by their authors?
Some of the more recent criticisms invoked the spectre of “platform capitalism”, and suggested the Facebook and Linkedin-ification of science by dumping all its data on a centralized platform would likely have a net negative effect.
This article was written in 2018, and most of the discussions I’ve read since then have suggested that the open science initiative has failed despite the rapid penetration of Jupyter and visualization tools in the scientific process. Perhaps, like most things, the unseen market will pick and choose the good out of the dubious.
I read the abstract, in reverse order. Discussion and Results first.
If it seems plausible ( much research is trash, churned out to pad a CV ) and interesting, then I scan the Methods section.
This heuristic helps classify 95% as utter trash or outside my current area of interest in less than 10 seconds.
If it seems seriously interesting, I then read the full text the same way ie: backwards.
At no point do I want pretty visualisations made by wannbe PhD candidates, full of misinterpretations, wishful thinking or outright fraud.
See Figure (5). Your argument doesn't really counter any part of the scientific notebook. A notebook will still have the abstract and conclusion (result & discussion). The tools mentioned in the article describes how to restructure the methods, data, and figures. You're note going to look at these anyways until the abstract and conclusion intrigues you.
This is not even close to true. Look up Tycho Brahe's observations, and he's only the earliest I can think of.
For this we can use cloud based environments controlled by funding agencies/universities that ensure every interaction with data is recorded from the very beginning.
Something like this would at least reduce the risk of p-hacking practices that would otherwise be there even if everyone used notebooks instead of papers.
But the article is actually about Mathematica vs. Jupyter notebooks. Still, it's well researched and very interesting.
Nevertheless, the question how to publish better remains open. I for one think that some progress could already be made if ArXiv published html articles by default, rather than those unwieldy PDFs that really only work best when printed on paper.
This is not too unlike maintainable code. The code platform itself matters to some extent, but far less than the extent to which the author wrote with maintainability in mind.
I was approached by a few academics about publishing what I'd worked on, but I never did. I never did because I did consume large stacks of papers every month, and I absolutely hated the pompous, obfuscated portioning out of ideas fragment by fragment. It was an unnecessarily time consuming, and often quite useless way of sharing information. Especially since source code often wasn't part of what was published so a lot of important information got lost (which I guess was the entire point of not publishing code).
I particularly remember a 4-5 page paper that was so poorly written it took me a couple of readings (weeks apart) to realize that it described something I too had worked on. How bad is a paper when it is so obfuscated that it takes effort to recognize something you have worked on too?
I wasn't interested in wasting time dressing up my notes in drag. And if my notes as they were were not good enough, well, then someone else would surely do the same work independently and publish something at some point. Lots of the things I worked on inevitably were described by other people.
I have a love-hate relationship to scientific papers for the simple reason that they sometimes aren't really about science, but about scoring points in academia and certain types of research organizations. Yes, a lot of interesting goodies are published, but my god there is a lot of garbage that gets published. Not least because people in academia are incentivized to get as many papers as possible out of what ought to be a single publishable unit.
If we incentivize authors to spam us, they will spam us.
regarding notebooks themselves, i feel like they're a high concept idea but i've yet to see them really click for me in practice. i find the small cells for code to be extremely unergonomic and that the interspersal of code and plots to be distracting from both the code and the plots (although pretty fantastic for demonstrating high level features of a library, programming language or environment).
on a more fundamental level, i completely agree that mathematical notation is lossy, and that it takes a lot of skill to go from some arcane notation to an actual sense of what the relationships are- but, it requires no specific functioning technology to do so. i can review a paper from 100 years ago and understand it, where running a computer program from 20 years ago can be a challenge at best.
i think that additional high touch experiences for data exploration and teaching are fantastic ideas, but i also think that maybe the base level of communication should be kept simple; both for the purposes of maintaining accessibility and history. where the linux kernel developers insist on 78 column listservs, maybe scientists should insist on camera ready documents when it comes time to share.
i think that everyone agrees that better science would come from full data and code being supplied with publications, but interop is quite difficult as-is keeping code alive. i suppose the big question is: does it make sense to move science towards how software is done, where every bit of code is actively maintained over the years to avoid code rot, or does it make sense to come up with a scheme of freezing and archiving computing environments used in science so those in the future may be able to reproduce results or errors as they see fit. (something like, every paper must ship with a vm image for a widely available architecture that includes no proprietary code and all data used for results)
interesting questions. how to fundamentally change scientific communication such that it is enriched with data and code properly is a harder/organizational problem that i think many have tried to solve (not to mention how this ties into another problem in science- idea validation/replication and knowledge rot). building software systems for exploration and data analysis (ie; computer as partner in exploration) sounds much more fun and likely to produce useful results!
If you haven't yet, maybe look at Andy Matuschak's "Why books donʼt work" [1] and " How can we develop transformative tools for thought?" [2] which connect these same ideas to education in general.
[1] https://andymatuschak.org/books/ [2] https://numinous.productions/ttft/
They do. First, just like with everything else there are brilliant, good, mediocre, and outright poorly written books. Among those some may work for you, others miss the mark completely depending on your prior experience and background (as the author rightly notices, books are just a medium). Second, did the author expect to become a domain expert after finishing a single book? Clearly, his expectations are unrealistic then. You start somewhere, then use references to deepen your knowledge. That's a task requiring interest and dedication, but no "several lifetimes of research" (of mnemonics and learning methods) as he puts it, will replace that.
The last thirty years have changed that game to basically no longer involve printing (other than for a very few select publications) and basically switching to a digital only publishing form.
I completed my phd, read thousands of papers (well skimmed mostly, it's a fine art to zoom in on the relevant stuff), all without visiting the library more than once or twice. I only printed the ones actually worth reading in detail. And these days I consume vast amounts of information on screen without ever using a printer. My printer is fifteen years old and I just installed the second toner cartridge I ever bought for it.
Yet we still pretend to have "journals" like we're in the 19th century. It's the equivalent to writing your friend a letter to inform them that your train is delayed by 5 minutes. Most sane people use some kind of instant messaging tool for that. Writing letters of course used to be a primary way to communicate for scientists. That too has stopped being a thing. People use email now.
The whole point of publishing is to convey information in a form that's convenient to the reader and to solicit endorsement from your peers (via peer review). Peer review used to be implied by virtue of an editor choosing to select a certain paper for publishing. That in turn implies they would have consulted a number of peers about the suitability of that paper. It's sort of the super tedious equivalent of a soliciting a thumbs up button in a social network.
If you publish on linkedin because you are some kind of wannabe influencer you basically need to get people to 1) read your stuff and 2) click the like or share button. Scientific publishing basically is not that different. You have wannabe scientist that want to get the attention of the influencers (reputable peers) so people will be convinced they know their shit. This ultimately translates into degrees, research funding, and tenure track positions. The whole process is kind of biased towards metrics because that's how universities choose to allocate their money.
An ambitious scientist behaves basically in a similar way as a linkedin influencer and will try to game the system by flooding the system with a lot of content and getting their buddies to sign off on it. There are a lot of mediocre articles that get published in obscure places with cliques of scientists basically doing each other favors by referencing each other's work; or worse self referencing. In linkedin terms, this would be the absolute drivel that nobody likes that gets re-shared by a few people that also don't manage to produce much content of interest.
So, here's a thought, maybe get this a bit more out in the open and give scientists some modern tools to endorse each other's work. The best endorsement is a reference. A link basically. Tracking links between bits of paper is super tedious. These papers need permanent URLs. And they need to be digitally signed by their authors so we can have some authenticity and prevent cheating. And scientists need a place where scientists can debate and exchange thoughts about these papers. That used to be a big tradition between scientist back when they still wrote letters to each other or used journals to criticize each other's work.
Curating and aggregating work by means of linking to it is a job that should not be reserved for fussy editors of non paper based journals that absolutely nobody ever reads cover to cover. HN for science; why not? Why not have a multitude of websites referring, editorializing and commenting on published work? How is that not a thing?
Sincerely, someone who reads a lot of research but contributes none because I’m an amateur.
Edit: when I say data, I mean the raw data.
But the code is secondary to the idea. The idea and the discussion around how it was arrived at and what it means is the key thing. The code is just there to implement it. You could code the same idea ten different ways.
> but contributes none because I’m an amateur
I don't mean to be rude but it seems relevant to point out. Papers aren't written for the benefit of amateurs. They're written for experts who actively work in that specific field. I don't think there's anything wrong with that.
For many journals and conferences there isn't even a way to submit the code or other digital artifacts with the PDF. A few have badging for whether digital artifacts are provided and whether the results have been reproduced or repeated by others - steps in the right direction at least.
As much as I intensely dislike their practices of overcharging for journals and milking digital library subscriptions to fund administrative overhead, the technical societies are technically non-profits and exist to serve their members and the research and professional community. This is really something they should be doing.
This part really downplays the considerable resources (ie time contributed by unpaid volunteers) required to do artifact evaluation.
Why not to stuff it into a repo and reference it in the text?
Of course, a central location within a stable institution helps with continuity of availability of such repo. But at least this gives you some control over it.
> version-controlled Jupyter notebooks
That's awfully field specific. It probably wouldn't work for most of STEM. Even for ML I shudder to imagine trying to make sense of the inevitable monstrosities. Writing a paper is part of the thinking process. It forces the author to sit down and work through things in an orderly manner and they're still often difficult to read.
I'm definitely in favor of all papers being accompanied by working source code when relevant though.
As a former academian: Papers are difficult to read primarily because the academic community does not value making them easier to read - no other reason. You may hear things like "papers should be written for other experts", but even that doesn't hold up to scrutiny.
They typically spend 99% of their research time on the actual research, and less than 1% on writing the paper. They definitely can afford the time and energy to make the papers easier to read, but game theory holds sway: Why should a particular researcher use his/her time to do it, when his/her peers will not appreciate it? It's purely an internal, cultural problem. There are no external constraints leading to this.
I've seen referees send papers back saying they contained too much explanation, and suggest leaving out most of the details - just include the big picture methodology and show the results. I can guarantee most who will read the paper would not be able to reproduce those details, if they ever want to. Likewise, I've found papers where I couldn't reproduce the results, because the results were wrong - but since including a derivation of your final expressions is discouraged, no referee caught the errors.
Explorables have all the same requirements to carefully think them through and prepare them for reading and clarity of exposition, but they also have the interactions that ease the introduction of concepts to their readers through a hands-on approach (rather than forcing them to read the mind of the writer by reverse-engineering their thought process, by running in your short-term memory the examples given in a non-interactive paper).
Nobody is preventing scientists from publishing code and data in addition to & before the paper, which imho itself should be as conservative in format as possible to provide the most universal baseline for understanding, reproducibility, and reliability.
Tools like Zenodo [1] are meant to solve this exact problem, and ensure these kinds of data don't suffer web decay.
It's sort of code, but more convenient for just getting work done. You can pass around the whole thing (data + code). No need to learn software development skills or set up a development environment to replicate results; it runs everywhere. Plenty of power to do advanced things. Already a standard in research.
My observation, from seeing papers that have been written in Jupyter, and observing how people work, is that Jupyter will first gain traction in disciplines that are already computation-heavy, and where open software is closer to the front end of the data pipeline.
For instance in my case, I develop measurement instruments, so everything I make is computerized, by me or my colleagues. While "raw data" may be in the form of things like voltages, they are almost immediately turned into a Python friendly data format by code that I wrote myself. So I'm up to my armpits in data and code just to get my experiments even barely working in the first place. I have a computer with coding tools literally at every bench in the lab. Jupyter is my lab notebook, and often my "report" is just the same notebook, dressed up with some readable commentary.
Now, contrast that with somebody like a synthetic bench chemist. The data that they get may be in computer readable form, but they rarely do any coding during the course of a project. For analysis, they're satisfied with the computations rolled into their instrument software, or Excel. And a fair amount of their analysis is in the form of explaining their way through an argument that connects data from disparate measurement techniques, using pictures and graphs. They don't program. The ones who can program have gone into software development. The ones who are using Jupyter are motivated to use it, as an end unto itself. Bringing that stuff together in Jupyter wouldn't help much. Many of their journals do require submission of raw data.
This is similar to questions about why so many people use Excel. I think you have to actually immerse yourself in the specific work environment an observe or even experience what people are experiencing, what they're actually studying, how they think, and so forth. There's a certain Chesterton's Fence aspect to discussions that start with the premise that some widespread activity is hopelessly broken beyond repair and must be immediately abolished.
What would help me is to have the old geezers consider GitHub issues, PRs, and commits as a type of citation and to have a better way of tracking when my code gets used by others that is more detailed than forks.
I also think citations of your work that find errors or correct things should count as a negative citation. Because otherwise you are incentivized to publish something early and wrong. Thus the references at the end of the paper should be split into two sections: stuff that was right and stuff that was wrong.
This may vary based on discipline, but in both the subdisciplines of experimental and theoretical physics I was involved in: No - very few will provide the data/derivation. My professors were very open about this: They don't want to lose their competitive edge. Almost no experimentalist I knew could take papers from his/her field and reproduce the results, because the papers lacked enough detail to do so. They would mention a technique, but there are lots and lots of nuances involved when building equipment to carry out the technique[1], and these are intentionally excluded. It's unlikely you'll be able to build the equipment the same way the original authors would.
[1] Most experimental physics involves building your own equipment, or at the least modifying existing equipment.
From someone who isn't an academic, isn't this letting politics come before science?
> We do expect that authors leave citations to their previous work unanonymized so that reviewers can ensure that all previous research has been taken into account by the authors. However, authors are required to cite their own work in the third person, e.g., avoid “As described in our previous work [10], … ” and use instead “As described by [10], …”
However, it is true that things like choice of research questions, approach, and equipment used can be quite suggestive of the authors' identity.
[1]: https://chi2020.acm.org/authors/papers/chi-anonymisation-pol...
And I don’t think it’s rude, that’s why I included that statement!
His expectations aren't unrealistic, the methods he's suggesting have decades of research suggesting to learning that's much more efficient than traditional reading.
Says who? I say he's doing it wrong.
> the methods he's suggesting have decades of research suggesting to learning that's much more efficient than traditional reading
Where are the results of that research then, the culmination in the form of medium superior to books? His mneumonic quantum book doesn't look like one. People need understanding, not memorization.
It doesn't, and it is a big "trust me". People review papers based on the merits of the idea and methodology, and then tend to trust the results. Of course, if the results are very "significant" (e.g. cold fusion), then it will be scrutinized more, people will fail to reproduce, and they will harass the author. 99% of papers don't fall in this category, though.
> isn't this letting politics come before science?
Yep. The games at play are often: "How do I write my paper in the most convincing way?" and "As a referee, this paper is hurting the research work I am currently doing. What is the best way to reject this paper?"
The extremely annoying part was I felt I was back to taking literature courses, where I'm graded on very subjective metrics. It was horrible, especially when all my work was extremely objective. However, the publishing system is not incentivized to be that objective.
Simple example: A colleague's paper was rejected because he explained a phenomenon using method A, and the referee complained there was no mention of method B. Method B was the hot topic of the day. Neither method A nor method B had good empirical data to support it - it was almost purely theoretical at that point. But that community was gravitating towards method B, and really did not want to see alternative explanations.
This isn’t anywhere close to my experience. 2-3 days of writing per year seems like a wild underestimate for any academic I know. I’d maybe believe only 20% of time spent writing, but for some folks even that’s probably way too low
And fine, even if 1% is an exaggeration, I doubt they spend more than 5% (about 18 days of the year).
I do think papers nowadays need to include a link to a zip file (or whatever other format - but it should be a boring old format unlikely to change or be abandoned, and not proprietary either) including all data, code, and so on. This data is necessary to verify the paper's results, but it is not the results themselves.
This is consistent with what I said: Papers are difficult to read by choice. Yes, they strive to make it as concise as possible, which translates to making them harder to read.
Where I would disagree is the claim that it is done to make it as quick as possible to read by experts. In my field, the experts would skim papers quickly to get an idea, but if they then honed in on a paper to actually extract the meat of it so they can use it in their own work, it would take a lot of time, and was a pain.
I've heard from math professors that it takes about a day to read and digest one page of a journal article in their field.
Also disagree on it merely being a matter of consulting references. My work was theoretical/computational. It was common to see the final equations without any derivation, and many experts would not be able to reproduce it. There are lots of tricks that go into the derivation, but they are not provided under the pretext that any expert should be able to solve the equations and derive them.
And in the day of digital media, it's quite trivial to write a paper the way you suggest, and then put all the extra details in appendices. I guarantee that they will be read by most people who want to read the paper in detail, as opposed to merely skimming it.
This is because the datasets were subscriber logs from mobile operators. They are both highly privacy sensitive and contain sensitive business knowledge. There is no way they will ever get published, even in some anonymized form.
Ultimately it always comes down to trust. You need to convince your peer reviewers to trust you that you have correctly done what you have claimed to have done. Of course, even when you publish datasets, you need to convince the peer reviewers to trust you that you didn't fake the data.
I like the idea of publishing the data with the paper but it's not feasible in every case.
In my case I'm currently finishing up a paper where the raw data it's derived from comes to 1.5 PB. It is not impossible to share that, but it costs time and money (which academia is rarely flush with), and even if it was easy at our end, very few groups that could reproduce it have the spare capacity to ingest that. We do plan to publicly release it, but those plans have a lot of questions.
Alternatively we could try to share summary statistics (as suggested by a post above), but then we need to figure out at what level is appropriate. In our case we have a relevant summary statistic of our data that comes to about 1 TB that is now far easier to share (1 TB really isn't a problem these days, though you're not embedding it in a notebook). But a large amount of data processing was applied to produce that, and if I give you that summary I'm implicitly telling you to trust me that what we did at that stage was exactly what we said we'd done and was done correctly. Is that reproducibility?
You could also argue this the other way. What we've called "raw data" is just the first thing we're able to archive, but our acquisition system that generates it is a large pile of FPGAs and GPUs running 50k lines of custom C++. Without the input voltage streams you could never reproduce exactly what it did, so do you trust that? Then you're into the realm of is our test suite correct, and does it have good enough coverage?
I think we have a pretty good handle on one aspect of this, is our analysis internally reproducible? i.e. with access to the raw data can I reproduce everything you see in the paper? That's a mixture of systems (e.g. configs and git repo hashes being automatically embedded into output files), and culture (e.g. making sure no one things it's a good idea to insert some derived data into our analysis pipeline that doesn't have that description embedded; data naming and versioning).
But the external reproducibility question is still challenging, and I think it's better to think about it as being more of a spectrum with some optimal point balancing practicality and how much an external person could reasonably reproduce. Probably with some weighting for how likely is it that someone will actually want to attempt a reproduction from that level. This seems like the question that could do with useful debate in the field.
certainly sharing apparatus is hard but you could release the schematics, board designs and BOMs of the electronics involved.
The problem now is that 1) very few even try to reproduce 2) very little money is available for reproduction
fixing those incentives would help alot.
My argument is that there are nuances and subtleties that are often omitted in a paper (accidentally or otherwise), but are nevertheless required to reproduce the research.
My understanding is the X-ray gives you a diffraction pattern which is hard to invert to a structure, while if you have the structure the diffraction pattern is easy to compute. The diffraction pattern therefore gives you a way to verify that one model is a better fit than another model.
It may not be perfect, certainly not. It might not even be correct once more data arrives. But if you predict a novel fold, and that fold matches the diffraction pattern significantly better than the current model, then it doesn't matter how you came up with the new fold, does it?
It could have been a dream. It could have been search software. The result is still publishable.
All of what you have said is true, but my point is for some research being able to verify the correctness of the result is all that matters, not being able to reproduce the research.
Can you reproduce Kekulé's dream?
However, others (myself included) see the the communication of methods as a primary function of the literature, because this is what enables others to understand, critique, and build upon the idea.
Generally you are expected to explain what you did in enough detail that the reader can replicate your experiment. If you're fitting a protein model to X-ray diffraction data, you aren't expected to include all the other protein models you considered that didn't fit, or explain to the reader your procedure for generating protein models, but you are expected to explain how you measured the fit to the X-ray diffraction data (with what algorithms or software, etc.) so that the reader can in theory do the same thing themself.
The result is still the same - a novel fold which is a significantly better fit than existing modules, based on measured vs. predicted x-ray diffraction patterns and whatever other data you might have.
Which is publishable, yes?
When the Wikipedia entry at https://en.wikipedia.org/wiki/Foldit says "Foldit players reengineered the enzyme by adding 13 amino acids, increasing its activity by more than 18 times", how is that much different than "A magical wizard added 13 amino acids, increasing its activity by more than 18 times"?
Or "secret software".
What's publishable is that the result is novel (and hopefully interesting), and can be verified. The publication does not require that all step can be repeated.
Unfortunately we have a long way to go to make it easy to repeat the calculation that a novel structure is "a significantly better fit than existing modules, based on measured vs. predicted x-ray diffraction patterns". (If I run STEREOPOLE and it says the diffraction pattern from your new structure is a worse fit, is that because I'm running a different version of IDL? Maybe there's a bug in my FPU? Or the version of BLAS my copy of IDL is linked with? Or you're using a copy of STEREOPOLE that a previous grad student fixed a bug in, while my copy still has the bug? And stochastic software like GAtor is potentially even worse.)
This is something we could and should completely automate. There's been work on this by people like Konrad Hinsen, Yihui Xie, Jeremiah Orians, Eelco Dolstra, Ludovic Courtès, Shriram Krishnamurthi, Ricardo Wurmus, and Sam Tobin-Hochstadt, but there's a long way to go.
And even in this exceptional case, the algorithm itself is interesting above and beyond the fact of its existence.
If some factors of those numbers are also large composites, without access to a good algorithm, nobody can truly verify your claims.
If not and you include all of those factors in an easily digestible way for computers to process (let's call that "code"), it will be easy for anyone to reproduce your results (run that code which multiplies all the factors and gets the resulting RSA numbers).
With code, they could easily check that there's not an error in your verification method too (eg. large number multiplication broken).
This would achieve both goals: you'd withhold your algorithm for security reasons, and your results would be easier to verify.
Edit: but to be honest, I think withholding the research is a bit of a special case. You are doing it on purpose, and you can easily offer a service to prove your algorithm works (eg. imagine a "factoring" web service that instantly gives you a hash of the resulting sequence of factors, and then only mails you the actual sequence in two days).
“But the proof is secondary to theorem. The theorem and the discussion around how it was arrived at and what it means is the key thing. The proof is just there to show it’s true. You could write a proof for the same theorem ten different ways.”
Which is all true. But man it wastes so much time having to re-prove everything. Also some lemmas/theorems are so hard to prove. It’s much easier when you see some incredible statement and can’t believe it’s true to look at the proof and see where the mistake / contentious part is.
If anything, I'd like to see more focus on giving evidence of generality of a result, vs just sharing everything needed to get back the same specific result
You're saying that the majority of CS papers only appear to work because the analysis code has bugs?
And that checking the code (presumably also the analysis code) is easier than "understanding the idea"?
Neither of those ring true to me, but your mileage may vary.
I don't think any specific field was mentioned
https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Alleg...
It's not like it takes a whole lot of time to just dump your code in a github repo once you're done and link it somewhere on the paper (if you wrote code at all while working on the paper).
Sometimes I did just want to run my own experiments with different datasets, and those algorithms aren't always trivial to implement :|
Yep but we should still show we did actually simulate our idea, and the methodology that gave rise to the simulation. Not because of the code but to test at all a simulation we describe actually outputs what we propose
Not everyone is a programmer but they could find one to confirm, or better yet, invalidate code my team relied on
Strong disagree. Given how much influence colleagues can have over one another's career prospects, how petty academic disagreements can get, admin focus on metrics like citation count, and how it's easier to prove someone else wrong than to do your own original work (both have value, one is just easier), it would end up with people ONLY publishing 'negative citations' (or at least the proportion would skyrocket). I think that would be bad for science and also REALLY bad for the public's ability to value and understand science.
> Thus the references at the end of the paper should be split into two sections: stuff that was right and stuff that was wrong.
This, on the other hand, is brilliant and I love it and want to reform all the citation styles to accommodate it.
Organizationally speaking, Reddit is a dumpster fire; check out the 'search' function (I'm just speaking on a taxonomical/categorization perspective, I can't speak to their dev practices).
Academic papers aren't. (They're a dumpster fire in their own ways: The replication crisis and the lack of publishing negative results comes to mind, but damn if they aren't all organized!)
There's two key differences:
1.) Academic papers have other supporting metadata that could combine with the more in-depth citation information to offer clear improvements to the discovery process. Imagine being able to click on a MeSH term and then see, in order, what every paper published on that topic in the past year recommends you read. I also think improving citation information would do a lot to make research more accessible for students.
2.) Reddit's system lets anybody with an account upvote or downvote. Given you don't even need an email address to make a Reddit account, there's functionally zero quality control for expressing an opinion. For academic publications, there is a quality control process (albeit an imperfect one). If only 5 people in the world understand a given topic, it's really helpful to be able to see THEIR votes: If they all 'downvote' a paper that would suggest it's wrong.
I think it'd be hard to math out right now since the skill doesn't exist in the departments who'd be doing the work, but in 20 years who knows?
I've seen stuff like this said before but I don't think it would work. Most citations are mixed in my experience. A few objections, a bunch of stuff you aren't commenting on, and some things you're building on. Or you agree with the raw data but completely disagree with the interpretation. Others are topical - see <work> for more information about <background>. Probably more patterns I'm not thinking of.
'We' meaning the librarians and archivists. You guys actually researching have more than enough to do.
I'm not sure if it would prove feasible in practice. It seems like it would aid the writing process in some cases by helping the author keep track of details. But in other cases maintaining all that metadata would become too much of a burden while writing, so it would get put off, and then it would all fall apart.
Very interesting to think about!
As a geezer myself I am imagining what this type of request would look like in less than 5 years from now.
"What would help me is to have the old geezers consider Tiktok videos and replies as a type of citation"
i.e. if you open this particular can of worms for a very restricted subset of users (not only programmers but specifically programmers who also happen to use github), you have to open it to everyone else. I am sure plenty of Youtube research qualifies as "citation" if you start counting Github commits.
Papers qua papers aren't the goal. The idea is to advance our collective understanding of a field. Papers are certainly a means to that end, but other things can too, like code, videos, and blogposts, even if they don't fit into the "6000 words and 6 figures" box.
I get that citations and citation metrics feel objective, but they emphatically aren't great measures of research/researcher "quality".
Definitely don't want to encourage papers to take even LONGER.
For example the paper on polymorphic inline caching, which is the key idea for the performance of many programming languages today, just described the idea, and didn't present any code. How was it evaluated? People sat and thought about it. Holds up today.
You can reason about an idea through other things than concrete code. Code is transient and incidental. Ideas persist.
Likely the tech stack you use is built on a tower of ‘just idea’ papers.
You are right. But that is why we are having this discussions, so we can improve situation.
Having even bad code (and corresponding data) available is always better than not. You can always just ignore it, and read the papers like today.
Honestly I am ok with just zip file of project directory that you have anyway, with hopefully list of versions of os, libs and programs used.
We could do a lot better than just a zip file, but that would be a nice start.
If you want to be that broad about it, science journals publish a lot more than just method development, including obituaries and opinion pieces on where funding should be directed.
Here's a famous paper showing that "Euler's conjecture on sums of like powers" is incorrect - https://www.ams.org/journals/bull/1966-72-06/S0002-9904-1966... . I will repeat the body in full:
> A direct search on the CDC 6600 yielded 27⁵ + 84⁵ + 110⁵ + 133⁵ = 144⁵ as the smallest instance in which four fifth powers sum to a fifth power. This is a counterexample to a conjecture by Euler [l] that at least n nth powers are required to sum to an nth power, n>2.
Do I need to know how the direct search was carried out to confirm Euler's conjecture was false?
No.
>>> 27**5 + 84**5 + 110**5 + 133**5 == 144**5
True
Just because you prefer one sort of scientific research doesn't mean other forms aren't science.
Again, is Kekulé's model of the benzene ring less scientific because it came to him in a daydream?
We accept Newton's publications where he secretly used the calculus, even though he didn't publish the calculus, because they could be proved through other more laborious means.
Why is it not scientific to write publications which use secret software, so long as we can verify the results?
For the Kekule paper [1] there is a significant amount of information about the context and reasoning for the claim. This is not an isolated concept and he wrote at length as to why the idea might be plausible given the current evidence. He also could have written solely about the dream without context, but that lacks a grounding in the reality he was attempting to describe.
If it is possible to write a paper where the result is possible to verify using already-known methods, then by all means write in that style. But this is a subset of the useful papers to be written, and in my experience a small one.
[1] https://gallica.bnf.fr/ark:/12148/bpt6k281952v/f102.item
Certainly. I never claimed otherwise.
But bloaf's and lonesword seem to think such papers are of only superficial merit at best, and that detailed steps to reproduce the research are essential.
I disagree with that viewpoint.