
“Extremely angry with the state of academic CS research right now” - zdw
http://neverworkintheory.org/2016/04/26/perception-productivity.html
======
tedmiston
For an engineering community manager, that was an awfully rant-y way to make
the point: "It's frustrating that researchers don't publish their code."

Not to say I disagree with the frustration... but it's also not something new.
It's been this way for decades. I'd much rather hear about who _is doing_ work
in this space and what they're working on. Here are the ones I know of:

1\. _The Center for Open Science_ ([https://cos.io](https://cos.io)) is one
such org trying to fix this with the Open Science Framework [1].

2\. _GitHub_ also recognizes the need for citable code and gives special
discounts for research groups, in fact, Mozilla is one they work with [2].

Two smaller related startups are:

3\. _Datazar_ ([https://datazar.com](https://datazar.com)) - A way to freely
distribute scientific data.

4\. _Liquid_ ([https://getliquid.io](https://getliquid.io)) - A scientific
data management platform. Somewhat like "Excel for scientific data as a
Service".

Also, a related HN thread from some years ago: "We need a GitHub of Science"
[https://news.ycombinator.com/item?id=2425823](https://news.ycombinator.com/item?id=2425823)

\---

1: [https://osf.io](https://osf.io)

2: [https://github.com/blog/1840-improving-github-for-
science](https://github.com/blog/1840-improving-github-for-science)

~~~
nextos
I like his rant. But at the same time, as an ex-CS grad student now doing
biology, I wish my field was 1/10th as rigorous, tidy and beautiful as
computer science.

Code is a joke, most data is processed using "pipelines", which in reality
means some irreproducible mess. People don't generally do research trying to
understand how cells or tissues work, they generally write papers about
"stories" they found. Only a small minority are trying to do some serious
modeling using serious math.

~~~
jldugger
> Code is a joke, most data is processed using "pipelines", which in reality
> means some irreproducible mess.

You're not wrong, and it's not limited to bioinformatics; Reinhart-Rogoff's
findings were reversed when an additional 5 rows were included in a
spreadsheet they used to calculate their correlation between GDP growth and
debt ratios. And of course, they insist that despite the actual outcome being
_twice as strong and in the opposite direction_ , they still support their
original position.

I wonder if one can get a CS PhD by producing enough retractions. Of course,
it won't win you many friends in the academy, and would probably lead to less
source code made available. But given the Perl code I've seen published who's
termination condition is a divide-by-zero exception, one can argue that peer
review in the information age has to include code review.

~~~
smartbit
Didn't know about the Reinhart-Rogoff controversy [0], interesting! They state
that they have been _careful not to claim that high debt causes slow growth,
but rather that it has an “association” with slow growth_.

[0] [http://www.huffingtonpost.com/2013/04/16/reinhart-rogoff-
aus...](http://www.huffingtonpost.com/2013/04/16/reinhart-rogoff-austerity-
research-errors_n_3094015.html)

~~~
linhchi
I read abt this quite a bit, the refutation is attacking a small portion of
the data, because that small portion is trendy and hot in politics.

Judging academically, the original paper and the refuting paper is a healthy
debate, but the dynamic of the society and politics ab-use them to attack a
whole school of thought at large (the austrian school: less bailout, less
intervention by government, less control over everything) in favor of
keynesian school (more bailout, more government spending, more public debt,
especially in recession and crisis).

Anyway, it remains a controversy, because theoretically one can do what one
wants, but once it involves policy and real life matters, it is hard to argue
for what method is right and what is wrong, in the presence of so many (ready
to b angry) interest groups.

~~~
Certhas
Excuse me? That single attacked paper was the intellectual blanket for an
unprecedented victory march of the Austrian school after the financial crisis
and the recession.

I agree that from a purely academic point of view this is nothing big to worry
about, but this paper played a completely outsized role. And the authors stood
by and let things run their course, without any attempt to reign in or
moderate the debate.

[https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt](https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt)

~~~
linhchi
Fair enough, but i add that academic life is sad, one has to pursue one's
endeavor at one's own cost. However, politicians and the public want too much
from us researchers. So sometimes, we do believe that our sweating formulas
have life impact, or to fancy, save the world.

------
xixi77
How is this different from any other academic research? What he is asking
about is neither openness nor reproducibility (which are, indeed, very
important). He is asking that researchers produce code that he can put into
production. Not only they have negative incentives to do so (for one,
providing such code will surely result in a stream of all kinds of support
requests), it would actually work against the reproducibility objective.

The purpose of the code written is usually very simple: to produce results of
the paper, not to provide a tool other people can use out of the box. Even
when such a tool is nominally provided (for example, when a statistics paper
is accompanied by an R package), there are good reasons to be very careful
with it: for example, the paper may include assumptions on valid range of
inputs, and using the package without actually reading the paper first would
lead to absurd results -- which is something that has happened. The way to use
academic research results is to (1) read and understand the paper, (2)
reproduce the code -- ideally, from scratch, so that his results are
(hopefully) unaffected by authors' bugs, (3) verify on a test problem, and (4)
apply to his data. Using an out of the box routine skips steps 1-3, which are
the whole point of reproducibility.

~~~
ef4
> reproduce the code -- ideally, from scratch, so that his results are
> (hopefully) unaffected by authors' bugs

This rests on a common false assumption that programmers make: they think it's
easier to write bug-free code when starting from scratch. The reality is that
it's almost always easier to start with something that's nearly working and
find and fix the bugs.

What really happens when you do a clean room reproduction is that you end up
with two buggy programs that have non-overlapping sets of bugs, and you spend
most of the effort trying to figure out why they don't match up. It's a dumb
way to write software when it can be otherwise avoided.

~~~
thaw13579
I wonder though, maybe non-overlapping sets of bugs are actually better for
science? That is, it could avoid systematic errors. Of course, one bug free
implementation is clearly better!

------
thaw13579
Two points to consider here. First, it's not fair to criticize the whole field
of academic CS research. Not everyone's work can be accurately represented by
a github repo. Second, even when it can, expecting the "Run author's
[whatever] against an up-to-date dataset." step to work is asking a quite a
lot. Typically there are a infrastructural assumptions baked in (file paths,
legacy code dependencies, etc) and manual steps to get to plotted results. In
an ideal world with enough resources, every lab would have technical staff to
help with this process, but most researchers unfortunately don't have
bandwidth to spend on this problem.

~~~
morgante
If your paper has results, it has code of some sort which can be put into a
github repo.

That's the bare minimum. If you don't know how to make code agnostic to file
paths or dependencies, that's too bad, but fortunately a field practitioner
picking your code up _will_ know how to work around those issues. At least
they're not starting from scratch on trying to rewrite your code.

~~~
eastWestMath
As a theorist, not it goddamn doesn't.

~~~
jononor
How do you validate your theories?

~~~
eastWestMath
I prove them.

------
visarga
On this topic I'd like to mention the machine learning papers site GitXiv,
which is wonderful. It publishes papers alongside the Github repo containing
the code.

[http://gitxiv.com/](http://gitxiv.com/)

~~~
jcrites
Thanks! That's awesome. I have long thought that a platform for scientific
publication where code and data are included would be valuable, and I'm glad
to learn it exists. Going further, imagine the code is like a container that
anyone can run to reproduce the findings, including statistical analysis and
summary on raw data sets -- the key findings should "build". Perhaps the
platform also provides continuous build during "development" (research) so
that researchers can work privately and then publish their "repo" publicly
along with their paper. An easy way to clone and reproduce the build after
publication: "fork my research"

As an extreme version of the idea, imagine if the actual paper itself (TeX)
and all the data within it are also built as part of the repository; any
graphs in the paper are rendered from data in the repo, any numbers are data
accesses, etc. This probably wouldn't be helpful to researchers, but it would
promote scientific reproducibility and aid everyone building on a researcher's
work. Tremendous work goes into authoring the papers themselves, sometimes
with methods or tricks that are private; laying it all out publicly would
greatly help students of science.

Going even further: to avoid cherry picking of positive results, review boards
expect experimental criteria to be published (at least privately to them) in
advance, for research that involves capital-E experiments. Perhaps this
includes analysis code at least in prototype form; like test driven
development, the acceptance criteria are written first. When the paper is
ready for review, the reviewers can compare the initial prototype analysis
logic to the final form. Perhaps the board also expects all data and trials
collected during experiments to be made available in the repository, whether
positive or not. All collected data should be in the platform, in the most raw
form it was originally recorded, as well as all steps of summary and analysis.

I wonder if a process and platform like this could contribute to the integrity
and quality and reproducibility of scientific research. People funding
research ought to ask for it, especially public funded research, and the whole
repo is made open eventually if not initially.

Perhaps as part of the platform's value prop to researchers (on whom it is
imposing probably more burdens than benefit, for sake of _public_ benefit),
the hosting is free and funded by a foundation, or steeply discounted. (OK, it
won't pay for LHC scale data sets, but otherwise ...) So using it to host your
data, code, and paper is free, at least up to a point. I would be interested
to contribute time and resources toward building or supporting a platform like
this.

~~~
gcr
I don't think research should be as structured as software development. In CS,
many of the most interesting papers come about when the authors discover
something unexpected or non-intuitive and choose to explore down that thread.
That's why it's research -- sometimes you can't know what you will find until
you're there.

~~~
tremon
_I don 't think research should be as structured as software development_

In many ways, it already is: good research requires meticulous log keeping in
order to reproduce results, and equal effort must be spent on maintaining
references to other literature, or you risk missing a citation in a published
paper.

------
rudi-c
I took a graduate course on software engineering just a month ago and read
many of those paper which use Mozilla's data. It's a very popular dataset in
the field since it is both open and large. I'm sure Mike Hoye meant to
criticize a part of that field, not academic CS research as a whole.

My impression of the field was there was a severe mismatch of skillset. The
set of people with the scientific background to carry proper experiments, and
the funding to do so, is very disjoint from the set of people who understand
the field. That made a lot of the papers feel "off". Almost like reading text
generated by a machine: individual sentences make perfect sense, the whole
doesn't seem to go in a relevant direction.

As someone who's done a fair bit of practical software engineering, seeing
academics study software engineers feels like seeing a WW2 veteran trying to
understand how youngsters use snapchat. It feels very awkward for the
youngster, just as it does for the software engineer. Which I imagine is one
reason why Mike is pissed off.

There is some irony that businesses are much more scientific in this
particular subfield than academia, because business incentives require the
results to be reproducible and meaningful, over a longer period of time.

------
modeless
The state of machine learning research these days seems pretty good.
Essentially all research is published on ArXiv and there is a lot of code
released too (though there could certainly be more).

I think openness has been a big contributor to the recent explosion in
popularity and success of machine learning. When talking to academics about
this, machine learning would be a great field to hold up as an example.

~~~
pmalynin
I'd say the opposite as a member of a group at my university who review ML
papers. First off right now there seems to be a drive to explain many
phenomena in ML in particular why neural networks are good at what they do. A
large body of them reaches a point of basically "they are good at modeling
functions that they are good at modeling". The other type of papers that you
see, is researchers drinking the group theory kool-aid and trying to explain
everything through that. At one point we got 4 papers from 4 different groups
that tried to do exactly that. All of them are flawed, either in their
mathematics or assumptions (that will most likely never be true, like
assumptions of linearity and your data sey being on a manifold). Actually
speaking of math, many papers try to use very high level mathematics
(functional analysis with homotopy theory) to essentially hide their errors as
nobody bothers to verify it.

~~~
eli_gottlieb
>First off right now there seems to be a drive to explain many phenomena in ML
in particular why neural networks are good at what they do. A large body of
them reaches a point of basically "they are good at modeling functions that
they are good at modeling".

Since this is closely related to my current research, yes, ML research is kind
of crappy at this right now, and can scarcely even be considered to be trying
to actually explain why certain methods work. Every ML paper or thesis I read
nowadays just seems to discard any notion of doing good theory in favor of
beefing up their empirical evaluation section and throwing deep convnets at
everything.

I'd drone on more, but that would be telling you what's in my research, and
it's not done yet!

------
KKKKkkkk1
For a computer scientist, reproducibility means more work that they aren't
paid to do. If I ask the Mozilla team to implement new feature X, the response
will be either (a) point me to a donation link, or (b) We're open source, so
why don't you implement the feature yourself? The computer scientist's
response is the same.

~~~
jcrites
Reproducibility for the computer scientist means including any code written
and data collected or relied on in the scientific publication itself. In
practice, getting there from here isn't literally zero work, since some actual
human action is needed to bundle the code and data, but that effort ought to
be negligible overall, especially if we make it a standard part of the
scientific process.

~~~
semi-extrinsic
Trust me, it's currently far from zero work to submit code with a research
paper. I was recently the corresponding author on a software paper sent to a
journal that at least verifies the code compiles and runs and produces
expected output. Since the poor person testing the softwares submitted is
permanently in the ninth circle of dependency hell, across all platforms and
libraries imaginable, it took about fifteen emails back and forth plus an OS
reinstall before everything checked out. And they said that wasn't anything
extraordinary.

~~~
jcrites
How about a platform centered around Linux containers (or maybe one of several
OS containers or VM images), as the repository image?

I'm not saying the work is zero now, but maybe we can get there. If a
researcher is developing on a platform where their repository is expressed as
a container-like image, then they should be able to publish it for anyone to
run exactly as-is. The container repo includes the data, the operating system,
and any languages and libraries, with an init system that optionally builds
the results.

~~~
semi-extrinsic
Yes, I think we need to go in this direction. The problem is that the
container system is yet another tool for researchers to learn. The first step
is to get everyone using VCS and nightly testing. Many are still at the point
of clumsily written, old Fortran code that gets emailed around and exists in N
different variants. (Not that there is anything wrong with Fortran.) Many are
at the point where if you email them a link to a git repo to clone, they're
clueless about what to do.

~~~
Al-Khwarizmi
It would help if Git didn't have such an awful learning curve (and I say this
as a git user that already went through it).

I know researchers that used Subversion when it was on the rise, but they just
abandoned version control altogether when Git became the generally preferred
option.

------
alantrrs
We're working on this exact problem at Empirical
[http://empiricalci.com](http://empiricalci.com)

We're building a way for CS researchers to run, share and compare experiments
including their whole research environment.

We just applied to Apply HN and would appreciate your feedback:
[https://news.ycombinator.com/item?id=11586948](https://news.ycombinator.com/item?id=11586948)

------
santaclaus
> They publish LaTeX-templated Word docs as paywalled PDFs.

Somewhat tangential, but do CS academics actually write papers in Word? During
my grad school days I did not encounter a _single_ paper 'typeset' in Word.
Writing was usually done with a LaTeX and and a makefile in a git repo.

~~~
nycticorax
Just adding another data point: When I was in CS grad school (late 90s),
everyone used LaTeX (including the journals). Then I started hanging out with
neuroscience people, and everyone (including the journals) used Word+Endnote,
with basically ad-hoc treatment of the figures.

~~~
santaclaus
How does Word work out when multiple collaborators are writing at once? One of
the cool things about version controlling LaTeX is that conflicts, merges, etc
are dealt with using standard tools, which is super helpful when you have
collaborators across the globe furiously writing and redoing figures a few
days before a deadline.

~~~
joshvm
Well when you deal with merge conflicts in Word, your standard tool is Word. I
don't think you can get much more standard than Office when you consider
collaborators outside academia. It's not great for simultaneous editing
(although I think this is now possible in 365).

It is, however, very good for tracking changes over versions. Many academics
are not familiar with git, diff and so on and it's nice to easily see
historical edits in the document. For simple documents like abstracts, it's
much easier to send a Word document than it is to send a tex file and assume
that everyone on consortium is going to be able to compile it (especially if
you work with industry).

~~~
fphhotchips

        > although I think this is now possible in 365
    

It would want to have improved since I used it a couple years ago in Word
2013. The main problem was with citation managers - Word would give a
paragraph lock to you whenever you edited a paragraph, and it would only
unlock that paragraph after a save (either auto or manual). Of course, when
you have a citation manager, they have the habit of changing all the
paragraphs when you insert a new citation that changes the numbering (ie. [1]
becomes [2], etc.).

------
einarvollset
As a recovering CS academic, I have to say - boo-fucking-hoo. Tell me exactly
where the funds to do this extra level of support is supposed to come from? Is
Mozzila paying for those grad student hours? No? Well then.

~~~
morgante
We as taxpayers are already paying for your research.

Given that, it seems reasonable that you should make the modicum of effort to
publish your code.

Also, don't construct a false dichotomy. Publishing your code (at all) does
_not_ mean you have to make it production-quality or provide support for it.

~~~
kardos
> Publishing your code (at all) does not mean you have to make it production-
> quality or provide support for it.

No. If you publish something that's incomplete or doesn't have all the right
dependencies listed, etc, it's not really of any use. Writing up compiling
instructions plus dependencies plus how to run it plus input files etc takes
time and by the time you've got it to the state that someone else can run it,
now it's "production-quality".

> Given that, it seems reasonable that you should make the modicum of effort
> to publish your code.

There's currently no incentive/requirement to publish code, it uses time and
does not increment your publication counter. Find the incentive and you'll
start seeing published code.

~~~
morgante
> No. If you publish something that's incomplete or doesn't have all the right
> dependencies listed, etc, it's not really of any use.

Again you're creating a false dichotomy which is just fundamentally false.

Even if you only published _half_ your code—to the point where it doesn't even
compile—that's still helpful.

In the status quo, I have to write your entire code from scratch.

If you published what you have, I would merely have to debug the issues and
figure out what the dependencies are.

I don't understand why it's so hard to understand that some code is better
than no code.

~~~
kardos
> I guess researchers really are so incompetent with software that they think
> code which requires debugging is useless.

So published papers are held to a high standard -- filtered through editors
and peer review -- while publishing code can be half-assed at best? I still
disagree; if it's worth doing, it's worth doing right.

------
alexholehouse
Reproducible is a major issue across science in general, but the difference is
there's no reason why one shouldn't be able to easily re-run a defined
analysis on a more recently updated data set to ask if conclusions drawn
previously still hold. I actually published a side-project paper on this (in
biological sciences) last year [1] - what was scary was there was such a lack
of discussion surrounding this idea, despite the fact that large databases of
biological data are CONSTANTLY changing and updating.

[1] Holehouse, A. S. & Naegle, K. M. Reproducible Analysis of Post-
Translational Modifications in Proteomes-Application to Human Mutations. PLoS
One 10, e0144692 (2015).
([http://journals.plos.org/plosone/article?id=10.1371/journal....](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0144692))

~~~
cderwin
The other difference is that as far as I know, computer science is the only
discipline for which industry has solved the problem of reproducibility; it's
one thing to be asked to design a method to run reproducible studies of
humans, it's another to ask researchers to run `git remote add
[https://github.com/user/repo](https://github.com/user/repo) && git push
--set-upstream`. That's not asking for any support, or other effort on the
researcher's part, and I frankly don't understand how the CS academic
community doesn't have this as a standard when it'd be so easy to implement.

------
jpolitz
It is a real issue. The programming languages and software engineering
communities have been working on evaluating software artifacts for a few years
now:

[http://www.artifact-eval.org/](http://www.artifact-eval.org/)

This doesn't address open access, but it does make sure the software is usable
by a non-author to reproduce the results of the paper.

~~~
kaeluka
@ open access: In my specific subfield of academic CS, a huge success story
was ECOOP (one of the biggest conferences in the field) going open access last
year. I'm hoping that there'll be other conferences following suit. My money
is on open access being commonplace in 10-15 years.

~~~
samth
Note that ECOOP went open access partly under the leadership of jpolitz's
advisor. :-)

~~~
kaeluka
You had me confused for a second, I didn't realise I was talking to jpolitz :)
Thanks for bringing that up. AFAICT, he (advisor) has done a lot to make that
happen, indeed.

------
dgacmu
Help me convince people to do a few things a bit better: [http://da-
data.blogspot.com/2016/04/stealing-googles-coding-...](http://da-
data.blogspot.com/2016/04/stealing-googles-coding-practices-for.html)

(tl;dr of my linked blog post: Apply a slightly lighter weight form of some
industry engineering practices in CS research coding. I think it's feasible.
It doesn't solve all of the problems, because as discussed elsewhere in this
thread, some of them are incentive-related and I'm not going to claim to have
answers to everything. :)

(a) Convince more research groups to do their research on GitHub by default --
ideally, in open repositories. They get good hosted SCM, the world gets a
better chance of seeing their code.

(b) Create more incentives, like the USENIX Community Award, for research that
puts out its code. I'd say that in the systems community, a pretty decent
chunk of the papers at SOSP, OSDI, and NSDI have code releases (of varying
degrees of usability) accompanying them, though that's not a scientific count.

Mozilla could throw $1k to help create community-award-style incentives in the
conferences they're interested in. Win-win. You get engaged with the
community, you create some incentive for people to do the right thing, and you
can use it as an onroad to deeper engagement with the winning authors (i.e.,
you can try to bring them in for internships. :).

------
moron4hire
What it will take is creating a new system of research and development that
ignores the traditional academic system. Because this has been a problem for a
while and they clearly are not hearing the message.

The reason academic research works is because it takes risk on potential
failures, because it's only donated or grant money anyways. But academic
institutions fetishize academic papers, which is the problem from the article.

We need to legitimize research outside of the academic institution. Pay people
to do research on their own time, if they perform it in an open-source,
reproducible fashion. Incentivize it based on the reproducibility factor, but
avoid attaching a profit motive.

Look at what Xerox PARC and Bell Labs were able to do before the penny-
pinching bean counters took over.

Otherwise, it just sounds like "we want all this risky work done, and we don't
want to pay for it."

~~~
nycticorax
Maybe I've drunk the kool-aid, but saying "academic institutions fetishize
academic papers" seems wrong to me. It's like saying that developers fetishize
working applications. It's not a fetish, it's the whole point of the thing.
The output of research is research papers. Yes, sharing raw data and code are
both good things, and should be promoted. But no one is going to take the time
to look at either unless the paper presents good evidence for some novel
result.

~~~
moron4hire
Papers don't do anything. Papers aren't executable. Papers are largely only
useful in regards to writing other papers. To make the whole point of the
exercise to be something that is inert and self-referential is the
fetishization part.

~~~
Retra
Papers teach people things. You can't really hand someone some working code
and expect them to iterate on it to produce something novel and insightful,
because it won't be written in a high-enough language to express the core
concepts flexibly.

~~~
moron4hire
I don't see how open source software is at all compatible with the view you've
just expressed.

~~~
Retra
I don't see how that's related. There are loads of open source software
projects that are maintained by only a single person precisely because they
haven't bothered to properly document why it exists, what came before it, what
important things it is demonstrating, or what needs to be done to improve it;
the archetypical content of a research paper.

------
Ericson2314
This is why I believe NixOS is so important. It allows one to completely
freeze the entire development environment on one hand, but also does so in a
declarative, well abstracted manner (vs VM image let's say) so
tweaking/porting is actually feasible.

Until well get to a point where building/installing/administrating is not
hours of bullshit, research (and free software) will suffer.

~~~
aub3bhat
As an academic researcher I find it absolutely hilarious that you think the
complex social problem of incentive structure and competition will be solved
by some Unix OS.

If you are interested just take a look at the complexity of
licensing/ownership of code written by a PhD student at a Research University
in United States.

If you look at most of my Open Source code, I use AWS AMIs to share both data
as well as OS + code, however I can do that only for side projects. The main
thesis projects are typically very high value and consequences of sharing it
far more complex to understand.

[https://github.com/akshayubhat/](https://github.com/akshayubhat/)

~~~
Ericson2314
No that is not what I think at all, see my follow-up comments below. I just
think the combination of shitty tools + incentive structure is even more
insurmountable. This is a tough problem that should be attacked from as many
fronts as possible.

> The main thesis projects are typically very high value and consequences of
> sharing it far more complex to understand.

Commercial value, the university is just more stringent on the
licensing/ownership restrictions, or something else?

~~~
aub3bhat
There are several factors.

1\. Commercial value.

2\. Future grant applications (a competing group not sharing code will have
better chance winning the grant.)

3\. Future of other students and collaborators in the group. If two PhD
students write a paper, the junior student might wish to write extension
papers without getting scooped.

And many more. Yet if a paper is important enough, independant researchers
will often attempt at replication, this nowadays routinely happens in Machine
Learning and Vision due to huge amount of interest. Also in several cases
replication is fundamentally impossible, e.g. consider a machine learning
paper that uses proprietary data from hospital attached to the university,
etc.

~~~
jcrites
I totally get that researchers' incentives are not aligned toward publishing
it, so no need to explain that further. There are costs and downsides and
probably not enough benefit to them. That's fine. Everyone works within their
system of incentives.

If it's paid for by public dollars, then the code and data belong in the
public domain eventually. I understand there are exceptions like hospital data
affected by patient confidentiality - that's fine. However the code released
by that researcher should be capable of reproducing their results with that
data set plugged in (such as by someone else who has access to it).

As a taxpayer, my concern for publicly funded research is maximizing benefit
to the public good. I understand your point about follow-on research, and I'm
not saying that I'd expect the code and data to be made available immediately
with publication, but that deserves to be the case some reasonable time
afterward (like a year). I understand that researchers' incentives are not
necessarily aligned toward making it public; I am saying that people who fund
research (including taxpayers through the political process) should require
and expect it. Keeping it private indefinitely is a degree of self-
centeredness that does not strike an appropriate balance between benefit to
the researcher and to the public in my opinion.

~~~
aub3bhat
I never understood the meme about "public funding" translating into "public
domain". Just because research is "publicly funded", does not means that the
"public" owns it or even has a right to ownership. Public education is
publicly funded does not means that government can ask for every drawing drawn
by 9 year old in classroom to be in the public domain :) . In fact its
actually opposite
([https://en.wikipedia.org/wiki/Bayh%E2%80%93Dole_Act](https://en.wikipedia.org/wiki/Bayh%E2%80%93Dole_Act)),
given that Universities can and do patent inventions from publicly funded
research.

Further funding arrangements themselves are very complex, a professor
typically procures funding from University, NSF, NIH, private companies,
donors etc. In such cases if NSF adopts a hard line approach that any research
touching its dollars ought to release code under say GPL, it would make it
impossible to collaborate. Finally all requirements aside, one can always
release intentionally poorly written code in form of MatLab .m and compiled
mex files. I have observed several such cases, where the code can demonstrate
a concept but is intentionally crippled.

Finally graduate students, graduate and are paid for doing research which is
publishing/presenting papers at peer reviewed conferences and journals. If
what funding agencies really seek is ready made software they ought to fund /
pay at the same level as software developers (as many companies do).

~~~
jcrites
> Just because research is "publicly funded", does not means that the "public"
> owns it or even has a right to ownership.

I didn't make the argument that the public owns it or has a right to
ownership, though I suppose that some people might and so I can see why you
would touch on that point.

I would describe my view as like this: public funding is subject to the
political process, and voting by taxpayers (directly or indirectly through
voting of politicians or their appointees). As a taxpayer, I prefer to make
public domain publication a requirement of publicly funded research, and I
think every taxpayer should too. I consider the goal of public funding of
science to be the benefit of public good, and believe that public good will
best be served by public domain publication of all data, code, and analysis
methods. (Whew, there's a lot of "pub" and "public" in there!)

One might reach my position by working backwards from, "Why do we as taxpayers
agree to fund science with government money?" It's certainly not to give
researchers prestige or jobs! (Those may be necessary parts of achieving
public good, but they're not the goal which is the public good, and if they're
in tension with public good then the public good probably needs to win.)

I don't seek ready made software; not at all. I only seek adequate disclosure
of data and analysis methods sufficient for others to easily verify it and
build on it. See for example the attempt at replication in
[http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04dea...](http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/)

> In such cases if NSF adopts a hard line approach that any research touching
> its dollars ought to release code under say GPL, it would make it impossible
> to collaborate.

I will need to think more about this issue. I might be willing to accept the
downside as a taxpayer. I'm not sure I understand it well enough what the
friction would be to collaboration at the moment. If you're referring to the
GPL specifically, then yes I agree that's probably the wrong license - public
domain would be more appropriate.

I would be OK if this was simply an electronic log of the data as well as all
machine commands that have been run on it - something that is recorded
automatically by the operating environment. I am truly not looking for
"working production code". But, those sequence of commands should be
reproducible if someone "replays" them; a verifiable digital journal.
Publishing an article that's difficult-to-reproduce feels like producing the
least possible public good while still getting credit. Publishing an article
that's fully and automatically reproducible, because it contains references to
all of the data and code that yield the results as an executable virtual
machine with source code, provides the maximum public good, and that's what I
want science funded with public money (and ultimately all science) to work
toward. (I realize that this is just like, my opinion man :)

~~~
aub3bhat
You are correct in expecting return of public investment. Actually NIH has a
policy that explicitly favors "Basic Scientific Research" over applied or
application of research. According to NSF and NIH, the primary goal of
government funded research is advancement of science, this is done via
conducting experiments, and publishing results at peer reviewed venues. The
peer review both during grant application and at publication stage factors
heavily into assessment by funding agencies. If tomorrow NSF were to give
significant weight to availability of source code (They actually do that to a
small extent), it might set up perverse incentives. A small percentage of
federally funded research goes into Computer Science and even a smaller
fraction involves results where there is enough demand for software. Another
aspect of academic funding that people don't get is that research grants
unlike say contracts have a significantly different set of expectations
associated with them. E.g. a student can get NSF Fellowship, claiming he want
to cure diseases using machine learning, only to later spend 3 years working
on music recommendation system (True Story!).

Regarding the economics study you linked to, I am very much familiar with that
study having seen the interview of graduate student on Colbert Report. For
non-CS fields the quality of code is anyway so bad that its much more
difficult. Further several researchers rely on proprietary tools, which only
make this task difficult.

In my opinion the correct way is not by having NSF impose rules, but rather by
having venues that accept papers (Conference & Journals) insist on providing
software. However this is easier said than done, since its a competitive two
sided market.

Regarding actual licensing issues, I can assure you that GPL is second
favorite license of choice favored by University IP departments, the first one
being "All rights reserved with modification explicitly forbidden, except for
reproduction of experiments."

------
chrisbennet
Funny, as students we were expected to solve the exam problem and "show your
work"...

I think the root a the problem is that the goals of the researchers are not
aligned with the goals of Science. This isn't a criticism of the researchers
but instead of the "game" they are forced to play.

For example,the goal of Science is to move the ball(knowledge) down the field
for the benefit of mankind. We don't reward researchers for doing that, at
least not very well. We reward researchers for writing _papers_ full stop -
not for making their research easy to reproduce or build on.

------
atishay811
Its not just the researchers' fault. Maybe the industry should help. Mozilla
is a major stakeholder of the web platform that makes distribution easy. Lets
make sure web is the best platform to do all the research.

* Provide great scientific and matrix manipulation libraries within the browser. WebAssembly isn't going to solve this. Why would the academia rewrite everything?

* Provide tools that help research being open. Uploading your code to Github isn't a solution. The real solution is making it easy to use and link other person't research. Can we make research as accessible as a javascript file that you include in your html file to run. And it shouldn't cost the creator to host/maintain it. Offline web sucks(still) and it costs money to host your servers.

* Provide incentive to use the web for everything. A great one would be an easy to use and debug toolset and an easy set of methods to get data in and out, an editing environment that can be setup with one click. The closest is iPython Notebook. And it takes work to get there.

Sharing should be default and easy. If it isn't we are no one to complain.

~~~
wyattpeak
>Uploading your code to Github isn't a solution

Why not? This sounds like exactly the solution. It's no burden on the
researcher - they don't have to alter their research methods to fit a new
system, they just dump the code, and leave it up to other users to
reimplement.

Building web tools to allow research sounds like precisely the wrong solution,
at least in the short to medium term. Research funding doesn't go far enough
as it is, you're not going to get researchers changing their processes
entirely for no gain. What if they need custom hardware, or access to tools
and libraries that haven't been implemented?

Sharing _is_ default and easy. That's why Github has exploded.

~~~
robotresearcher
You forgot to consider IP issues. Schools vary a lot on their policies by but
for many the code belongs to the school and can not be open sourced without
permission which requires extra work. Funding sources have their own IP deals
to consider too.

It all sounds so easy until you look at the actual constraints. Professors are
usually smart and experienced and they have thought about this stuff a lot. If
it was as easy as you thought, it wouldn't be a problem.

I publish all code as a matter of lab policy. I chose where to set up my lab
partly so that I was able to do this. Not everyone has this luxury or makes
this a priority.

~~~
wyattpeak
You're right, I did, but I wasn't actually addressing ease in some absolute
sense, I was talking about the appropriate tools for the job.

If researchers have the legal right to publish their work, I can't see any
reason why github wouldn't be exactly the place they'd share it, rather than
some custom online research system as proposed by the parent.

That said, I don't have any experience in CS research, it's not my field, so I
may be wrong about that, do tell if so.

~~~
robotresearcher
We share our code on github and publish the commit hash for the code that
generated the results in the paper. That way we can continue to develop the
code after publication but readers can retrieve the exact code described in
the paper if they wish. Simple and effective.

But again, the IP rules at my university allow this at my sole discretion,
which is unusual.

------
emeryberger
Jean Yang @ CMU (together with Sam Tobin-Hochstadt @ Indiana) posted a
response to this on her website:

[http://jxyzabc.blogspot.com/2016/05/myth-cs-researchers-
dont...](http://jxyzabc.blogspot.com/2016/05/myth-cs-researchers-dont-publish-
code.html)

Releasing code is a widespread practice in the programming languages and
software engineering communities, and one that is getting stronger (see
[http://artifact-eval.org](http://artifact-eval.org)).

If you are a CS researcher, please fill out this survey of open source and
data in computer science:

[https://docs.google.com/forms/d/1U6I3B2XAqGJDYq101e14PS78jhG...](https://docs.google.com/forms/d/1U6I3B2XAqGJDYq101e14PS78jhGl4pTHy6eDx2zx0j4/formResponse)

------
msiemens
I guess I can see why many researchers are not inclined to publish their code.
I've worked as a research assistant in the institute of microelectronics of my
university for a year and the code quality was somewhere between mediocre and
downright terrible. And that's not to mention the absence of sane software
engineering standards like bug tracking or code reviews.

------
intrasight
From reading this discussion, it seems that IP rules are inconsistent, and
therefore part of the solution needs to be to perhaps have funding agencies
enforce a standard license. With that barrier eliminated, scientists wouldn't
expend valuable time negotiating with their school's layers.

------
gravypod
Personally, I'd feel code with light commenting would be better suited for
getting a "practicable" point across to readers then a paper will.

You won't need to put so much weight on describing some of your practices, you
can just how how your solution is better.

------
raverbashing
> But that never happens. Because CS researchers don't publish code or data.
> They publish LaTeX-templated Word docs as paywalled PDFs.

Correct

Publishing in pseudocode made sense some 30y. ago

Today if you're publishing in pseudocode I personally think it's someone who
is doing that because can't write a Hello World in C

Oh and also let's stop the "research comes only from universities" idea

------
mehrdada
Fund the research you want to see in the world. Incentivize it as you think is
right. Don't whine.

------
tachim
Codalab is designed to solve _exactly_ this problem:
[https://worksheets.codalab.org/worksheets/0x818930127c4d47de...](https://worksheets.codalab.org/worksheets/0x818930127c4d47de84c1ceaadf04d014/).

------
spot
making research reproducible with publications that actually contain their
code (in addition to prose and graphs, all connected together) is one of the
goals of Beaker Notebook, which has a publication server where you can share
such results, and then with one click open them up in the cloud, modify them
as you see fit, and then run them again (if you have seen Beaker before, this
last part, the cloud hosted version, is new as of last week).

[http://beakernotebook.com](http://beakernotebook.com) and
[https://lab.beakernotebook.com/#/publications/featured](https://lab.beakernotebook.com/#/publications/featured)

~~~
diimdeep
Wow, looks impressive and that it comes from hedge fund is cool too. It's like
Jupyter but much more robust, multiple languages in single notebook and
ability to share data across languages out of box is really nice.

------
mikeskim
I would not be surprised if most academic CS research is not reproducible.
This is true for many other fields outside of CS, and I've seen it first hand
in machine learning. It's a problem but it's also just how things are.

------
askyourmother
There are other issues right now, like impartiality and bought research, like
Microsoft at Cambridge University.

I recall the professors at our uni treating the undergrads with contempt as
wasting their precious lucrative research time.

~~~
drjesusphd
Precious? Yes. Lucrative? Hardly.

------
tango12
Maybe it can be thought of as a tooling problem? Say, a plugin that allows a
one-click publish code + data from Matlab, and then it all goes up on a well-
indexed page so that others can download/run it.

~~~
simonster
I doubt the problem is that academic CS researchers don't know how to publish
their code, but rather that the disincentives are usually stronger than the
incentives.

~~~
cbhl
Is there even code to publish? I am under the impression a lot of papers from
Bugzilla data are of the form "we imported the data into Excel and had a hand-
crafted one-off spreadsheet".

~~~
jcrites
In that case, yes: the spreadsheet itself consists of data and analysis over
data (aggregations over columns and rows, etc.) so the spreadsheet itself
would ideally be version controlled and published.

The idea isn't to ask researchers to formalize what they make more than
before, but to include fully reproducible details in the publication. A
spreadsheet is totally fine because you can see how it works, reproduce the
result, and tweak the inputs/methods to build on it.

------
skybrian
It seems like politely writing to the researchers and asking if they still
have the code lying around might have good results. (If nothing else, it lets
them know someone cares.)

~~~
clintonb
A colleague was working on a replication study. We got the code from the
original researcher and another researcher who did a follow-on study. The code
barely runs, and the results seem off. I spent days debugging to no avail.
Just because researchers provide code does not mean it is well-written code,
or that it necessarily works.

~~~
jcrites
Then I'd be skeptical about those results. Releasing the code allows others to
judge the likely accuracy and integrity of the results. A lot of things can go
wrong in complex, multi-step computational processes. If care and rigor has
not been put into them, then I'd have no reason to believe in the integrity of
the output. I want the general public right to judge that, as well as build on
it when it's useful and valuable.

Every publication involving data and technical analysis should publish them to
a degree that makes validation possible in at least as detailed a way as
portrayed in "Does High Public Debt Consistently Stifle Economic Growth? A
Critique of Reinhart and Rogo ff"
([http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04dea...](http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/))

------
jilljennV
What about this promising recent link?
[https://gym.openai.com](https://gym.openai.com)

------
jokoon
I really wonder, what's the difference between computer science and
mathematics?

It seems like CS is a subset of math.

~~~
yarper
In a word - specificity. The same question arises in a number of older fields
of study, for example "is chemistry the same as physics?"

~~~
mcguire
Here's a guess: the short answer is that the physicist would say 'yes', but
would be unable to solve problems that the chemist would consider trivial.

------
snissn
I would extrapolate that to all of academia. If you want to work on something
that's useful, insightful and will make the world a better place, the ivory
tower of academia is great if you want to live in denial in an echo chamber.

~~~
rudi-c
That's a strong claim. While I'm sure the sentiment is understandable for some
fields of academia, others have produced significant results that have been
adopted by the industry, sometimes very quickly.

Machine Learning has a lot of theoretical results produced by academia, but
the more practical techniques (decision trees, SVMs, neural networks, etc)
also all came from academia. The engineers scaled the algorithms to run on
bigger datasets, but the initial work is still driving those systems.

Graphics research has seen comparable contributions between industrial
research labs and academia. Sure, a higher proportion of papers from academia
don't end up practical, but the number of papers that are very practical makes
that quite irrelevant. It's to be expected since you can't predict what works
in advance, and industrial research labs just don't bother publishing negative
results, not that they don't get any.

Many programming languages and compiler techniques came from papers.

I could go on but you see the point - it depends on the field.

~~~
snissn
But that doesn't say anything about the person in academia who writes a
speculative generic research paper about a way too simple implementation of a
decision tree. I don't want to demean that accomplishment, but for me it would
be hard to have sustained excitement from that. Also I think that a counter
argument is that just because the initial work of X, Y and Z algorithm were
sketched out in academia, they still very much so would have been scaled in
industry once the actual need has been found for them.

------
sklogic
What percentage of the CS research based on any data at all? It is mostly
purely theoretical. Still, it would be nice if they published their proofs as
a Coq code.

~~~
moyix
I think you're conflating a very narrow slice of CS (theoretical computer
science) with the larger field. There's a huge amount of CS research that
relies on gathering and analyzing data, building systems, etc. Theoretical
computer science is actually a very small slice of the research pie.

~~~
jroesch
At least in programming languages, systems, and formal verification project
code is both available and often evaluated along side the publication. For
example FSCQ([https://github.com/mit-pdos/fscq-impl](https://github.com/mit-
pdos/fscq-impl)) from MIT, Verdi
([https://github.com/uwplse/verdi](https://github.com/uwplse/verdi)) from UW,
Dune([http://dune.scs.stanford.edu/](http://dune.scs.stanford.edu/)) from
Stanford, and many more.

These are just the areas I work in, its very hard to making sweeping statement
about CS as a full field since it is very diverse sub-community to sub-
community.

