Never Work in Theory

I wanted to point you at https://neverworkintheory.org/ which attempted to bridge the gap between academia and software engineering. Turns out the site shut down, because (quoting their retrospective) > Twelve years after It Will Never Work in Theory launched, the real challenge in software engineering research is not what to do about ChatGPT or whatever else Silicon Valley is gushing about at the moment. Rather, it is how to get researchers to focus on problems that practitioners care about and practitioners to pay attention to what researchers discover. This was true when we started, it was true 10 years ago, and it remains true today. The entire retrospective [1] is well worth a read, and unfortunately reinforcing your exact point about software development being fashion/emotion driven. [1] https://www.computer.org/csdl/magazine/so/2024/03/10424425/1Ulj1Qa8tJ6

Whoa there. You are being way too rude and aggressive about this. Please help keep HN civil; that's much more important than any argument. Peopleware is a classic, or at least many software professionals regard it that way, but it is a classic of folklore rather than science. Here is an example that I asked one of the authors about years ago: https://news.ycombinator.com/edit?id=1995716. The research literature itself on these matters is extremely flimsy. There is not even any empirical evidence for the stuff we all agree on, such as that version control is good (http://www.neverworkintheory.org/?p=451) let alone on the controversial points such as productivity. Incidentally, how do you measure productivity? I'd be happy to be wrong about this, though. Perhaps you could cite a study or two that really does hold up to scrutiny?

http://www.neverworkintheory.org/?p=58 After controlling for lines of code, most complexity metrics become worthless.

> Another example is TDD. People espouse the benefits, then some study comes along (http://www.neverworkintheory.org/?p=139) saying the benefits are largely illusory and that code reviews are more effective. The study is much more thoroughly debunked in the comments than I will attempt to reproduce here. Important points: 1) The study's hypothesis was that TDD would produce software with equal, or fewer defects than inspection. That would be one, and depending who you talk to, by no means the most significant benefit of TDD. 2) The actual data collected by the researches showed no statistically significant difference in the two approaches. Reading studies about the effectiveness of different tools and methods is a great idea. Treating them uncritically is not. :)

(Note: I don't mean for this comment to be perceived as bragging or showmanship. It's just that... well, you asked about my background.) Computers have fascinated me since before I can remember. I spend most of my waking hours in front of them. I've been honing my craft for over 20 years. I've written and maintained projects in C, C++, C#, Java, JavaScript (both browser and server-side)[1], Perl, PHP (we all make mistakes), Python, Ruby, and a couple Lisps. In my travels, I've discovered and reported bugs in popular software such as Firefox, Chrome, Node.js, Apple's XNU kernel, and libxml2. I completely agree that some programmers are like machine guns, firing off vast quantities of poorly-aimed code. I try my best to avoid that. I hate sloppy code. I hate repetitive code. Most of all, I hate re-inventing the wheel. If a decent library exists, I'll use it. I have no qualms with something Not Invented Here. I pair sometimes. I do code reviews often. And I use as many profiling, testing, and static analysis tools as I can get my hands on. It sounds like your ordeal made you a better coder. Those sorts of experiences are indispensable, but I've found it takes more to keep improving. It's very useful to become an expert on programming, not just an expert at programming. There's a growing body of literature to aid anyone interested. McConnell's Code Complete is still great. Michael Feathers has a book called Working Effectively with Legacy Code. It contains some great…

While this list is certainly educational, remember that most of these "laws" are heuristics based on personal experience. The discipline of software development is about as rigorous as parapsychology. To learn what we really know about software, I recommend Making Software: What Really Works & Why We Believe It[1], The Leprechauns of Software Engineering[2], and browsing It Will Never Work in Theory[3]. And no matter what languages, testing frameworks, or methodologies you use, always remember: Every "bug" or defect in software is the result of a mismatch between a person's assumptions, beliefs or mental model of something (a.k.a. "the map"), and the reality of the corresponding situation (a.k.a. "the territory").[4] If you want your map to reflect the territory accurately, you'll need more than a few "laws" handed down as gospel. 1. https://www.amazon.com/Making-Software-Really-Works-Believe-ebook/dp/B… 2. https://leanpub.com/leprechauns 3. http://neverworkintheory.org/ 4. http://lesswrong.com/lw/2rb/why_learning_programming_is_a_great_idea_e…

I just read the post that inspired this post: http://programmers.blogoverflow.com/2012/08/20-controversial-programmi… While a lot of these opinions sound reasonable, I want to know if they're actually true. All of these statements are expert opinions, but it's unclear how many of them are backed by research. Without controlled studies, experts can easily believe incorrect things. If you hold an opinion, look at the evidence backing it up. If it's not strong, reduce your confidence. Or even better: Gather evidence, then form your opinion. And remember: Anecdotes don't count. I wish more of my colleagues would do this, but it seems most of them haven't heard of very many software-related studies. If you want to learn more, I recommend http://www.neverworkintheory.org/ as a starting point. After reading some papers, you'll be surprised how limited our evidence-based knowledge is. Looking at software engineering studies made me realize that I'm not allowed to poke fun at psychology anymore. Even that field is more evidence-based than ours.

https://neverworkintheory.org/ is the web site associated with this (from the 1st reference in the paper)

There are quantitative studies showing that TDD has little to no impact development time or code quality[0]. What has been found, however, is that writing code in short increments helps a lot (something that can be caused by using TDD). For more information on studies covering the topic (and much more), I highly recommend watching Greg Wilson's Software Engineering's Greatest Hits[1]. [0]: https://neverworkintheory.org/2016/10/05/test-driven-development.html [1]: https://youtu.be/HrVtA-ue-x0?t=448

Here is one of the first empirical Scala vs Java studies: http://www.neverworkintheory.org/?p=375 Here is the actual paper: http://www.rz.uni-karlsruhe.de/~kb95/papers/pankratius-Scala-ICSE12-pr… Quote from the conclusion: With respect to effort, this study refutes the claim that Scala programs are faster to develop: in comparison to Java, Scala requires more effort, and especially more testing and debugging effort.

Greg Wilson indeed is tremendously helpful in facilitating "the industry" to think about our craft: https://github.com/gvwilson edit: wow, in his project "It will never work in theory" he's fairly sober about the ability of "the industry" to reflect on "the craft" https://neverworkintheory.org/ > about the project: > People have been building complex software for over sixty years, but until recently, only a handful of researchers had studied how it was actually done. Many people had opinions—often very strong ones—but most of these were based on personal anecdotes or the kind of "it's obvious" reasoning that led Aristotle to conclude that heavy objects fall faster than light ones. in the 2024 retrospective: > Conclusion > The comedian W.C. Fields once said, “If at first you don’t succeed, try, try again. Then quit. There’s no point in being a damn fool about it.” Thirteen years after our first post, it is clear that our attempts to bridge the gulf between research and practice haven’t worked. We look forward to hearing what actionable plans others have that will find real support from both communities.

Here are a bunch of blogs from my RSS reader that I think are worth reading. Some of them haven't updated in a while, but the archives may be worth a look anyway. * Blogs with a math focus: http://blog.sigfpe.com http://bit-player.org http://mathr.co.uk/blog/ https://www.johndcook.com/blog/ * Blogs with a programming languages focus: https://existentialtype.wordpress.com https://bluishcoder.co.nz http://blog.ezyang.com http://akkartik.name http://lambda-the-ultimate.org http://mrale.ph * Blogs with a UI design focus: http://www.brandonwalkin.com/blog/ http://ignorethecode.net/blog/ http://interuserface.net * Blogs with a graphics / 3D rendering focus: http://c0de517e.blogspot.com http://diaryofagraphicsprogrammer.blogspot.com https://directtovideo.wordpress.com http://graphicrants.blogspot.com http://www.reedbeta.com http://www.realtimerendering.com/blog/ https://blog.demofox.org http://eelpi.gotdns.org/blog.wiki.html * Other interesting / more general programming blogs (many of these have a "systems" focus): http://danluu.com http://ewontfix.com http://chneukirchen.org/blog/ http://neverworkintheory.org http://apenwarr.ca/log/ https://www.snellman.net/blog/ https://www.pvk.ca http://psy-lob-saw.blogspot.com https://randomascii.wordpress.com http://fastcompression.blogspot.com https://fgiesen.wordpress.com http://blog.plover.com http://chneukirchen.org/trivium/ http://rachelbythebay.com/w/

I read your essay on programming languages. I found it was awesome btw. I saw you made the comment `o be concrete - object oriented programming is easy in python, because it's possible to program methods of an object in a way that doesn't assume a particular representation.` I thought you would be really interested in usage of polymorphism in python (http://neverworkintheory.org/2016/06/13/polymorphism-in-python.html). Quote from the article: `Our findings show that the receiver in 97.4% of all call-sites in the average program can be described by a single static type`. It looks like even when that restriction is removed it doesn't really help that much. Another great podcast on evidence based programming: https://www.functionalgeekery.com/episode-55-andreas-stefik/ Evidence-based programming language design : a philosophical and methodological exploration: https://jyx.jyu.fi/dspace/handle/123456789/47698 Is another great read if you've got the time.

If you want an overview of the ideas behind this sort of research and a quick summary of some results, Greg Wilson gave a great talk on it[0]. I haven't read through the site to see what is there, but software engineering methodology and technique research* uses techniques from research of management techniques in business, making it closer to psychology or sociology. For more information, the blog "It Will Never Work in Theory"[1] does a good job of highlighting these sorts of results that are directly useful and has some explanation of the tools they are using to study software engineering practices. The book Making Software[2] goes into much more detail on software engineering research methodologies if you are interested. *As opposed to CS theory research that could be used in software engineering, which is usually math. [0] http://vimeo.com/9270320 [1] http://neverworkintheory.org/index.html [2] http://shop.oreilly.com/product/9780596808303.do

Ah yes, the front page featuring the recorded talks kind of hides the real substance. Here's the full collection of papers, by category https://neverworkintheory.org/category/

After skim reading I couldn't see it mentioned, but when SourceForge started bundling malware[0][1] into the software they hosted, it was their death toll. [0]: https://neverworkintheory.org/2022/04/21/decline-of-sourceforge.html [1]: https://news.ycombinator.com/item?id=31110206 As my memory recalls it, that triggered an exodus to Google Code, and whilst GH was gaining traction it was somewhat in their shadow. When Google announced they were going to kill Code that was the blessing for GH.

A good resource for research on topics like this is It Will Never Work in Theory: https://neverworkintheory.org/category/#programming-languages, https://neverworkintheory.org/category/#types

Thanks, yes, I put the very interesting https://neverworkintheory.org/2022/08/30/software-engineering-research… in my newsletter yesterday :-)

I quite like https://neverworkintheory.org/reviews/ which has summaries of recent empirical research papers

You might like this site https://neverworkintheory.org/date/ It reviews academic papers in that kind (empirical software engineering).

This can't be said enough. At the end of the day programming is done by human beings, and it needs to be treated as a human endeavor. If I enjoy what I'm doing I'm going to do it better, it's really that simple. There is actual science to back up this idea as well http://neverworkintheory.org/2014/05/01/happy-sw-devs-solve-problems-b…

Via http://neverworkintheory.org/2016/10/02/filtering-spreadsheet-data.htm…

We changed the URL from http://neverworkintheory.org/2016/10/05/test-driven-development.html, which points to this. When the topic is controversial and the paper is not so specialized that only a few people here can understand it, changing the URL to that of the paper tends to help make a discussion more substantial. Especially when the blog post is more of a gloss on the paper than an in-depth commentary on it.

And if you want a TL;DR: http://neverworkintheory.org/2016/09/30/rethinking-git.html

I've recently seen some news about research trying to teach people programming. People learning language with randomly named keywords fared no worse than with English keywords (at least C-like). http://neverworkintheory.org/2014/01/29/stefik-siebert-syntax.html

Oh, or this investigation into programming language syntax for learnability! > Recent studies in the literature have shown that syntax remains a significant barrier to novice computer science students in the field. While this syntax barrier is known to exist, whether and how it varies across programming languages has not been carefully investigated. For this article, we conducted four empirical studies on programming language syntax as part of a larger analysis into the, so called, programming language wars. http://neverworkintheory.org/2014/01/29/stefik-siebert-syntax.html

I saw a cool study recently (summarized well here[1]) with an empirical experiment on how well code coverage predicts how well a test suite catches bugs. They found that the number of test cases correlated well with the test suite's effectiveness, but, when controlling for the number of tests, code coverage didn't. It was a pretty thorough study: > Our study is the largest to date in the literature: we generated 31,000 test suites for five systems consisting of up to 724,000 lines of source code. We measured the statement coverage, decision coverage, and modified condition coverage of these suites and used mutation testing to evaluate their fault detection effectiveness. We found that there is a low to moderate correlation between coverage and effectiveness when the number of test cases in the suite is controlled for. Given their data, their conclusion seems pretty plausible: > Our results suggest that coverage, while useful for identifying under-tested parts of a program, should not be used as a quality target because it is not a good indicator of test suite effectiveness. That's certainly how I approach testing: I value having a thorough test suite, but I do not treat coverage as a target or use it as a requirement for other people working on the same project. [1]: https://neverworkintheory.org/2021/09/24/coverage-is-not-strongly-corr…

Oh man, it is really funny that he ends it with telling people to read the study. To clarify the linked study is attempting to replicate https://dl.acm.org/citation.cfm?id=1070834, THE seminal study in Test Driven Development. Well to be more precise it was replicating an existing replication of that study which failed to replicate the original results. They were trying to modify the design so as to account for issues in the experimental design that may have led to the replicated study being inconclusive. This is significant because if you were not aware of the failed replication, and believed that TDD was supported scientifically as more productive because of that original study, then you SHOULD be reconsidering its place in your development process. If that isn't the case your opinion is unchanged by these particular results(even in the article inspiring this one the author admits that their opinion was already based on a much more thorough analysis, see: http://neverworkintheory.org/2016/10/05/test-driven-development.html). Now what I want to know is why people insist on writing articles in this awful conversation format. It wastes a lot of words to make a simple argument poorly.

Certainly, experience is a reasonable way to evaluate ideas, but that doesn't mean the evaluation is valid. I agree that the bloodletting example was meant to undercut the assertion. I also think the assertion - judging only on experience - is incorrect, and that the example is a valid one. We have many examples where experience lead to incorrect conclusions. By experience, Aristotle argued that things in motion come to rest. Everyone knew that ... until Galileo, further codified by Newton. One's experience can be wrong. Hence the reason for software usability research. See "Making Software", edited by Greg Wilson, for lay results of some of that research. Here's an example of a more direct, empirical study: http://neverworkintheory.org/2011/10/24/an-empirical-comparison-of-the… . You write "human beings rely on experience at all times to evaluate." We also use comparison testing, as with that paper link I gave you. A hope is to minimize preconceptions based on one's experience. We see this in the medical field all the time, where we find that even single blind testing can unduly affect the results. We also use prediction to evaluate. Einstein predicted a certain bending of light around the sun, which wasn't based on experience. Therefore, I place a weak meaning to "human beings rely on experience at all times to evaluate." We use it to guide our understanding, but also use other…