LGTM Blog

https://lgtm.com/ LGTM recently came by my project offering to integrate with GitHub. It's found a decent number of bugs other linters and my test suite didn't find, and it didn't require any setup at all; you just open your project on their website. Their staff also reacted really quickly when I asked them questions or reported bugs. For the most part, their only false positives are things like unreachable code, which I sometimes used to leave in for future-proofing, but I eventually did things their way, because it does make sense that it could be confusing to a third party. The only alert I've ever suppressed was a string search for "example.com", which could have matched "example.com.untrusted-attacker.com". It was just code for displaying a reminder, not anything security-critical, so I didn't think it was worth fixing.

C-Reduce has been a godsend at my job. I work on the tool that converts C/C++ programs to our internal database format for https://lgtm.com/ and I regularly have to investigate issues we have handling large open-source projects. C-Reduce is the difference between having to debug megabytes of (preprocessed) source code and a few tens of bytes. Great job by all involved.

Semmle | Research/Software/Security Engineer | Oxford, UK | Onsite, Full-time Semmle helps some of the biggest software and financial companies, such as Microsoft, Google, and Credit Suisse, develop more secure software, faster. If you'd like to get a taste of our technology go to https://lgtm.com where we analyse over 70,000 of the most popular open source projects, or check out the vulnerabilities we've recently reported at https://lgtm.com/blog We are looking for great engineers work on a mix of problems ranging from compiler work and program analysis to vulnerability finding to building solid 1000-node distributed systems. Go to https://semmle.com/careers or email [email protected]

I wonder if it is or will be possible to set this up without littering the repo with another configuration file. In the past, I've done a couple one-off scans at https://lgtm.com/

Has anyone tried LGTM / Semmle QL for automated code review? They claim 100K OSS projects are using the service. It's a bit hard to find technical information on the product, but they have found CVEs in mainstream products, including iOS. https://lgtm.com & https://semmle.com/ql

Not terribly impressed. This wouldn't pass my code review: https://github.com/TTimo/doom3.gpl/blob/aaa855815ab484d5bd095f34716319… Function is 200 lines long, with 8 or so levels of nesting. Also "goto breakout" To address the objections below, the function reads from a file into a pixel buffer. It's not some tricky in-place update. That's a great candidate for a more functional style. Here's more ick: https://github.com/TTimo/doom3.gpl/blob/aaa855815ab484d5bd095f34716319… Surely that could be factored better. Down-voters go read Carmack's own article: https://gamasutra.com/view/news/169296/Indepth_Functional_programming_… I think it's funny that people disagree here. This is exactly the stuff a modern linter would flag in an automated code review. Sure enough, here's a linter. I think this is essentially the same codebase: https://lgtm.com/projects/g/Edgarins29/Doom3/context:cpp "Code quality: D" (on an A-F scale) And by the way, I have the utmost respect for Carmack. I just wouldn't hold up this codebase as great.

I started Sage, and worked on it a huge amount 2004-2012, but less so recently (due to working mostly on CoCalc). Actual contributions to and work on Sage by the community have gone UP a lot since 2012, as this plot (https://github.com/sagemath/sage/graphs/contributors) at Github shows. Much of that work is from research mathematicians that are adding large swaths of important functionality (e.g., in differential manifolds) to Sage. But there's also a very welcome push for quality, e.g., the number one recent contributor, fchapoton, has been fixing every little detail in the codebase revealed by https://lgtm.com/ and other tools, and there's also been a lot of work to fully transition Sage to Python 3 (which is difficult since so much is written in Cython and make nontrivial use of comparisons). I would have likely made Sage pip-installable from day 1 if I had known that pip would turn out to be so popular. It was really unclear what was going on with Python packaging 15 years ago. I'm really glad that one package manager (pip) has become so good at this point.

Yeah, found this which explained it for me :) https://lgtm.com/rules/2154840805/

I'm pretty sure you could google each of these with the word 'dangerous' For example: https://lgtm.com/rules/2154840805/

> Security vulnerability finding is almost certainly the wrong target for…

First project I look up on lgtm.com is rust.. Second alert I find is this: https://lgtm.com/projects/g/rust-lang/rust/snapshot/f5aa590b8ca98e925e… exist_ok is available from python 3.2, so this isn't a good impression. https://docs.python.org/3.7/library/os.html#os.makedirs

Also as an example of "always deserialize to known types". Flexible boolean values can be convenient since it's relatively human-readable, but "deserialize into [whatever the heck you think is appropriate]" is a problem for quite a few reasons beyond confusion: https://lgtm.com/blog/swagger_snakeyaml_CVE-2017-1000207_CVE-2017-1000… (same techniques have been used against other kinds of serialization in many languages for many years)

Systemd and tesseract-ocr both use it for example: https://github.com/tesseract-ocr/tesseract https://github.com/systemd/systemd systemd have also written their own QL query: https://github.com/systemd/systemd/blob/master/.lgtm/cpp-queries/fgets… https://lgtm.com/projects/g/systemd/systemd/alerts/?mode=tree&rule… (full disclosure, I also work at Semmle)

Disclosure: I’m the founder of Semmle. As noted above, this conversation is a dup but I wanted to answer the questions that came up. Semmle’s core technology is an engine for variant analysis - finding all instances of a mistake that led to an incident, especially for security. Here's how Microsoft uses Semmle: https://blogs.technet.microsoft.com/srd/2018/08/16/vulnerability-hunti… . The engine does support inter-procedural data flow analysis and simulates execution in that sense. For examples of such data flow analysis on Java, see https://lgtm.com/blog/apache_struts_CVE-2018-11776-part2.

To sum up your link, the vulnerability is the use of an unsafe deserialization similar to: ObjectInputStream ois = new ObjectInputStream(input); MyObject obj = (MyObject)ois.readObject(); https://lgtm.com/blog/finding_unsafe_deserialization_with_ql

Regarding CVE-2017-9805, I’m genuinely in awe over how remote code execution in Java is even possible. Why should it even be possible to deserialize data on the wire into executable code? Also, can someone shed light on the technical side of this? E.g., how does the JVM compile Java code it receives on the wire into Java byte code? Does the JVM runtime have a built-in Java compiler that outputs Java byte code that it itself can execute? https://lgtm.com/blog/apache_struts_CVE-2017-9805

No, the vulnerability was reported to the Struts project on July 17. https://lgtm.com/blog/apache_struts_CVE-2017-9805

To be honest, I feel bad for the engineering team at Equifax. The vulnerability that compromised their system was a bug in an open-source Java library, Apache Struts, and security researchers only noticed it a few days ago. It seems that the Equifax team had very little time to react and update their software. In some sense, I feel that more blame should be placed on the engineers who built the highly popular open-source software, not the Equifax team. Some large number of Fortune 100 companies also experienced the same vulnerability simply because they trusted a widely used library. Makes me wary of trusting other big OS libraries, but since rebuilding every part of the stack from scratch is infeasible and unproductive, we don't have much choice but to use them. Technical announcement: Severe security vulnerability found in Apache Struts using lgtm.com (CVE-2017-9805): https://lgtm.com/blog/apache_struts_CVE-2017-9805_announcement