Hypothesis Blog

On the topic of fuzz testing, Python has an excellent library for property-based testing called Hypothesis [0] [1]. I don't think it does guided testing like AFL or libFuzzer (which OSS-Fuzz uses), but it's very powerful nonetheless. [0] http://hypothesis.works/ [1] https://github.com/HypothesisWorks/hypothesis-python

Regarding numbers 2 and 3, I believe you are describing "property-based testing"[0]. A Scala version of this is ScalaCheck and can be found here[1]. There appears to be at least one Rust library which claims to provide same, but I am not a Rust developer so cannot recommend any for fitness of purpose. 0 - https://hypothesis.works/articles/what-is-property-based-testing/ 1 - https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md

For those looking for property based testing for Python: https://hypothesis.works

Let's not forget that Postgres has first-class support for Python [0] (unlike JavaScript etc which are maintained, or not, by third parties). I don't advocate going for Python for simple stuff but when you do use it, it has the added benefit of having plain ol' Python functions that you can unit test thoroughly (including with the wonderful Hypothesis [1] property testing framework). [0] https://www.postgresql.org/docs/12/plpython-funcs.html [1] https://hypothesis.works/

I find (variants of) QuickCheck https://hypothesis.works/articles/quickcheck-in-every-language/ are pretty good at finding at least a lot of the invariant problems that TLA+ finds, and if you're lucky your language has it integrated already. The F# implementation FsCheck is pretty amazing.

The poorly named but very interesting "hypothesis"[0] is a quickcheck implementation (extension?) for python that might provide the "easy" access to these tools that you're looking for. [0] https://hypothesis.works/

No. I mean, conceptually, sure, it's very similar. But QuickCheck is data type driven. The underlying library here is ddmin, and it seems to operate on []byte directly: https://github.com/dgryski/go-ddmin/blob/master/ddmin.go --- The corresponding paper is here: https://www.st.cs.uni-saarland.de/papers/tse2002/tse2002.pdf (I am somewhat surprised that the paper doesn't mention QuickCheck at all. Although, I believe QuickCheck was published in 2000, so it had only been out for two years by then. So maybe I shouldn't be surprised actually.) See also: http://hypothesis.works/

I worked through some of these[0] a while back as a way to learn property based testing in Python with Hypothesis[1]. Never finished them, but it was a fun exercise. [0] https://github.com/avyfain/Cryptopals-Hypothesis [1] http://hypothesis.works

Some property testing tools like Python's Hypothesis[1] allow specifying specific example values for properties in addition to a general set of values so you get some specific deterministic tests. Hypothesis also saves falsifying values it found previously in a database that it reads from the next time you run it. [1] http://hypothesis.works/

I'm definitely not advocating for unnecessarily complex or long conditionals on a single line or any other hard-to-grok code, but this comment reminded me of something I just learned recently! Someone on my team introduced me to property-based testing, which generates ranges of test cases that would otherwise be very repetitive to write manually. We've used http://hypothesis.works/ in a few places recently and caught bugs that we might have missed if we had just written tests the usual way.

I have to admit the last time I seriously worked with QuickCheck was quite a while ago. > My understanding is that Haskell’s QuickCheck can shrink functions [...] Are you talking about shrinking a value of type (a -> b) (for some concrete a and b)? In Hypothesis, if you write a generator that spits out functions, you get a shrinker for free. So that shrinker will shrink functions. As far as I can tell, QuickCheck's shrinking is based on values. I just checked, the type is still `shrink :: a -> [a]`, so that can't have changed. In Hypothesis shrinking is based on the generation process. In Hypothesis generators act a bit like 'Functors' in Haskell, ie you can map over them. (You can also filter and `<>` and do monadic binds etc.) So in Hypothesis you can take a generator that produces integers, and create one that produces only even numbers by (in Haskell terms) mapping (2) over it. Now, the shrinker you get for free will also only produce even numbers. That's not possible (or at least not easily possible) in Haskell. You would have to define at least a newtype wrapper. Have a look at https://hypothesis.works/articles/compositional-shrinking/ and https://hypothesis.works/articles/integrated-shrinking/ for some more write-up, especially about how Hypothesis can preserve information even when shrinking past a monadic bind. If you follow some links in the two articles above, you land at https://github.com/icicle-lang/disorder.hs-ambiata/tree/master/disorde…

I agree. One nice feature of property-driven testing is that assumptions often end up causing test failures. For example (in ScalaTest): "Average of list" should "be within range" in { forAll() { (l: List[Float]) => { val avg = l.average assert(avg >= l.min && avg <= l.max) } } This test will fail, since it doesn't hold for e.g. empty lists. Requiring non-empty lists will still fail, if we have awkward values like NaNs, etc. The following version has a better chance of passing: "Average of list" should "be within range" in { forAll() { (raw: List[Float]) => { val l = raw.filter(n => !n.isNaN && !n.isInfinite) whenever (l.nonEmpty) { val avg = l.average assert(avg >= l.min && avg <= l.max) } } } Getting this test to pass required us to make those assumptions explicit. Of course, it doesn't spot everything; here's an article which explores this example in more depth (in Python) https://hypothesis.works/articles/calculating-the-mean

People can have different definitions and still communicate usefully, and I think there is not 100% agreement on the exact boundaries between the two. That said, for me: they are distinct but related, and that distinction is useful. For example, Hypothesis[1] is a popular property testing framework. The authors have more recently created HypoFuzz[2], which includes this sentence in the introduction: “HypoFuzz runs your property-based test suite, using cutting-edge fuzzing techniques and coverage instrumentation to find even the rarest inputs which trigger an error.” Being able to talk about fuzzing and property testing as distinct things seems useful — saying something like “We added fuzzing techniques to our property testing framework” is more meaningful than “We added property testing techniques to our property testing framework” ;-) My personal hope is there will be more convergence, and work to add convenient first-class fuzzing support in a popular language like Go will hopefully help move the primary use case for fuzzing to be about correctness, with security moving to an important but secondary use case. [1] https://hypothesis.works [2] https://hypofuzz.com

Python may be terrible, but this poster doesn't know Python at all. Here's how to fix the first snippet: leaf = object() Huh, that's funny, it's shorter than the Haskell? Why is that? Let's keep going. def merge(lhs, rhs): if lhs is leaf: return rhs if rhs is leaf: return lhs if lhs[0] <= rhs[0]: return lhs[0], merge(rhs, lhs[2]), lhs[1] return rhs[0], merge(lhs, rhs[2]), rhs[1] Ugh. My mouth tastes funny. Livecoding on this site is always disorienting. I need to sit down for a bit. Exercise for the reader: Continue on in this style and figure out whether the Haskell really deserves its reputation for terseness and directness. Edit: I kept reading and was immediately sick. __dict__ abuse is a real problem in our society, folks. It's not okay. def insert(tree, elt): return merge(tree, (elt, leaf, leaf)) The Haskell memes are growing stronger as I delve deeper into this jungle. The pop-minimum function here is a true abomination, breaking all SOLID principles at once. I can only imagine what it might look like in a less eldritch setting: def popMin(tree): if tree is leaf: raise IndexError("popMin from leaf") return tree[0], merge(tree[1], tree[2]) We continue to clean up the monstrous camp. def listToHeap(elts): rv = leaf for elt in elts: rv = insert(rv, elt) return rv The monster...they knew! They could have done something better and chose not to. They left notes…

I brought in the QuickTheories[0] library for property testing at work after being exposed to the articles here[1] (the final rule test is particularly interesting). My first use case was to test a generated parser for a restricted subset of the grammar for content security policy URLs, but I think this might be a bad use case since I'm not exactly testing "properties" so much as using the system for test data generation and quick code coverage. e.g. I have a test called something like testUrlPaths() that's mostly exercising the parser to make sure it doesn't forbid a valid but strange URL path that someone could enter. However while writing that test I discovered the parser was really doing the right thing, I found some violations with paths containing a '%' but not two hex digits immediately following. Unless I was writing unit tests for every bit of the grammar and paid attention to that part, I doubt I would have thought to test for them myself... Maybe the happy path of '%20' but not the fact that '%2' should be invalid. In the end I extracted those cases into three specific tests that will be there in case someone gets the funny idea to replace the validator with something not generated from a grammar (or generated from the wrong grammar), which isn't that unlikely since the spec allows for a lot more than what one might naturally consider allowing on your own (e.g. ftp resources). Like with certain kinds of unit testing, I have the sense that the process of using a…

We have about ten engineers, so nowhere near the size where people consider formal methods "appropriate". Nonetheless it's still been incredibly useful for our work. I'm a pretty huge evangelist of TLA+, but I don't think it's the silver bullet of software correctness. It just happens to be the tool I'm most familiar with and the one I thought could benefit most from a free guide. If people start widely using TLA+, I'll be ecstatic. If people ignore TLA+ but start widely using Alloy, I'll still be ecstatic. Software correctness is a really huge field and there's lots of really cool stuff in it! Speaking of making methods more accessible, I'm working on a tutorial about Stateful Testing. Hypothesis (https://hypothesis.works) is an absolutely incredible property-based testing library for Python, and I think it could potentially make PBT a mainstream technique. One of the more niche features is that you can define a test state machine that runs by randomly selecting transitions rules and mutating your program state, then running assertions on the new state. It's really neat!

You might try this blog entry (or other blog entries that talk about shrinking): https://hypothesis.works/articles/how-hypothesis-works/

The distinction between property based testing and fuzzing is basically just a rough cluster of vibes. It describes a real difference, but the borders are pretty vague and precisely deciding which things are fuzzing and which are PBT isn’t really that critical. - Quick running tests, detailed assertions —> PBT - Longer tests, just looking for a crash —> fuzzing. - In between, who knows? https://hypothesis.works/articles/what-is-property-based-testing/

> and also to automatically shrink any counterexamples it finds to be as simple as possible. This actually is a problém with most quickcheck clones (and quickcheck itself), that's why it's better to use a library that is inspired by Python's Hypothesis (combining generation and shrinking, in a nutshell). So Hedgehog for Haskell and F# (instead of Fscheck) and Rapid for Go https://github.com/flyingmutant/rapid But tables and property tests complement each other, even if you can get rid of some tables by using property testing. Example article about the shrinking of Hypothesis vs. Quickcheck https://hypothesis.works/articles/integrated-shrinking/

https://hypothesis.works/articles/ They have a bunch of articles here including a variety that delve into the "how" of Hypothesis. Here is a 2016 article titled "How Hypothesis Works": https://hypothesis.works/articles/how-hypothesis-works/

Can AI work better than property-based testing? https://hypothesis.works/articles/what-is-property-based-testing/

Property testing [1] is about semantics, the same as regular testing, and it does not require code annotations. [Edit: It's more like "fuzzing" than "model checking".] Type systems on the other hand aren't usually powerful enough to capture semantics. They allow you to say things like, this function takes in a list of Foo objects, and returns one Foo object. But they don't really let you express whether the object returned is or is not part of the original list, and if it is, that it was chosen according to the right mechanism. That's what tests are for. Without being able to express the semantics of code, I don't see how you can trust it. There may not be any type mismatches, but there sure can be lots of bad logic in there. [1]: http://hypothesis.works/articles/what-is-property-based-testing/

This library does more than "random" testing: when it finds a failure, it will try to reduce the input (called "shrinking") to report a minimal example. Here are some similar tools for other languages: http://hypothesis.works/articles/quickcheck-in-every-language/

FWIW, rantly (https://github.com/abargnesi/rantly) rather than propr seems to be the most active and usable Ruby property based testing library. (I maintain a list here: http://hypothesis.works/articles/quickcheck-in-every-language/)

Actually, I'd say writing perfect code in the mathematical domain is extremely hard - floating point types are a minefield and infinite-precision arithmetic is a minefield also if you consider the possibilities for running-times exploding. For example, code for the mean of a list of numbers is a hard problem when looked at in its full generality [1] - and that's using forgiving definition of "perfect" - alway correct and terminating, running at a reasonable speed. [1] http://hypothesis.works/articles/calculating-the-mean/

> In Hypothesis shrinking is based on the generation process. In Hypothesis generators act a bit like 'Functors' in Haskell, ie you can map over them. (You can also filter and `<>` and do monadic binds etc.) You are right that the generators in Hypothesis correspond to `Functor`s, so you can map over it. Indeed, in QuickCheck, its `Gen` [1] (type of `arbitrary` [2], the generator in QuickCheck) is a `Functor`, and moreover a `Monad`, so you can do monadic binds on `Gen`. > So in Hypothesis you can take a generator that produces integers, and create one that produces only even numbers by (in Haskell terms) mapping (2) over it. Now, the shrinker you get for free will also only produce even numbers. > That's not possible (or at least not easily possible) in Haskell. You would have to define at least a newtype wrapper. Since `Gen` in QuickCheck is a `Functor`, to multiply its output by 2 to generate only even numbers, you can do this in Haskell as: fmap (\x -> x*2) (arbitrary @Int) Or, for those preferring OOP-style chaining: arbitrary @Int <&> (*2) where `<&>` is `fmap` with arguments flipped, and `(*2) = \x -> x*2`, and `@Int` is type application to specialize `arbitrary` to generate `Int`. > Have a look at https://hypothesis.works/articles/compositional-shrinking/ and https://hypothesis.works/articles/integrated-shrinking/ for some more write-up, especially about how Hypothesis can preserve information even when shrinking past a monadic bind. I think what…

Well, normally Hypothesis just calls a function with some random arguments and checks that it "works", operationalized as "doesn't crash". But that's inadequate for testing stateful APIs, in which calling a single function isn't useful — for stateful APIs, you need to go through a series of calls, not just a single call. So the rule-based state machine thing gives you a way to describe the valid series of calls, and Hypothesis then generates 100 random sequences according to your spec and verifies that they all work. David wrote the article about it in 2017 that felixyz already linked: https://hypothesis.works/articles/rule-based-stateful-testing/ In many cases, though, it's better if you can refactor your code to support a stateless interface, and then you can write simple single-step function tests. How do you know what to test? There's a whole spectrum. Just testing that valid inputs don't crash is often pretty useful. (If you're not sure what valid inputs would be, maybe try writing the README before the tests.) In the case of Dumpulse, I went to the other extreme; its actual semantics are extremely dumb (thus the name), so the test suite includes a reimplementation in Python of the intended semantics, and the test verifies that the implementation in C has the same behavior. (But in constant space and strictly limited runtime, which the Hypothesis test doesn't check.) I picked a few functions and classes at random to see what kind of testable properties I…

Fuzzing can refer to randomly generated test inputs without bothering to minimize them when bugs are found. Property-based testing tools like Hypothesis and QuickCheck (which pioneered this approach), in addition to randomly generating test cases, when they find a test case that violates an assertion, they then start reducing the test case to try to find the minimal test case that violates the assertion. Also, while they're very similar in a mechanical sense since they're both about randomly generated test inputs, in a broader sense my impression is that fuzzers have a very different focus from property-based testing. I think fuzz testing typically refers to testing very complex software, like compilers, interpreters, and virtual machines, and searching for crazy edge cases, especially ones that may expose security vulnerabilities. So a lot of effort goes into generating test inputs that are unlikely to happen with a naive testing strategy (automated or manual), yet still interesting and relatively likely to trigger a bug (rather than spending too much time in the very large space of uninteresting test inputs that a naive testing strategy wouldn't come up with but also is unlikely to trigger a bug). By contrast, I think property-based testing is typically used for testing programs that are of ordinary complexity, and quickly finding bugs due to simple but common programmer errors, which can then be quickly fixed. So a lot of effort goes into reducing any bugs that are…