Jeffrey Kegler

I've had great success with a parser-generator known as "Marpa" [1]. One of the things I particularly liked about it is how simple it made to emit accurate and useful error messages when parsing complex languages. In addition, it can handle anything that can be expressed in BNF, and it's quite fast. [1] https://jeffreykegler.github.io/Marpa-web-site/

my favorite Gödel, Einstein story is when he was prepped by Einstein for an US immigration hearing, and then told the judge he found some loopholes in the constitution. Was it his character shaping his work, or his work and surrounding shaping (making) him, and if so what shapes a man like Gödel (or Freud, Einstein, Popper, Russel, ...) many others of that time. It seems there was a huge density of brilliant people concentrated in Europe at the time. Hardly anyone who died within the past 30 years will be remembered as those from that generation I'm sure. Now everyone is writing AI papers which all look identical. Sure academia has changed. A Contradiction in the U.S. Constitution https://jeffreykegler.github.io/personal/morgenstern.html Gödel’s Loophole https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2010183

Because any system that lets you modify the rules is inherently broken. There’s some apocrypha involving Kurt Gödel and a loophole in the US Constitution that gets at this point [1]. But it’s really not the system’s fault, it’s the users’ fault. Any rule-making system worth writing home about is exploitable, because that’s the only way it’s useful. [1] https://jeffreykegler.github.io/personal/morgenstern.html

It's often I stumble upon discussion mentioning the term LL and LR and other stuff related to parsing[0]. Without a proper CS background, it's quite hard to follow along. That first link[1] and the wikipedia page[2] mentioned there are really great. Many thanks for posting those. It really shed some light about those terms. [0]: https://jeffreykegler.github.io/personal/timeline_v3 [1]: http://blog.reverberate.org/2013/07/ll-and-lr-parsing-demystified.html [2]: http://en.wikipedia.org/wiki/Tree_traversal

Superb website with loads of content. This https://jeffreykegler.github.io/personal/timeline_v3 is also worth your time twofolds.

The author of the PEG script cannot in general know what language it actually describes -- it's just too hard. For more, see https://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/201… It has references, which you can follow up on. You're safe if the grammar is LL(1) -- what you sees is what you get. After that it gets mysterious fast.

Yeah, Marpa is quite amazing—it inspired me to build nearley. It's probably more powerful than any other parser generator out there right now (and the fastest, because it has Leo optimizations and Aycock-Horsepool nullables). In fact, Kegler responded to this same article with http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2012…

LR parsers like yacc are obsoleted by Earley parsers, which Cox apparently didn't know about in 2010. Quoting <http://loup-vaillant.fr/tutorials/earley-parsing/what-and-why#Why>: The biggest advantage of Earley Parsing is its accessibility. Most other tools such as parser generators, parsing expression grammars, or combinator libraries feature restrictions that often make them hard to use. Use the wrong kind of grammar, and your PEG will enter an infinite loop. Use another wrong kind of grammar, and most parser generators will fail. To a beginner, these restrictions feel most arbitrary: it looks like it should work, but it doesn't. There are workarounds of course, but they make these tools more complex. Earley parsing Just Works™. On the flip side, to get this generality we must sacrifice some speed. Earley parsing cannot compete with speed demons such as Flex/Bison in terms of raw speed. It's not that bad, however: • Earley parsing is cubic in the worst cases, which is the state of the art (and possibly the best we can do). The speed demons often don't work at all for those worst cases. Other parsers are prone to exponential combinatorial explosion. • Most simple grammars can be parsed in linear time. • Even the worst unambiguous grammars can be parsed in quadratic time. My advice would be to use Earley parsing by default, and only revert to more specific methods…

maybe have a look at "PEG: Ambiguity, precision and confusion": https://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/201…

Yeah, agreed. I just wish someone would write a "how Marpa parsing works for dummies" article. I've been curious about how it actually works ever since I was colleagues with Aria Stewart, who ported it to JS once upon a time and mentioned it's existence to me (we worked remotely and in opposite time-zones, so I never had a chance to sit down with her for a good old in-person explanation). [0] https://savage.net.au/Marpa.html [1] https://jeffreykegler.github.io/personal/timeline_v3 [2] https://github.com/aredridel/lotsawa

For more parsing history, see Jeffrey Kegler's "Parsing: a timeline": https://jeffreykegler.github.io/personal/timeline_v3 For a more elegant solution, see these tiny languages parsed using Floyd's operator precedence parsing: http://breuleux.net/blog/language-howto.html https://news.ycombinator.com/item?id=8554084

From what I infer from articles like https://jeffreykegler.github.io/personal/timeline_v3 , the original Earley paper had a bug, and wasn't a good fit for 1960s hardware, and had poor performance for some types of grammars. By 1991, "Most researchers see the Parsing Problem as "solved" -- a closed issue. Earley parsing is almost forgotten, and Leo's discovery is ignored. Two decades will pass before anyone attempts a practical implementation of Leo 1991." It takes Aycock and Horspool's work in 2002 and Kegler's work in 2010 in Marpa to have a "practical implementation" (quoting that link). (I quote that also because Aycock distributed SPARK, an Earley parser, which was included as part of the Python distribution, in the Parser/ subdirectory, and a couple of people here on HN report having used it.)

I haven't looked at this book yet but I did like HPMOR. In a different vein, "The God Proof" (a mystery novel revolving around Kurt Gödel's ontological proof of God's existence) was a fun read: https://jeffreykegler.github.io/personal/gp.html

There is some more information at https://jeffreykegler.github.io/personal/morgenstern.html There's also an article about this question at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2010183

In my experience LPeg does a good job of being comprehensible for most reasonably-sized use cases. It can even parse the grammar of Lua itself. I've personally had not that much trouble translating BNF grammars to PEG, though the result is typically longer. It still satisfies the goal of separating parsing logic from data logic, which is a big step. For more info on the issues with PEGs -- and a paper showing how to correctly translate general LL(1p) grammars to PEG -- see: https://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/201…

> PEGs are however are easier to reason about, no ambiguities, so that was good enough reason. I frequently hear this mentioned ("PEGs don't have ambiguity"). It is literally true, but I don't think it's true in the sense that actually matters. I've blogged about this in the past (https://blog.reverberate.org/2013/09/ll-and-lr-in-context-why-parsing-… but I'm not the only person saying this: > PEG is not unambiguous in any helpful sense of that word. BNF allows you to specify ambiguous grammars, and that feature is tied to its power and flexibility and often useful in itself. PEG will only deliver one of those parses. But without an easy way of knowing which parse, the underlying ambiguity is not addressed -- it is just ignored. https://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/201…

A practical implementation of Earley parsing, with some cool features, is Marpa [1]. In particular it is practical in that it parses in linear time * all the grammar classes that recursive descent parses * the grammar class that the yacc family parses * most unambiguous grammars * all ambiguous grammars that are unions of a finite set of any of the above grammars. The author's blog [2] presents some interesting features of Marpa, as well as interesting discussion of parsing in general. [1] http://jeffreykegler.github.io/Marpa-web-site/ [2] http://jeffreykegler.github.io/Ocean-of-Awareness-blog/

I had expected this would be a retelling of how Kurt Gödel (of Incompleteness Theorems fame) claimed to have found a logical flaw in the U.S. Constitution and was dissuaded from brining it up at his naturalisation interview. I was pleasantly surprised it was nothing of the sort. https://jeffreykegler.github.io/personal/morgenstern.html

I know nothing about parsers or Marpa, but I can offer a link for the curious: http://jeffreykegler.github.io/Marpa-web-site/

Yes. https://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/201… have some discussion on it, and references for further reading (and a whole lot of making his own parser look powerful in comparison. It certainly is. I just wish the code of his parser was closer to the C I've seen before).

In the comments, there's a link to this timeline of parsing techniques with one or two things I hadn't seen before: https://jeffreykegler.github.io/Ocean-of-Awareness-blog/ Author then modernizes one of the better ones.

I believe that the Marpa parser may be what you're looking for. A C library with the most refined API in Perl5, improvements are in the works to make it easier to produce interfaces to the library in other languages. https://jeffreykegler.github.io/Marpa-web-site/ http://savage.net.au/Marpa.html

Seems to be written in C: "Marpa is available as a open-source library. It is written in C, and the C library can be used directly or via a Perl interface." http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2012…

> I'm currently delving into Earley Parsing, so I can write compilers easily, so I can write DSLs economically, so I can write short programs in any language, so I can spend less time writing those programs, so I can generate less bugs from those programs, so these programs ultimately cost less money$$$. 'Earley parsing' immediately trips a flag in my memory, so I'll just ask—have you seen Jeffrey Kegler's Marpa parser (http://jeffreykegler.github.io/Marpa-web-site, with associated blog)? (I don't know you, so I apologise in advance if you are Jeffrey Kegler. :-) )

I believe YAML has serious issues and I would not be using it as a configuration format. But I think a lot of the criticism around YAML misses the mark. YAML is not excessively overused as configuration language. It is not dominant like XML was back in its heyday. TOML has a pretty strong mindshare, and many projects are still using JSON or XML. I would pick TOML over YAML any day for this usage, but I still think YAML beats XML and JSON (just think about comments and terrible merge resolution whenever you add an item to an array or an object). But YAML does reign supreme in the world of DevOps-oriented DSLs. YAML is more suited for DSL compared to alternatives, like TOML, JSON, JSON5 or XML: - YAML is the least verbose of all alternatives (especially XML, but even JSON5 and TOML). - YAML requires the least of amount of "syntax noise" on top of your DSL. The "syntax noise" is mostly limited to indentation, colons, dashes (for arrays), indented text block markers and occasionally quotation marks. JSON, JSON5 and TOML require quoting all string values and explicitly marking arrays and object boundaries, which make the DSL seem less fluent. XML is even worse. - YAML supports indenting multi-line strings, so you can easily embed other formats. This makes adding markdown descriptions (in Swagger/OpenAPI) or embedded shell scripts (in YAML-based CI/CD DSLs) a breeze. - YAML syntax is hierarchical. The same is true for JSON and XML, but not for TOML. TOML syntax only has two…

At the end, Kegler links to this comprehensive overview of the history of parsing: https://jeffreykegler.github.io/personal/timeline_v3 which contains this easily overlooked but IMHO extremely significant statement: "a recursive descent implementation can parse operator expressions as lists, and add associativity in post-processing" Personally, it has always seemed like a no-brainer to me that this is clearly the Right Answer. It is a mystery to me that the computing world at large has spent so much effort on a problem whose solution is actually very straightforward if you just give in on one tiny little piece of theoretical purity. See http://www.flownet.com/ron/lisp/parcil.lisp for my own implementation of such a parser. As you will see if you count LOCs, it's very, very simple by parser standards, and yet it handles all the "hard" problems: associativity, precedence, infix and prefix operators.

Yes, yes they are. I've used them successfully for various parsing projects. And in fact Bison recently picked up GLR parsing to deal with multiple options (https://www.gnu.org/software/bison/manual/bison.html#Generalized-LR-Pa… But for something even more fun, I suggest you check out Marpa (https://jeffreykegler.github.io/Marpa-web-site/). It is, by far, the nicest parsing tool I have ever worked with. Being able to parse ANY BNF is huge. And the state tables for debugging! Knowing what possibilities you have makes things way easier. Especially for getting stuck in an ambiguous spot of your grammer. Finally, using ruby slippers to get out of a tight spot is hugely useful. My one complaint is that the only implementation is in Perl. The core engine is written in C though, so someday I would like to convert the Perl into Python. I started several months ago, and haven't been able to get back to it. (https://github.com/srathbun/pyMarpa)