Gunnar Morling

Great to see more love for using Lucene on Lambda via GraalVM native binaries! Explored that same idea few years back for my personal blog [1]. It worked pretty well, but I moved eventually away from Lambda as I wanted to close down my personal AWS account. One thing I always meant to explore was embedding the (in my case small) Lucene index directly into the binary image using Lucene's RAMDirectory via Quarkus' bytecode recording machinery, but I never got to it. [1] https://www.morling.dev/blog/how-i-built-a-serverless-search-for-my-bl…

> -XX:+TraceClassLoading -XX:+TraceClassUnloading The best insight into the operation of the JVM is now obtained via a single mechanism, JFR (https://dev.java/learn/jvm/jfr/), the JDK's observability and monitoring engine. It records a whole lot of event types: https://sap.github.io/SapMachine/jfrevents/ See here for examples related to tracking memory: https://www.morling.dev/blog/tracking-java-native-memory-with-jdk-flig…

haha, I've definitely seen some of those. Comment standards are worth obeying, personally I like the code review pyramid https://www.morling.dev/blog/the-code-review-pyramid/

Great writeup. Writing your own search index and crawler from scratch is a big undertaking, but sounds like the sort of thing you might have to do due to the constraints of Lambda. For the search index, the blog you link to (https://www.morling.dev/blog/how-i-built-a-serverless-search-for-my-bl… does use Apache Lucene within Lambda (compiled into a native binary via Quarkus and GraalVM to make startup time viable, although not a distributed index and doesn't need to be because of its relatively small size), and for the crawler, it sounds like you were nearly there with Colly (except for memory issues). On the non-tech side: >"In February 2021 the Australian Greens Party called for a publicly owned search engine to be created and be independent and accountable like the ABC." Which is an interesting idea, given search has become something of a utility.

Implement some none-trivial algorithm and then try to optimize the heck out of it. Focus on the concrete CPU, architecture, cache, and memory in the hardware you have in front of you. Forget portability for a moment. This comes to mind: https://www.morling.dev/blog/one-billion-row-challenge/ Read how others have done it. Here's an example in Java that goes relatively low-level to squeeze out performance: https://questdb.io/blog/billion-row-challenge-step-by-step/

I haven't had chance to use it properly yet, but I think JfrUnit[0][1] is a good option here to test non-functional behaviour like memory usage and garbage collection. [0]: https://github.com/moditect/jfrunit [1]: https://www.morling.dev/blog/towards-continuous-performance-regression…

I found an online version https://icu4c-demos.unicode.org/icu-bin/collation.html but there you first have to select the language from the drop-down menu. So in general, you would first need to know the country where your weather station was located, before you can correctly collate its name. I don't believe the fastest entries are doing all this(?) Edit: In the examples [1], the guy writes a Polish city using its English name "Cracow". So you can't choose the alphabetical ordering based on the geographical location of the weather station, but you need to somehow detect in which language its name is written in, in the data. [1] https://www.morling.dev/blog/one-billion-row-challenge/ Edit2: I guess you could declare that either the "Default Unicode Collation Element Table (DUCET)", or perhaps the American English "en-US-u-va-posix" locale is the correct way to alphabetize.

Yes[1], but you need a relatively new version of JFR and you also need to enable a few flags to get much value out of it. [1] https://www.morling.dev/blog/tracking-java-native-memory-with-jdk-flig…

Does it provide hard latency bounds like Azul does (did?), and are they lower than disk/network latencies on modern hardware? I moved to c++/rust years ago because those languages do, and tens of milliseconds matter for network services. At the time Java could pause for 10’s of seconds, which was 1000x worse than waiting for a spinning disk to seek. These days, disks are 100s micros to single digit millis, so I guess if Java GC is finally working 30 years after they “fixed” its performance problems, then I’d want to be able to tune ZGC to not pause the app for more than ~ 500us, max. This article is from last year, but suggests they’re still off by an order of magnitude: https://www.morling.dev/blog/lower-java-tail-latencies-with-zgc/ Also, that’s measuring a 30 second window. If you hammer a 100GB-1TB heap in steady state with small allocations for, say, a month at 100% CPU, does it eventually do the typical Java thing, where a major compaction takes the process down for seconds or even minutes, or does it just slow down application requests so it can keep up with load?

Thanks for the comment! Your comment regarding DDL is interesting. Today, this is what happens: 1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value). 2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-line blocking reasons being: this is anti-pattern and should be rare, in which case - it's okay to cause an err and require manual intervention for this breaking change. 3/ If the column has been dropped from the source, you as the end user can decide whether this column should be also dropped in the destination, or not. The default is not to drop it. ^ We hear more customers explicitly don't want columns to be dropped because it could cause downstream errors, such as other views / tables not compiling due to referencing a non-existent column. We haven't heard much from folks that don't even want columns to be added. If there is a need, we can definitely add that as a config option to provide maximum configurability. > Finally, the biggest issue with CDC always ends up being the seed loads, recoveries and the incremental snapshot strategies. Yep totally. On the recovery bit, this is exactly why we are leveraging Kafka. If there are any particular issues, we simply don't commit the offset and cause head-of-line blocking. On the incremental snapshot and…

If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format (<insert audible sigh here>) it can definitely make sense to use a high-speed optimized ingester. One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which means any form of SIMD optimization is right off the table (okay, maybe something could be done in Java, but it seems incredibly complex, see https://www.morling.dev/blog/fizzbuzz-simd-style/): - PHP isn't optimized anywhere, but at least it's C: https://github.com/php/php-src/blob/1c0e613cf1a24cdc159861e4406abfcee7… - Python's C implementation is the same: https://github.com/python/cpython/blob/main/Modules/_csv.c - Java doesn't have a "standard" way at all (https://www.baeldung.com/java-csv-file-array), and OpenCSV seems the usual object-oriented hell (https://sourceforge.net/p/opencsv/source/ci/master/tree/src/main/java/… - Ruby's CSV is native Ruby: https://github.com/ruby/ruby/blob/bd65757f394255ceeb2c958e87f862139dd3…

Much better, because you still do have the JIT data, it is similar to having done a couple of GraalVM runs gathering profile data. Here is an overview of using it with Maven https://www.morling.dev/blog/building-class-data-sharing-archives-with… And here is the documentation for the JIT/AOT and JIT server preview for J9, https://www.eclipse.org/openj9/docs. By the way, this is also the way that ART evolved after the short lived AOT attempt. Nowadays a modern version of ART will have a tiered execution of fast startup interpreter (when no native code is available), a JIT with PGO data gathering, AOT compilation from only the parts that matter when the device is idle, upload of PGO profiles into the store so that every device of similar class doesn't start from zero when an APK gets installed. If the PGO dataset changes, then it backs into JIT again, and recompiles the native code again on idle.

Neat! > Incidentally searching around for prior art I found this blog post https://www.morling.dev/blog/how-i-built-a-serverless-search-for-my-bl… about building something similar using lucene, but without storing the content and only on a single lambda. See also: Sqlite FTS (no server): https://news.ycombinator.com/item?id=27016630 EdgeSearch (Workers): https://github.com/wilsonzlin/edgesearch EdgeSql (Workers): https://github.com/lspgn/edge-sql InfiniCache (Lambda): https://news.ycombinator.com/item?id=25788893 quickwit (Lambda): https://news.ycombinator.com/item?id=27074481 > I had been working on a bloom filter based index based on the ideas of bitfunnel which was developed by Bob Goodwin, Michael Hopcroft, Dan Luu, Alex Clemmer, Mihaela Curmei, Sameh Elnikety and Yuxiong He and used in Microsoft Bing https://danluu.com/bitfunnel-sigir.pdf. Reminds me of wavelet trees: https://alexbowe.com/succinct-debruijn-graphs/ https://alexbowe.com/wavelet-trees/