Quickwit Blog

I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have. Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup. A couple of months ago I took a brief look at that when setting up logging for my own homelab (https://pv.wtf/posts/logging-and-the-homelab). Mostly looking at the memory usage to fit it on my synology. Quickwit[2] and Log-Store[3] both come with built in web interfaces that reduce the need for grafana, but neither of them do metrics. - [1] https://vector.dev - [2] https://quickwit.io/ - [3] https://log-store.com/

Can I get your use case? What is the data you have to index? If it's logs, indeed, Meilisearch will not be the best fit. I would recommend https://quickwit.io. If it's structured or unstructured data, even if 10TB seems huge for our use cases, I would love to put you in contact with my engineering team. We would find a way to manage it.

Quickwit | Senior Rust engineer | Remote (EU, ASIA) | https://quickwit.io Quickwit is building the next-generation open-source search engine for petabyte-scale data. We are also behind tantivy, the Rust search engine library that has rallied a great community of users since its inception. We are looking for an engineering generalist with extensive experience writing Rust code and leading technical architecture. Working with large data sets, dealing with complex distributed systems, and developing backend services should be second nature. More info about the position: https://quickwit.io/jobs/distributed-software-engineer

Quickwit (https://quickwit.io/) | Paris, France | Onsite and remote (based in Europe) | Full-time The company is fully remote but we also have a small office in Paris. We prefer candidates based in Europe but can make exceptions for the right profiles. * Senior Software Engineer 80-110k€ + 0.25-1% equity based on experience. We’re looking for a senior software engineer to contribute to Quickwit(https://github.com/quickwit-oss/quickwit), our open-source search and analytics engine. We have an ambitious roadmap for the next 18 months (performance optimization, distributed storage, support for SQL, query optimizer, revamp of our execution engine, etc.), and this is a great opportunity to shape the future of Quickwit while tackling fun and challenging problems in the field of distributed databases. Full job description: https://quickwit.io/jobs/distributed-software-engineer * Developer Advocate 80-110k€ + 0.25-1% equity based on experience. We’re looking for a Developer Advocate to take the reins of content creation and community management to boost our growth. You will get the opportunity to define our advocacy strategy and execute it. Your work will be key to the success of Quickwit, and you will get the full support of all Quickwiots. Full job description: https://quickwit.io/jobs/developer-advocate Contact us at: hello at quickwit dot io

Meilisearch is great, used it for a quick demo However if you need a full-text search similar to Apache Lucene, my go-to options are based on Tantivy Tantivy https://github.com/quickwit-oss/tantivy Asian language, BM25 scoring, Natural query language, JSON fields indexing support are all must-have features for me Quickwit - https://github.com/quickwit-oss/quickwit - https://quickwit.io/docs/get-started/quickstart ParadeDB - https://github.com/paradedb/paradedb I'm still looking for a systematic approach to make a hybrid search (combined full-text with embedding vectors). Any thoughts on up-to-date hybrid search experience are greatly appreciated

It's a bit sad that many modern databases were recently acquired. They had the potential to bring a lot of innovations. 1. https://www.warpstream.com/ 2. https://www.orioledb.com/ 3. https://quickwit.io/

Can you clarify: Does the full-text search for logs linearly search all logs like Loki does, or can it speed it up with an index? The docs at https://www.hyperdx.io/docs/search don't seem to talk about this key design decision. I have a couple 100 GB to few TB logs (all from `journald` or JSON lines), just want to store them forever, and find results fast when searching for arbitrary substrings. Loki does not use an index, so it's pretty slow at finding results in TB-sized logs (does not return results within a few seconds, so it's not interactive). https://quickwit.io is one thing I'm looking at integrating, that can solve much of the index-based log search. (Note I'm not super familar with the capabilities of ClickHouse itself regarding indexed full-text search.)

Sounds like a source-unavailable version of Quickwit? https://quickwit.io/

Oh! Damn it I forgot about manticore -- I had seen it before but forgot to include it. Eventually all of these projects will be highlighted on Awesome F/OSS (https://awsmfoss.com), but for now I'm just going to dump my bookmarks here for other people, since I'm leaving awesome projects out: Search Engines AWS…

Quickwit is another project aiming at this sort of stateless search: https://quickwit.io/ . I've been keeping an eye on it for a possible project that would involve wanting to offer search over a lot of data for an application that wouldn't have very many users, such that keeping a big ES box around all the time would be needlessly expensive; I had been looking for a search situation where my only always-on costs were for storage, and I could pay for compute as needed (and ideally also separate paying for indexing compute, which doesn't happen often in this application, from search compute). Interesting that ES might also end up with this kind of an offering.

And for those who cannot wait, there is quickwit :-) https://quickwit.io

Hi. I was a big Elasticsearch user during several years, I wasn't convinced by Grafana Loki which is far less expensive because the data are stored in object storage, but have poor read performances because it's not a real search engine. Then I discovered Quickwit which is a combination of the advantages of both world. With Quickwit you can ingest the logs the way you're already used to : through OTLP/grpc, with a log collector like fluentbit or Vector which can ingest the stdout of your pods and forward to Quickwit using the http API, etc. And you can then use Grafana with pretty much the same features available for the Elasticsearch datasource. https://quickwit.io/docs/log-management/send-logs/using-fluentbit The read performance of Quickwit are incredible because of their amazing indexing engine (which is kind of like Lucene rewritten in Rust) and the storage is very cheap and without limitation other than your cloud provider capabilities. https://quickwit.io/blog/quickwit-binance-story

How is it compared to https://quickwit.io ?

Quickwit was bought by Datadog, so I feel there's some risk quickwit-oss becomes unmaintained if Datadog's corporate priority shifts in the future, or OSS maintenance stops providing return on investment. Based on the Quickwit blog post, they are relicensing to Apache2 and releasing some enterprise features, so it seems very possible the original maintainers will move to other things, and it's unclear if enough community would coalesce to keep the project moving forward. https://quickwit.io/blog/quickwit-joins-datadog#the-journey-continues

The architecture of Husky seems fairly similar to quickwit. https://www.datadoghq.com/blog/engineering/introducing-husky/ https://quickwit.io/docs/main-branch/overview/architecture

I would say here: Features https://quickwit.io/docs/overview/introduction#key-features

QuickWit also sends telemetry by default: https://quickwit.io/docs/telemetry

According to the documentation[1] Kafka is just one of the supported inputs for ingestion, so it should be possible to run Quickwit without it if you're not intending to write logs into Kafka. Jaeger also seems like another optional dependency. Same also probably for Zookeeper? 1: https://quickwit.io/docs/ingest-data/kafka

As the Docusaurus site maintainer, I believe MkDocs is great (and probably simpler) but our plugin system is more powerful, MDX/React makes it easier to make the docs interactive, and we have more flexibility for theming. See the carousel of site screenshots here: https://docusaurus.io/blog/2022/08/01/announcing-docusaurus-2.0#themin… I'm curious to see highly customized MkDocs sites that truly respect your branding. What I mean by respecting your branding: - Courier: https://www.courier.com/docs/ - Figma: https://www.figma.com/plugin-docs/ - Iota: https://wiki.iota.org/ - Hasura: https://hasura.io/docs/latest/index/ - Ionic: https://ionicframework.com/docs - Quickwit: https://quickwit.io/docs/ All those sites are customized, they do not look like a stock template. You don't need to be Stripe anymore (with a full engineering team) to get a beautiful and interactive docs website

Just browsing the Quickwit documentation it seems like the general architecture here is to write JSON logs but stores them compressed. Is this just something like gzip compression? 20% compressed size does seem to align to ballpark estimates of JSON GZIP compression. This is what Quickwit (and this page) calls a "document": a single JSON record (just FYI). Additionally you need to store indices because this is what you actually search. Indices have a storage cost when you write them too. When I see a system like this my thoughts go to questions like: - What happens when you alter an index configuration? Or add or remove an index? - How quickly do indexes update when this happens? - What about cold storage? Data retention is another issue. Indexes have config for retention [1]. It's not immediately clear to me how document retention works, possibly from S3 expiration? So, network transfer from S3 is relatively expensive ($0.05/GB standard pricing [2] to the Internet, less to AWS regions). This will be a big factor in cost. I'm really curious to know how much all of this actually costs per PB per month. IME you almost never need to log and store this much data and there's almost no reason to ever store this much. Most logs are useless and you also have to question what the purpose is of any given log. Even if you're logging errors, you're likely to get the exact same value out of 1% sampling of logs than you are with logging everything. You might even get more value…