ClickHouse Blog

I'd not heard of clickhouse before. It does seem interesting, but I just can't get behind a project that says: > The easiest way to download the latest version is with the following command: > curl https://clickhouse.com/ | sh Like, sure, there is some risk downloading a binary or running an arbitrary installer. But this is just nuts.

I would recommend to look at ClickHouse[0] which is Apache-License and is completely open-source. You can self-host it and it is one of the fastest growing OLAP database. It has good integrations with BI tools and it can be used for wide varieties of use cases. It is widely adopted[1]. I would recommend you to give it a try and validate it for your use-case. I work for ClickHouse and available for any queries or help you need. [0]: https://clickhouse.com/ [1]: https://clickhouse.com/docs/en/about-us/adopters

I recommend checking out clickhouse-obfuscator. It's a more sophisticated tool for dataset obfuscation. Installation (single binary Linux/Mac/FreeBSD): curl https://clickhouse.com/ | sh ./clickhouse obfuscator --help Docs: https://clickhouse.com/docs/en/operations/utilities/clickhouse-obfusca…

Lovely! I only learnt today that clickhouse has a git-import tool from my colleagues at ClickHouse. So if you also want to give it a go: Download clickhouse: curl https://clickhouse.com/ | sh Check out documentation for git-import: ./clickhouse git-import --help Then the tool can be run directly inside the git repository. It will collect data like commits, file changes and changes of every line in every file for further analysis. It works well even on largest repositories like Linux or Chromium. Example of a trivial query: SELECT author AS k, count() AS c FROM line_changes WHERE file_extension IN ('h', 'cpp') GROUP BY k ORDER BY c DESC LIMIT 20 Example of some non-trivial query - a matrix of authors, how much code of one author is removed by another: SELECT k, written_code.c, removed_code.c, round(removed_code.c * 100 / written_code.c) AS remove_ratio FROM ( SELECT author AS k, count() AS c FROM line_changes WHERE sign = 1 AND file_extension IN ('h', 'cpp') AND line_type NOT IN ('Punct', 'Empty') GROUP BY k ) AS written_code INNER JOIN ( SELECT prev_author AS k, count() AS c FROM line_changes WHERE sign = -1 AND file_extension IN ('h', 'cpp') AND line_type NOT IN ('Punct', 'Empty') AND author != prev_author GROUP BY k ) AS removed_code USING (k) WHERE written_code.c > 1000 ORDER BY c DESC LIMIT 500

> I can't find any other open source time series database except Mimir/Cortex which allows this much scale (clustering options in their open source version) The following open source time series databases also can scale horizontally to many nodes: - Thanos - https://github.com/thanos-io/thanos/ - M3 - https://github.com/m3db/m3 - Cluster version of VictoriaMetrics - https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html (I'm CTO at VictoriaMetrics) > Can we use Prometheus/Mimir as general purpose time series database? This depends on what do you mean under "general purpose time series database". Prometheus/Mimir are optimized for storing (timestamp, value) series where timestamp is a unix timestamp in milliseconds and value is a floating-point number. Each series has a name and can have arbitrary set of additional (label=value) labels. Prometheus/Mimir aren't optimized for storing and processing series of other value types such as strings (aka logs) and complex datastructures (aka events and traces). So, if you need storing time series with floating-point values, then Prometheus/Mimir may be a good fit. Otherwise take a look at ClickHouse [1] - it can efficiently store and process time series with values of arbitrary types. [1] https://clickhouse.com/

ClickHouse code style recommends unsigned in every case when you don't need the sign: https://clickhouse.com/docs/development/style

Clickhouse offers the same through https://clickhouse.com/docs/materialized-view/incremental-materialized… I personally LOVE this feature and concur with the gp that they're really elegant solutions to difficult problems. Just to give an example, I ingest otel trace spans individually and in a materialized view calculate the total duration of the whole trace among other things.

Note that the headline is from Langfuse, not ClickHouse. Reading the announcement from ClickHouse[0], the headline is "ClickHouse welcomes Langfuse: The future of open-source LLM observability". I think the Langfuse team is suggesting that they will be continuing to do the same work within ClickHouse, not that the entire ClickHouse organization has a goal of building the best LLM engineering platform. [0] https://clickhouse.com/blog/clickhouse-acquires-langfuse-open-source-l…

Clickhouse chDB is excellent, much less buggy than DuckDB https://clickhouse.com/chdb

Thanks for your detailed reply. It is great to see that you have been experimenting with this approach. We published a public demo of the Agentic Data Stack, I'd love to hear your feedback https://clickhouse.com/blog/agenthouse-demo-clickhouse-llm-mcp Keep in mind that it's not fully "fair", since these public dataset are often documented in the internet so already present in pre-training of the models underneath (Claude Sonnet 4.5 in this case)

There is a largish category of tools now where, unlike in OLTP systems, a big focus is scanning data but quickly (O(n) but with a good constant): Redshift, Trino/Athena, ClickHouse, DuckDB among others. Bloom filter indexing seems like a great fit if you ever need to do substring searches in a context like that, and for log searching in general. I haven't dug into what all packages have it, but it looks like at least ClickHouse does: https://clickhouse.com/docs/optimize/skipping-indexes#bloom-filter-typ…

Why add RedPanda/Kafka over using async insert? https://clickhouse.com/docs/optimize/asynchronous-inserts It’s recommended in the docs over the Buffer table, and is pretty much invisible to the end user. At ClickHouse Inc itself, this scaled far beyond millions of rows per second: https://clickhouse.com/blog/building-a-logging-platform-with-clickhous…

Sai from ClickHouse here. Very compelling story! Really love your emphasis on using the right tool for the right job - power of row vs column stores. We recently added a MySQL/MariaDB CDC connector in ClickPipes on ClickHouse Cloud. This would have simplified your migration from MariaDB. https://clickhouse.com/docs/integrations/clickpipes/mysql https://clickhouse.com/docs/integrations/clickpipes/mysql/source/rds_m…

For getting started, we have a really easy to use helm chart that I'd recommend checking out first: https://clickhouse.com/docs/use-cases/observability/clickstack/deploym…

Afaik this is always best effort, e.g.: https://clickhouse.com/docs/operations/settings/settings#insert_dedupl… "For the replicated tables by default the only 100 of the most recent blocks for each partition are deduplicated" This doesn't work under failure conditions either (again afaik), e.g. if the clickhouse server fails.

Clickhouse also has managed service (https://clickhouse.com/)

It's linked to at the end, but I'm surprised that this article didn't feature/mention ClickHouse's own write-up of this undertaking. Part 1 here: https://clickhouse.com/blog/building-a-data-warehouse-with-clickhouse, part 2 here: https://clickhouse.com/blog/building-a-data-warehouse-with-clickhouse-… Aspects of the post seem to borrow quite heavily from the original write-ups, which are worth a read.

Here’s a guide on exploring 120 years of global weather measurements with Clickhouse. I enjoyed following this guide over a weekend. Also it gives you a sense of the dataset, even if you’re not interested in the clickhouse part. https://clickhouse.com/docs/en/getting-started/example-datasets/noaa

The count() invocation is a small but excellent feature in ClickHouse. There are many other great dialect extensions. -If combinators are another great example. [0] SQL would be a better language if there had been more innovation focused on code usability over its long history. [0] https://clickhouse.com/docs/en/sql-reference/aggregate-functions/combi…

There are various knobs in ClickHouse that allow you to trade memory usage for performance. ( https://clickhouse.com/docs/en/operations/settings/query-complexity#se… e.g.) But yes, I've seen similar issues, running out of memory during query processing, it's a price you pay for higher performance. You need to know what's happening under the hood and do more work to make sure your queries will work well. I think postgres can be a thousand or more times slower, and doesn't have the horizontal scalability, so if you need to do complex queries/aggregations over billions of records then "return result" doesn't cut it. If postgres addresses your needs then great- you don't need to use ClickHouse...

we've been pretty happy with just a Clickhouse DB and sending metrics directly from api servers to Clickhouse HTTP https://clickhouse.com/docs/en/interfaces/http . Hook up Grafana and you have a nice raw SQL (our team loves SQL) Grafana dashboard.

I'll second this. Clickhouse is amazing. I was actually using it today to query some CSV files. I had to refresh my memory on the syntax so if anyone is interested: clickhouse local -q "SELECT foo, sum(bar) FROM file('foobar.csv', CSV) GROUP BY foo FORMAT Pretty" Way easier than opening in Excel and creating a pivot table which was my previous workflow. Here's a list of the different input and output formats that it supports. https://clickhouse.com/docs/en/interfaces/formats

Yep, that trusty simple algorithm always works. I have some grafana dashboards that plot a lot of 3D data (x,y,z lines coloured by some sensor readings) using a plotly plugin. I can't pre-aggregate too much in the database, or I'll loose too much detail. So I ended up adding a little ad-hoc js implementation to reduce my number of points by a factor ten on the fly for plotting, works great. I was just about to write down that it would be too difficult to de RDP simplification in the (ClickHouse) database, but then I recalled PostGIS has it built in and low and behold, there is also something in ClickHouse for this [0]. Back to the drawing board. [0] https://clickhouse.com/codebrowser/ClickHouse/contrib/boost/boost/geom…

Disclaimer: I work at ClickHouse Thank you! Glad it helped you. ClickHouse is in particular well-optimized to take on real-time analytical workloads from BigQuery - both functionally and from a cost perspective. For anyone wanting to try it yourself, there guide to follow when syncing or migrating workloads from BigQuery to ClickHouse, with some benchmarks and back-of-the-envelope cost comparisons: https://clickhouse.com/blog/clickhouse-bigquery-migrating-data-for-rea…

> Download clickhouse: curl https://clickhouse.com/ | sh Does this check the useragent to change the response? Clicking that link shows their home page.

Hi. Sorry if my query offended you. I basically executed literally what Clickhouse recommends at their guides for deduplication https://clickhouse.com/docs/en/guides/developer/deduplication. Of course you can also materialize with aggregations or just use a group by, or even force optimize of the table. But my point is that you don't really get exactly once guarantees. Whoever is querying that table needs to be aware than a `SELECT * FROM tb` might contain duplicates and needs to adapt their queries accordingly.

ClickHouse | REMOTE| FULL-TIME| Various locations globally ClickHouse is the fastest and most resource efficient open-source database for real-time apps and analytics. Our engineering teams invest heavily in improving the most loved ClickHouse open source project as well as building ClickHouse Cloud a fully cloud Native SAAS offering with separation of compute and storage. We are a globally diverse and distributed team, united behind a common goal of creating industry-leading, real-time analytics. Here, you will have an opportunity to solve some of the most cutting edge technical challenges and have direct ownership of your work and vision. If you are a contributor by nature, a thinker as well as a doer - we’ll definitely click! Lot of engineering positions open across different teams- please apply on our website: https://clickhouse.com/company/careers

Great product! Thanks for sharing it! Question: I thought Clickhouse already has native support for flattening JSON [1], although it was released recently (version 22.3.1). Did you start working on it [2] before that? Or is it a different take? I'm curious about the pros and cons of each one. [1] https://clickhouse.com/docs/en/integrations/data-formats/json#semi-str… [2] https://scratchdb.com/blog/flatten-json/

I was contemplating on various ways to achieve this at my job for last few days, here's something worth considering. [0]https://clickhouse.com/blog/analyzing-aws-fow-logs-using-clickhouse

Don't insert the logs/events/analytics into your Application DB. Usually, you send those to specialist datastores (OLAP etc) that process such high volume of data. This way, you keep the load and storage on your App db low AND if the analytics data is not working, it doesn't impact your Core Application. You can use something like clickhouse [0] for example or use 3rd party SAAS solutions like posthog [1] etc that are built on top of clickhouse [0] https://clickhouse.com [1] https://posthog.com