Heap Blog

> The worst part? Even if you add it immediately, you'll have to wait 2-4 weeks to get a sufficient amount of data. I think this was the problem a product like Heap [1] was designed to solve: just track all user actions, forever, and then assign pipelines after the fact based on what you want to check up on. Don't work at Heap or anything, just love the team and product. [1]: https://heapanalytics.com/

Agreed. I built Heap (https://heapanalytics.com) to eliminate the clumsiness you mention. Heap's approach to analytics is a little different. Instead of forcing you to manually instrument code, Heap automatically captures all user actions. You can then define events retroactively, without writing a bunch of brittle tracking code. Would love to hear your feedback on this. Does Heap's approach get you closer to what you need?

I know that applying on jobs pages is not popular here on HN but, well, sometimes it works. In my opinion, having some projects on GitHub certainly helps, even if they are just hobby projects with no use in real world. So it doesn't mean that you must spend months working on open source projects on the evenings and weekends, hoping to get noticed. Same for technical blog posts. It definitely provides more signal to the person who is reviewing your application. Also, very few companies are 100% remote, and making remote team work is not trivial, so it's important to ask the right questions when interviewing. Don't forget that's it is a two way street! Personally, I was looking for these two key things: - at least ~30% engineers should be remote; - there should be a strong culture of written async communication. Once I discovered a company which I found interesting ( https://heapanalytics.com ), checked the jobs page, noticed that they are hiring remotely and just applied a few months later. Had several interviews over video chat / slack, no white board coding or CS trivia. In the end it worked out and I got an offer. Totally recommend applying at Heap - we are doing cool and challenging technical things (querying hundreds of TBs of data in seconds and making it reliable), the interview process is great and our small distributed remote team is made up of engineers from 4 continents, from North America to Australia!

Congrats to Mixpanel on the raise. I have heard many great things from them and used them at a previous company. They've done a great job of making it a no-brainer for startups to start using more advanced analytics tools (not just Google Analytics) from day 1. I think we're all better off for the ability it gives product developers to improve their product based on actual usage--since it means the overall quality of products is far better than if vanity metrics drove all product decisions. I noticed they mentioned Heap Analytics (https://heapanalytics.com/) as one of their competitors. We've been using Heap for over a year and it seems like the logical and magical next step in analytics. Mixpanel gave you smarter analytics on things you had the foresight to track, but Heap automatically tracks everything from the day you integrate it. That means you can get smart analytics even on things you didn't have the clairvoyance to start tracking 6 months ago, or didn't have the resources to insert tracking code in. For startups, Heap's automatic and retroactive tracking is huge. It means we can iterate on product features and marketing/outreach schemes way more quickly while still getting insight into what's successful and what's not. It's not perfect--a couple times we've added special class names to our HTML elements so Heap can distinguish them, but that's still easier than adding manual tracking code--but it's a huge improvement over the old way. I noticed Heap has a page…

If you want to leave Google, but are not opposed to proprietary infrastructure, you could use something else with event tracking, like HEAP (https://heapanalytics.com/), which is very good. Or if you prefer open source, Matomo seems like a good alternative, with both hosted and self-hosted options (open source, https://matomo.org).

I appreciate your willingness to share your perspective here. The problem is that marketing/PR is a profession dedicated to finding whatever just-barely-technically-true weasel wording is required to make what they're getting paid for sound at least "ok" and preferrably "great!" If you have time (definitely not expected) I would like to hear more specifics regarding this aspect of your statement: > We don't do cross-domain tracking, or aggregate behavioral information across customers vs. https://heapanalytics.com/terms > We may also use data in an aggregated form for our own purposes

Heap Analytics -- https://heapanalytics.com And I run everything through Segment rather than embedding individual trackers directly. Segment either relays data to other services or loads required JS on page load.

If any one here is doing serious DB based business logic with PostgreSQL (which I do recommend), I'd implore them to investigate and start using LATERAL JOIN. http://blog.heapanalytics.com/postgresqls-powerful-new-join-type-later… This is no doubt a game changer in the way you organize (modularise for reuse) your views, functions and stored procedures. In the MS SQL world it's CROSS APPLY.

I'M a huge fan of Heap Analytics (https://heapanalytics.com/). Rather than trying to be precise about what to track and how to instrument the analytics system, you just install their SDK (or JS on your website) and then you have every single thing tracked. Since you collect all the data from day one, as your products / apps grow and mature, you can develop new ratios / rates of change, etc and have them work retroactively. It's magical. :)

Great question; I'm an engineer at Heap. This data model fit our needs at the time, however, you're absolutely right that these user rows do become quite cumbersome as they grow large. > Well... generally this is solved via indices. An event log indexed by user ID would have the same effect. You get all the user info, and if you really need to grab historical or event data, it's there when necessary for parts of the app that actually need it. Indeed! We're investigating a more traditional schema, as the topics of our more recent blog posts might suggest (http://blog.heapanalytics.com/speeding-up-postgresql-queries-with-part…

Heap [1] actually does a good job of what you're asking for. They have these flows that chart the user's path through your site, so you can see where, when and why they dropped off where and when they did. 1. https://heapanalytics.com/

Heap Analytics is in the process of moving over our documentation to ReadMe.io and so far the process has been amazing. The ReadMe team is really responsive and helpful, the documentation is beautiful and the interface goes above and beyond. Just to give you an idea of how polished the service is, if you select a background color that contrasts poorly with the text, the interface will let you know. It also automatically generates a "white" logo for darker pages. This has almost nothing to do with writing documentation, but has a lot to do with making documents readable for users, which is easy to overlook if you're not a designer. See our docs: http://docs.heapanalytics.com/ Right now they are a transcription from http://heapanalytics.com/docs but we're planning more robust docs and ReadMe will make that process a lot smoother.

I like Heap's (https://heapanalytics.com) approach - capture everything, query later. Depending on your needs (Heap is front-end UI oriented), might not be the right fit.

Why not a more automatic approach like http://heapanalytics.com? Their js snippet starts automatically collecting all click events, then I group and analyze events on their website. How is this "big data" when I have to make backend code changes to manually add each event using keen's api library? There's no way I can get as many events adding them one-by-one.

Hi. I'm one of the database engineers at Heap. This is a good question. There are several reasons why we use EC2. First of all, I will say I love RDS as a product. We actually do use RDS for a number of our services. We use Postgres on EC2 only for our primary data store. As for reasons why we use EC2: Cost - Our primary data store has >1 Petabyte of raw data stored across dozens of Postgres instances. The amount of data we store is at the point where RDS is too expensive for us. The cost of an instance on RDS is more than twice the cost on EC2. For example, an on-demand r4.8xl on EC2 instance costs $2.13 an hour, while an RDS r4.8xl costs $4.80 an hour. Performance - The only kind of disk available on RDS is EBS. EBS is slow compared to the NVMe the i3s provide. We used to use r3s with EBS and got a major speedup when we switched to i3s. As a side note, the cost of an i3 is also less than the cost of an r3 with an equivalent amount of EBS. Configuration - By using EC2 we can configure our machines in ways we wouldn't be able to if we used RDS. For example, we run ZFS on our EC2 instances which compresses our data by 2x. By compressing our data, we get a major cost saving and a major performance boost at the same time! There isn't an easy way to compress your data if you use RDS. Introspection - There are times where we've needed to debug performance problems with Postgres and EXPLAIN ANALYZE won't suffice. A good example is we used flame graphs to see what Postgres was…

Heap (YC W13) | Software Engineer | San Francisco, CA | Fulltime | https://heapanalytics.com Help us bring data science to the masses. We're a small team building the analytics engine that powers 6,000+ companies and helps them make smarter decisions. Heap automates away the pain of data. Other analytics tools require you to define events upfront and manually instrument code. Instead, Heap automatically captures everything: clicks, submissions, taps, etc. This lets our users analyze data instantly and retroactively, without writing code. We’d particularly like to get to know you if: - you communicate with clarity and precision. We care about this almost as much as your technical ability. - you're passionate, even if it's not about analytics. There should be something you know more about than anyone on our team. - you’ve written 10,000 lines of code that look embarrassing in retrospect. ... and you've written 10,000 lines of good code since then. Our app-layer stack is TypeScript, React, Node.js, Redis, and PostgreSQL. Under the hood, Heap is powered by CoffeeScript, Scala, ZooKeeper, Kafka, and CitusDB. If you are interested, please email at [email protected] or visit https://heapanalytics.com/jobs

Sounds like heap analytics to me https://heapanalytics.com/

You need to make sure you have enough samples in order to know if you rejected the null hypothesis by chance. Stopping your test early, is a form of p-hacking. See: https://heapanalytics.com/blog/data-stories/dont-stop-your-ab-tests-pa…

I am not the OP but from looking at their profile - https://heapanalytics.com/terms you can check there

Writing effective one sentence product descriptions can be boiled down to: Describe in simple terms a major product feature that either or both: a) solves the biggest pain point for your customer b) makes you the most different from your closest competitor Examples: https://heapanalytics.com: Automatically track all website and app user actions - no coding required https://mailchimp.com/: Easiest to use email marketing platform https://www.pipedrive.com/: Drag and drop interface CRM - view all deals by type at a glance https://www.getrevue.co - Effortlessly send a weekly newsletter to engage your audience http://www.artofemails.com/ - Proven prewritten sales email templates

The demo page (https://heapanalytics.com/dashboard/demo) fails in IE8. Is it just me or IE8 is just not considered by web startups anymore ? Sorry to deviate a little but every time I try to look at a "Show HN" in IE8 (work computer), it fails for about 85% of the time. I could understand that some startups heavily depend on latest browsers but what about others ?

Your move from self-hosted WordPress assets to S3 left some weird stray URLs on your blog that you could fix too... http://blog.heapanalytics.com/wp-content/uploads/2013/03/blog_2-26-13_… for example. Now can I have an invite? ;) mattl [at] gnu org

While you do need to create tables for your stats, you are able to ALTER TABLE just as you can for normal Postgres without problem. It may not be as pain-free as NoSQL in that regard, but building tooling for this or automating this is certainly possible. Alternatively if you do want pure blog storage, Postgres's JSONB datatype works as well, although with some performance trade-offs. This can work quite well if you have some structured data that lives alongside other unstructured data. [1][2] Building or putting a REST front-end in front of PostgreSQL should not really be an issue (we built one for our hosted version), and there are already a fair number of PostgreSQL clients/libs. That said, for ease of use we are already thinking of adding an HTTP interface. For Grafana: We were actually working on our own connector to support PostgreSQL, but found out that one is already in the works by the Grafana team (which will work out of the box with Timescale, because each of our nodes look like PostgreSQL to the outside world). This doesn't seem like it'll be an issue for very long. We've mentioned retention policies elsewhere, but you do bring up a good point. Instead of dropping data, support for aggregating data in older chunks to a more coarse-grain resolution is something we are already looking into. [1] https://www.citusdata.com/blog/2016/07/14/choosing-nosql-hstore-json-j… [2] https://blog.heapanalytics.com/when-to-avoid-jsonb-in-a-postgresql-sch…

For 1 TB of data and 1 TB of data growth yoy, you might be able to get away with vanilla Postgres with reasonable sharding/partitioning/data structure. Without too much details, I am working on a project that handles automated reports/metrics and our only problem with Postgres has been our write-heavy work load (554 M transactions a day with ~250 GB a week). This is a still very early goings and a fraction of our "production" target scale, but we haven't had any read issues. Our problem is that our constant writes to tables mean that our checkpoints [http://dba.stackexchange.com/questions/61822/what-happens-in-postgresq… started to take significant periods of time and happen more and more frequently. Postgres also has some write amplification [http://blog.heapanalytics.com/speeding-up-postgresql-queries-with-part… and VACUUMING challenges [https://www.postgresql.org/docs/current/static/routine-vacuuming.html]… But again these issues are specifically due to our write-heavy, timeseries data. For now we mitigate the effects with sharding and partitioning as we transition to Cassandra...but it sounds like you don't have a similarly write-heavy workload. So I think you might be able to get away with just Postgres. https://www.postgresql.org/about/ http://stackoverflow.com/questions/21866113/how-big-is-too-big-for-a-p…