Brendan Gregg

brendangregg.com · Personal blog · on HN since 2010

Brendan Gregg, a performance engineer known for his work at Netflix, writes about Linux performance analysis, eBPF tracing, and flame graphs.

Visit blog ↗ RSS HN submissions ↗

Systems & Performance Open Source & Linux

What HN saysCommenters recommend it as a source of strong monitoring and performance content that is regularly posted on HN, including his Linux crisis tools post.

450HN people shared it

281linked it in comments

295stories submitted

18Kstory points

3 months agolast shared

2010people per year2026

Most-shared posts

1
Linux Performance
35 people · 765 points · first shared 2014 · 64 comments on HN
2
Flame Graphs (2014)
30 people · 7 points · first shared 2015 · discussion on HN
3
Homepage
21 people · first shared 2014
4
Linux Perf Examples
20 people · 145 points · first shared 2014 · 14 comments on HN
5
BPF Performance Tools
17 people · 2 points · first shared 2019 · discussion on HN
6
Linux Load Averages: Solving the Mystery
15 people · 661 points · first shared 2017 · 86 comments on HN
7
Linux EBPF Tracing Tools
15 people · 78 points · first shared 2017 · 3 comments on HN
8
CPU Flame Graphs
15 people · 17 points · first shared 2014 · discussion on HN
9
CPU Utilization is Wrong
10 people · 672 points · first shared 2017 · 91 comments on HN
10
AWS EC2 Virtualization 2017: Including Nitro
10 people · 155 points · first shared 2017 · 52 comments on HN
11
The USE Method (2014)
10 people · 73 points · first shared 2014 · discussion on HN
12
An Unbelievable Demo
9 people · 2.6K points · first shared 2021 · 463 comments on HN
13
DTrace Toolkit
9 people · 60 points · first shared 2010 · 11 comments on HN
14
Strace Wow Much Syscall
9 people · 4 points · first shared 2014 · discussion on HN
15
Systems Performance: Enterprise and the Cloud, 2nd Edition (2020)
9 people · 2 points · first shared 2020 · discussion on HN

What HN people wrote when they linked it

jdenning · 2024-12-21

1) in addition to the excellent recommendations already mentioned: Brendan Gregg has a lot of good stuff about monitoring and performance analysis https://brendangregg.com/ https://github.com/brendangregg Also Jess Frazelle (lots of good stuff, esp around containerization): https://blog.jessfraz.com/ https://github.com/jessfraz

brendangregg · 2022-04-16

I touch it now and then; search for "BSD" on https://www.brendangregg.com. Netflix has hired more great BSD folk, so I haven't needed to do a lot on it for a while. I do enjoy it though, and it's usually much easier to read and understand the kernel code. Most recently I've been talking to folk about eBPF on BSD. Maybe I should do a blog post about it (force me to go talk to everyone again and get their updates).

GrayShade · 2019-12-03

Brendan has a lot of great content that gets posted here regularly: http://www.brendangregg.com/ (I'm still trying find the time to get through it though). There's also https://github.com/iovisor/bcc#tools as an easy way to get started using BPF.

cthalupa · 2014-12-20

I love atop - and install it everywhere that I can. The in depth metrics + historical view is excellent. That being said, I wouldn't call it a one stop tool for performance analysis. Particularly due to the word 'analysis' - it tells you what resource was used, and what was using it. It doesn't tell you anything about /why/ the process was using it. It doesn't help with hunting down regressions. It doesn't give you insight in to what these processes are actually doing. For that, you need to utilize strace/ltrace/ftrace/dtrace/ktap/systemtap/perf/sysdig/whatever (If interested in the actual analysis/engineering side of performance, I recommend following Bredan Gregg's blog and buying his systems performance book - http://brendangregg.com/ )

yankcrime · 2025-07-29

Agreed, highly recommended reading. A slightly more up-to-date post of his which recommends tools in such situations is: https://www.brendangregg.com/blog/2024-03-24/linux-crisis-tools.html

29athrowaway · 2024-08-18

Follow Brendan Gregg's USE method https://www.brendangregg.com/usemethod.html

tanelpoder · 2024-07-03

Brendan Gregg's site (and book) is probably the best starting point (he was involved in DTrace work & rollout 20 years ago when at Sun) and was/is instrumental in pushing eBPF in Linux even further than DTrace ever went: https://brendangregg.com/ebpf.html

gmuslera · 2024-03-25

Provisioning is something that may be out of the picture you described, Terraform, Ansible, templates, whatever, more than creating that server try to recreate it after destroying it (and backups while at that). Monitoring using your own tools instead of (or along with) your provider tools will give you a view of not just when it is working wrong, but also when is working right. Security is another thing that goes both for the cloud and your own server, but for long running instances firewalls and vulnerabilities in what is exposed gets more urgent. Regarding performance, if you want to dig in that topic, don't wait for a crisis for using those tools, try to understand how the system run, even if not under high load. Gregg have more tools (https://www.brendangregg.com/linuxperf.html) and a few great books.

moggi · 2024-03-03

If you want to learn how to understand the performance of the whole system I can recommend Brendan Gregg's Systems Performance: Enterprise and the Cloud (https://www.brendangregg.com/blog/2020-07-15/systems-performance-2nd-e… It is a good book that teaches a lot of basics and techniques and gives a good understanding of the impact different system components can have on performance.

boberoni · 2024-03-03

> This is an area that is new to me, but a big part of my new job. Can you tell me more about what your new job is, without releasing anything sensitive? If you are running applications on Linux containers in the cloud, then I would recommend Brendan Gregg's blog and books (https://www.brendangregg.com/overview.html). He does a lot of knowledge sharing from his experiences at Netflix.

__turbobrew__ · 2024-02-17

I recommend consuming everything Brendan Gregg produces. His work is mostly around kernel performance profiling and tracing, but you can learn a lot from just doing that. Also, the vast majority of time when you need to dive into kernel internals it is for performance reasons. https://www.brendangregg.com/linuxperf.html If you prefer physical books “Systems Performance” by Brendan is good as well. As the sister comment says, Linux Programming Interface is really good for learning linux system programming topics but it is mostly focused on user space. For more userspace stuff I like Chris Wellons‘ blog at nullprogram.com

roland-s · 2024-01-31

Linux now has eBPF, which is essentially a VM running inside the Linux kernel. You can run your own programs on this little VM and extend the kernel to a staggering degree. Some clever folks have used this to build open source tracing tools that rival and even surpass Dtrace in some ways. Brendan Gregg, a Dtrace wizard and old colleague of some of the Oxide team back at Sun and Joyent, has some great resources on the subject https://www.brendangregg.com/ebpf.html https://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.htm…

teleforce · 2023-09-07

Great overview of the Linux network queues as provided in the Figure, should paste it on the wall somewhere. Brendan's System Performance books provide nice coverage on Linux network performance and more [1]. It's already in the second edition, both are excellent books but the 2nd edition focuses mainly on Linux whereas the 1st edition also include Solaris. There's also a more recent book on BPF Performance Tools by him [2]. [1] Systems Performance: Enterprise and the Cloud, 2nd Edition (2020) https://www.brendangregg.com/systems-performance-2nd-edition-book.html [2] BPF Performance Tools: https://www.brendangregg.com/bpf-performance-tools-book.html

unmole · 2023-07-10

SVGs are amazing for interactive visualisation too. Like Flamegraphs: https://www.brendangregg.com/flamegraphs.html

funcDropShadow · 2023-02-09

That is one of the reasons why Brendan Gregg's USE [1] methodology is so great. USE stands for utilisation, saturation, errors. For every component, resource, or subsystem you should have at least a metric for each of these. Utilisation tells you: How much is it used? Saturation tells you: How near is it to the capacity limit or how much does it slow down because of load, and errors tell you e.g. when k8s pods restart all the time. [1]: https://www.brendangregg.com/usemethod.html

shoo · 2023-01-25

Learn how to use a decent profiler. if you're running linux, that's probably perf: https://man7.org/linux/man-pages/man1/perf.1.html https://www.brendangregg.com/perf.html Here's a fun article from the cloudflare blog that gives an example of using of perf to diagnose performance of a small utility: https://blog.cloudflare.com/when-bloom-filters-dont-bloom/ Matt Godbolt's compiler explorer is also worth checking out: https://godbolt.org/

rwmj · 2023-01-18

The performance hit (~2%) is observable at runtime, assuming you're doing things which are CPU bound. For cpython programs the especially large hit (~10%) is well understood though not fixed: https://pagure.io/fesco/issue/2817#comment-826636 The flip side to this is that when you have good profiling it changes how you write and debug software. You can actually look at the whole system (eg with Flamegraphs https://www.brendangregg.com/flamegraphs.html) and find out what's taking all the time, and hopefully direct your fixes to the right place. This potentially gives you great performance benefits. Apparently Meta recompile all software with frame pointers and use perf to do this kind of analysis.

charcoalhobo · 2022-10-15

Love deep dive troubleshooting like this. I haven't heard of systemtap before; looks nice. When I had to troubleshoot a kernel bug [1] I used perf [2] probes which are also really nice for this kind of debugging. [1] https://www.spinics.net/lists/xdp-newbies/msg01231.html [2] https://www.brendangregg.com/perf.html

yakubin · 2022-08-19

There are two excellent books by Brendan Gregg which can teach you about eBPF: 1: Systems Performance: Enterprise and the Cloud, 2nd Edition <https://brendangregg.com/systems-performance-2nd-edition-book.html> 2: BPF Performance Tools <https://brendangregg.com/bpf-performance-tools-book.html> Contrary to the title, I would actually recommend the first one more and only if you feel starved for more to reach for the second one. You can learn a lot about performance tools and methodologies on Linux from the first one, not just eBPF. I've never had to recompile a Linux distro to start using eBPF btw. Most distros already use a kernel compiled with options enabling eBPF. That may not be true on some "embedded Linux" devices though.

buescher · 2022-04-23

Someone else posted a great link explaining why you probably don't want tclsh or wish as your login shell. Today zsh or fish is the better bash, really. Here's another dead end unexplored though - dtksh! https://www.brendangregg.com/dtkshdemos.html

TheDong · 2022-01-21

Yes. It's the default output for "perf report". I recommend reading this: https://www.brendangregg.com/perf.html However, the short 2-line way to get that output is the following: perf record -F99 -g --pid $(pidof consul) # Wait a few seconds and hit ctrl-c perf report You'll get similar output to what they show if you have consul running with a similar load :)

bostonsre · 2021-10-03

https://www.brendangregg.com/blog/2020-07-15/systems-performance-2nd-e… I highly recommend it to anyone who gets close to systems.

scary-size · 2021-09-20

Here are mine: - https://franz.hamburg - https://www.project-daily.com Some blogs I follow via RSS (using the awesome NetNewsWire app): - https://brendangregg.com - https://brandur.org/ - https://ferd.ca/

puptelpete · 2021-07-03

For me, BPF is the most exciting development in the kernel world for ages. Brendan Gregg, the author of the posted blog entry, works with BPF on observability at Netflix and delivered a keynote at UbuntuMasters 2019. The video is on his blog is a great intro. [0] I've been watching BPF for a couple of years now, and it seems slow on the uptake, but I hope Gregg is right that people will eventually start writing new drivers, firewalls, observability and security tools, loaded from userspace, but all running safely in a kernel vm, maybe even written in new async-first programming languages. [0] https://brendangregg.com/blog/2019-12-02/bpf-a-new-type-of-software.ht…

Bayart · 2021-06-08

I've got his Systems Performance book, and it's really fantastic. This guy is incredible. I really need to pick his BPF one. He's got a good blog too [1] ! [1] http://www.brendangregg.com/blog/

jupp0r · 2021-06-03

Using CPU utilization as a performance metric can be extremely misleading. My favorite article on the subject is from Brendan Gregg: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.h… A much better way to test the influence of the new compiler would be to test the actual throughput at which saturation is achieved (which is what the benchmark in the C++ grpc library measure to assess their performance).

WatchDog · 2020-12-21

See Brendan Gregg's excellent work in this space http://www.brendangregg.com/blog/2014-06-12/java-flame-graphs.html

lathiat · 2020-11-20

You should also check out bpftrace which is a specific DSL to write both the kernel and userspace part in one language - rather than the mixed python/C approach people mostly took before that. And you can output things potentially as text or json for parsing. https://github.com/iovisor/bpftrace I would also strongly recommend Brendan Greggs book: http://www.brendangregg.com/bpf-performance-tools-book.html

tirumaraiselvan · 2020-11-11

For those interested in similar chart but for linux processes: http://www.brendangregg.com/linuxperf.html Amazing stuff.

schaefer · 2020-11-10

The video references and recreates a debug session which is originally from a great blog post by Brendan Gregg: “gdb Debugging Full Example” [1]. This original blog post uses only free tools to reach the same conclusions as the Pernosco session. So it’s interesting review and compare them both. [1] http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html

`/`	Search
`j` `k`	Next / previous blog
`Enter`	Open selected blog
`v`	Visit the blog site
`f`	Follow / unfollow
`l`	Switch grid / list view
`d`	Switch dark mode
`g` `h`	Home
`g` `p`	Top posts
`g` `t`	Topics
`g` `f`	Following
`g` `l`	Liked
`g` `y`	History
`g` `q`	FAQ
`Esc`	Back / close