packagecloud Blog

Packagecloud have a great article series which goes into much more detail and with code study. If you really want to learn the network send and receive, these are the articles to read: https://blog.packagecloud.io/monitoring-tuning-linux-networking-stack-… https://blog.packagecloud.io/monitoring-tuning-linux-networking-stack-… https://packagecloud.io/blog/illustrated-guide-monitoring-tuning-linux…

I'm not familiar with the Windows version of ProcMon, but judging by the Linux version, that looks like it's primarily replicating the function of `strace` in a CLI GUI. Worth reading Joe Damato's excellent article on strace: https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-work/

packagecloud's in-depth writeups on the Linux networking stack are a great read: https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-ne… https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-ne… The illustrated guide to receiving data is also solid: https://blog.packagecloud.io/eng/2016/10/11/monitoring-tuning-linux-ne…

This is a great summary, thank you. Also attribution for things like (https://blog.packagecloud.io/eng/2016/10/11/monitoring-tuning-linux-ne… is very much appreciated.

I really like the packagecloud one: https://blog.packagecloud.io/ With posts like these one: https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-ne…

This is a follow-up to: https://blog.packagecloud.io/eng/2017/02/21/set-environment-variable-s…

I wonder how much of the linux kernel Brendan Gregg has read. I bet it is a decent bit. Also, this guy has posted numerous blogs on the very intimate details of the Linux kernel and even given talks on it. I was lucky enough to watch one of his talks last year @ Monitorama and it was my favorite talk of the entire conference for the level of technical depth it went into. Whether or not you think he is good based on your dislike for the top doesn't make it any less of a fantastic technical post. This was the talk I saw, and it was amazing: https://blog.packagecloud.io/eng/2016/06/29/monitorama-2016-all-of-you…

Thanks for this! I recently wrote a deep dive in to both strace [1] and Linux system calls [2] which may be interesting to folks reading this great article. [1]: https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-work/ [2]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-li…

2 processes = 2 GIL There is no avoiding the GIL within a single Python process (even with asyncio IIRC, though I've been using JS lately). Multiprocessing is usually the most efficient way to execute I/O intensive, independent parallel operations. Of course you can also run threads within each process. I do wonder where the 300mb memory is coming from. Surely it can't all be python interpreter? It doesn't look like he's importing 300mb of modules, unless MongoClient really is that big. In that case he could create a separate worker process for persisting data, and only that worker process needs to load the MongoClient module. One explanation for the memory overhead might be conntrack tables within the network namespace of the container. However I would expect that conntrack table to be on the host, where SNAT is performed. As an aside, the default Docker networking configuration is really not well suited to concurrent network requests, whether inbound or outbound. If you can avoid NAT (and therefore a conntrack table), that is preferable. This stack could also benefit from tuning some kernel parameters, both within the containers and on the host. Great blog post with details: https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-ne…

This one is pretty good: https://blog.packagecloud.io/the-definitive-guide-to-linux-system-call…

"Syscalls" is a topic in systems programming. You can Google "syscall faq" for example. https://blog.packagecloud.io/the-definitive-guide-to-linux-system-call… Understanding how syscalls are invoked from user space typically involves knowing what calling conventions are, knowing what an ABI is, etc. https://man7.org/linux/man-pages/man2/syscall.2.html

Usually, the packages themselves are not signed with GPG, only the Release file is (containing the hashes of all .deb files). This is actually the default of both Debian and Ubuntu. I never quite understood the reasons behind it... I'd not expect this vuln to happen, though. More info: https://blog.packagecloud.io/eng/2014/10/28/howto-gpg-sign-verify-deb-…

I'm somewhat surprised that no-one has (yet) linked to this post [1] by Joe Damato of Packagecloud, which digs into attacks against GPG signed APT repos, or the paper which initially published them [2]. The post makes clear in the third paragraph: "The easiest way to prevent the attacks covered below is to always serve your APT repository over TLS; no exceptions." [1]: https://blog.packagecloud.io/eng/2018/02/21/attacks-against-secure-apt… [2]: https://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf

Unfortunately, even the maintainers of dpkg/apt, who rely on GPG extensively, don't manage to plug all holes: https://blog.packagecloud.io/eng/2018/02/21/attacks-against-secure-apt… Personally, I'm of the opinion that APIs should be as misuse-resistant as possible, and GPG is notoriously intransigent about actually acting as a well-designed API. Additionally, The Update Framework (TUF, https://theupdateframework.github.io/) prevents these issues by design.

and https://blog.packagecloud.io/eng/2018/02/21/attacks-against-secure-apt…

To be utterly pedantic, on Intel Linux only "legacy" 32-bit int 0x80 mechanism uses iret to return. 32-bit "fast" syscalls use sysenter/sysexit. 64-bit "fast" syscalls use syscall/sysret. Haven't really looked but I suspect sysexit and sysret are somewhat special cased versions of iret. https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-li…

Read 'em and weep: https://blog.packagecloud.io/eng/2018/02/21/attacks-against-secure-apt… https://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf "GPG signing a Debian package does nothing because package signatures are not verified by default on any major distribution when packages are installed with apt-get install. See your /etc/dpkg/dpkg.cfg file for an explicit comment to this effect." And: "When APT software (such as apt-get, or reprepro) or folks offering APT repositories mention GPG signatures in their documentation they are typically referring to GPG signatures on repository metadata, not on packages themselves. Likewise, when you configure the SignWith option of reprepro (documented here), you are telling reprepro to sign your repository metadata with the specified GPG key; this does not sign any of the packages, though."

This is a thing that generates many useless syscalls despite vDSO: https://blog.packagecloud.io/eng/2017/02/21/set-environment-variable-s…

This is especially relevant if you're running in the cloud. We've recently run into slow performance with gettimeofday on Xen systems, such as EC2 (see: https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slow… This hits particularly hard if you're running instrumented code. Switching the clocksource from xen to tsc seems to help and we're still evaluating the ramifications of doing so (various sources including the above seem to suggest that tsc is unsafe and prone to clock skew, but others suggest that it's ok on more modern processors.)

I wonder if they tried this: https://blog.packagecloud.io/eng/2017/02/21/set-environment-variable-s…

Note that for modern x86 processors there are the fast systemcall instructions syscall/sysret (AMD64) and the older sysenter/sysexit. In contrast to int 80h, those instructions do not bear the full interrupt overhead. I couldn't find this info anywhere on the website. See e.g. [1] for a more comprehensive overview concerning the handling of system calls on Linux. [1] http://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-lin…

Here is what "Date" is used for [1][2] if that helps you decide what impact it will have on your site. FWIW I have removed it before but I also don't care about search engines. As for wasting CPU, the date is already fetched by the server used for logging each request even if logging is disabled. Excluding the header will reduce the header size but unlikely improve CPU load of the server. If the goal is a reduction of server load related to headers, then telling the web server daemon to not generate ETag's will have a more significant effect and also impact search engines, proxies and CDN's An origin server MUST NOT send a Date header field if it does not have a clock capable of providing a reasonable approximation of the current instance in Coordinated Universal Time. An origin server MAY send a Date header field if the response is in the 1xx (Informational) or 5xx (Server Error) class of status codes. An origin server MUST send a Date header field in all other cases. So one could make the argument that if one choose to not have the correct time on their server then one must not send Date. [Edit] I should that that FWIW if one wanted to tinker with time, one could change the system clock and/or recompile the web daemon to call time differently but this is a whole can of worms in and of itself. [3] [1] - https://stackoverflow.com/questions/1610254/whats-the-rationale-behind… [2] -…

>It's so much harder to build your own software packages on Debian. Even patching an existing Debian package can be painful, and having to write your own packages is even worse. I can't say that's been my experience, once I figured out dh_make [0] that made it pretty easy by handling most of the boilerplate. Building an existing package is as simple as doing `apt source` and then `dpkg-buildpackage` to rebuild it (you may have to do `apt build-dep` first too). I agree with the rest of your comments though. Edit: I haven't looked into the details of how it works but I would imagine that YMMV with trying to use makepkg that way. The hard part here isn't getting the packages to build, it's in reconciling the library dependencies across all the tens of thousands of packages that make up both distros. [0]: https://blog.packagecloud.io/eng/2015/07/14/using-dh-make-to-prepare-d…

Except none of these comments take into account the fact that actually no one verifies apt package signatures either, and this fairy tale world where all of this is a solved problem in some other domain is not manifest. https://blog.packagecloud.io/eng/2018/02/21/attacks-against-secure-apt… The reason that package signing never really matters that much is that once you boil the thrat model down to package publish credentials are compromised or package repository infrastructure is compromised, the form of credentials involved is of little consequence to the prior and uninvolved in the latter. The threat is against the client, not intermediates. The original developer here reused credentials. There is nothing in signing that protects from this attitude. This attitude is the one that also reuses credentials for signing keys, if encrypting them at all - I'd bet this user has numerous stale ssh keys and never encrypted any of the secrets. Some of the top eslint contributors have multiple short rsa keys on their GitHub. None of them have modern keys. There are more effective places to invest to better protect users. Auditing infrastructure for example.