Sysdig Blog

Sysdig [1] great monitoring platform for containers on K8S https://sysdig.com/

Another great tool to look at is Sysdig [0]. The technical founder has an amazing background in deep low level Linux stuff, and security. [0]: https://sysdig.com/

Sysdig | Dev, PM, Design, Technical Marketing, Technical Customer Success | Bay Area, London, & Remote | Full time | https://www.sysdig.com Sysdig is the container intelligence company. We build monitoring, security, and troubleshooting tools for containerized and orchestrated environments. It's all based on our open source linux visibility project (www.sysdig.org). As we're at the forefront of the next big platform shift, We're growing rapidly across the board. We operate a SaaS platform and sell software to large enterprises. * Devs who understand analytics & underlying open source technology like cassandra and elastic search * Front end engineers and visualization specialists familiar with javascript, ember, and D3 * Infrastructure engineers who understand how to effectively package up software for easy consumption and management * Product Managers with monitoring, security, or analytics experience * Product designers who love data, analytics, and devops * Technical customer success managers who can learn and help customers learn "how-to". Act as their advocates within sysdig. * A technical marketing manager who understands the competitive landscape, partnerships, and creating great content to enable the team. Sysdig is a good place to work. It's made up of a lot of smart, honest people who are passionate about what they do. We've got full medical, dental, PTO as-you-need-it, and other great benefits like a monthly learning stipend and a monthly house cleaning…

Here is the source report cited in the article: https://sysdig.com/2023-cloud-native-security-and-usage-report/ It is provided by: "Sysdig Secure: Cloud, container, and Kubernetes security that closes the loop from source to run." It's the best interest of security companies to sell scareware.

"Sysdig vs DTrace vs Strace: A technical discussion." https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technical-discuss… There are sysdig chisels that reference gdb. https://github.com/openai/retro : > Gym Retro lets you turn classic video games into Gym environments for reinforcement learning and comes with integrations for ~1000 games. It uses various emulators that support the Libretro API, making it fairly easy to add new emulators.

Hmm, that seems like a weird miss from my side. i.e. We covered this across several articles like this one about tags: https://sysdig.com/blog/toctou-tag-mutability/ This other one about file integrity monitoring (Disclaimer: A rather commercial one) https://sysdig.com/blog/file-integrity-monitoring/ And I recall others more explicit on the read-only part, but I’m away from my laptop now. Edit: Found it (point 1.3 in https://sysdig.com/blog/dockerfile-best-practices/ ) Thanks for pointing it out. Definitely it should be more explicit.

Basically what you want is a Docker image static analysis tool in your CICD that rejects your pull request if it seems to contain a secret. I'm not really sure what would be the best tool for this, but googling got me to https://sysdig.com/blog/20-docker-security-tools/ which has enough scanning tools to research. I don't think any will be foolproof though. Maybe you can also do what this guy did, use conftest with some OPA rules to detect what seems like a secret: https://cloudberry.engineering/article/dockerfile-security-best-practi… If your secrets all follow RFC 8959 ( https://tools.ietf.org/html/rfc8959 ) then you could make it fool proof by just searching for secret-token: with the tool from the previous blog post. However I've yet to find a place that uses this RFC for all secrets.

I agree. One of my all time favorites was https://sysdig.com/blog/container-isolation-gone-wrong/, it kept me hooked til the last line, like a thriller ^_^

Sysdig/Senior Software Engineer,Backend/San Francisco,CA/ Competitive Benefits + Salary/ https://sysdig.com/ Role: Senior Software Engineer Backend for Sysdig Secure https://grnh.se/bo0bf1r41 Product:Container Intelligence Platform.The first unified approach to container security, monitoring, and forensics. Team: Container obsessed and leading the container revolution! Join the Sysdig family; we’re rapidly growing, with over 250+ customers and have doubled in size in the last year. We are backed by top tier venture firms like Accel Partners and Bain. This is the chance to join a thriving startup who is changing and disrupting the container intelligence space. If you are looking for your next move please check out our careers page at https://sysdig.com/jobs/ or reach out to our Talent team at [email protected].

Sysdig/Senior Software Engineer,Backend/San Francisco,CA/ Competitive Benefits + Salary/ https://sysdig.com/ Role: Senior Software Engineer Backend for Sysdig Secure https://grnh.se/bo0bf1r41 Product:Container Intelligence Platform.The first unified approach to container security, monitoring, and forensics. Team: Container obsessed and leading the container revolution! Join the Sysdig family; we’re rapidly growing, with over 250+ customers and have doubled in size in the last year. We are backed by top tier venture firms like Accel Partners and Bain. This is the chance to join a thriving startup who is changing and disrupting the container intelligence space. If you are looking for your next move please check out our careers page at https://sysdig.com/jobs/ or reach out to our Talent team at [email protected].

Great to see sysdig and falco used in this way! For another earlier discussion of monitoring the behavior of intruders using sysdig, check out https://sysdig.com/blog/fishing-for-hackers/.

> Why are the storage and networking implementations "out of tree" (CNI / CSI)? Given the above question, why is there explicit support for Cloud providers? eg: LoadBalancer supports AWS/GCP/Azure/.. Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. The upcoming 1.31 release will drop a lot of existing, already deprecated support for AWS & others from Kubernetes proper. https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-… There's some plan to make this non-dosruptive to users but I haven followed it closely (I don't use these providers anyhow). > Why are we generating a structured language (YAML), with a computer, by manually adding spaces to make the syntax valid? There should be no intermediate text-template representation like this one. Helm is indeed a wild world. It's also worth noting that Kubernetes is also pushing towards neutrality here; Helm has never been an official tool, but Kustomzie is builtin to kubectl & is being removed. https://github.com/orgs/kubernetes/projects/183/views/1?filterQuery=+s… There's a variety of smart awesome options out there. First place I worked at that went to kube used jsonnet (which alas went unmaintained). Folks love CUE and Dhall and others. But to my knowledge there's no massive bases of packaged software like exists for Helm. Two examples,…

No, I'm sorry, I seriously don't understand how that applies in this case. How is "trusting the content you get from `apt-get`/`dpkg`/`pacman`" safer than curling-to-bash? In both cases, you are downloading and installing content from a trusted source (assuming you are curling via https to avoid MITMs). In the curl-to-bash case, you have the _ability_ to at least inspect the code first before you install it (though, realistically, how many folks would really have the dedication and the ability to do so?), whereas with package managers you're installing "blind". Doesn't that make curling-to-bash the _safer_ option? I clearly must be missing something, here, because everyone decries "curling-to-bash" as a bad security practice, but I haven't seen an explanation of _why_. I tried doing some research for myself - this page[1] suggests that package managers have built-in checksumming and signature verification, but then goes on to propose an attack vector where the load balancer is compromised, so (presumably?) those same checksums/signatures could also be intercepted and faked. This gist[2] has a lot of people speaking against it, but from following those links, the only downside I could see was here[3], pointing out that the shell will keep executing if a connection closes mid-stream - which, yes, is certainly a bad thing and a good argument to use package managers, but not what I would call a security concern (unless you don't trust the entity on the other end of the stream -…

(Same discussion as on lobste.rs) Thank you for this link, I've only seen the initial CVE announcement. > This is not FUD [...] The site is practicing FUD, it accomplishes communicating a message in an untruthful fashion by mixing two different things into one. (Just check out their stack overflow links, it is not clear if they are talking about root or `--privileged`) People are confused wether `docker root == host system root` and this site doesn't help them to get a better understanding whether or not it is the case. (It isn't) Plus it misses what its main goal should be, running a Secure Docker environment. You are talking about a previous exploit, not a permanent issue. Keeping your host system up to date and additional hardenings is always going to be necessary in exposed environments. > Use Docker containers with SELinux enabled (--selinux-enabled). This prevents processes inside the container from overwriting the host docker-runc binary. Authors recommendation is also using SELinux, this also helped with outer Docker/Kernel related vulnerabilities in the past. Why isn't the page even mentioning this? --- I think it is important to give a proper outlook on how problematic things are and not to confuse people with super high expectations. You often end up running containers that you have only little control about. 1. Avoiding root in self-built containers is definitively the way to go, since it reduces (unnecessary) attack surface, but 1. It requires some…

Yes because other operating systems never have a decade old vulnerability? https://www.sysdig.com/blog/detecting-cve-2024-1086-the-decade-old-lin… And yes because their UI folks should be spending time on the kernel. What next? If Apple didn’t have so many people working at the Genius Bar they could use some of those people to fix security vulnerabilities?

> Given that, what are the concrete reasons people still consider Docker unsafe? Docker shares the same kernel as the host and is at risk of allowing container escapes and all it takes is for it to be exploited at the kernel level. [0] As long as you run Docker on the host (which doesn't provide kernel isolation), the host will always be at risk of container escapes. [0] https://www.sysdig.com/blog/runc-container-escape-vulnerabilities

The blog article is a bit more descriptive : https://sysdig.com/blog/stratoshark-extending-wiresharks-legacy-into-t… tl;dr version: system calls, but in the wireshark ui. (I've probably oversimplified that!)

Sysdig released a blog on friday. "For runtime detection, one way to go about it is to watch for the loading of the malicious library by SSHD. These shared libraries often include the version in their filename." The blog has the actual rule content which I haven't seen from other security vendors https://sysdig.com/blog/cve-2024-3094-detecting-the-sshd-backdoor-in-x…

There already is malware focusing on unsecured docker daemon's hoisted onto a tcp port [1] [1]: https://sysdig.com/blog/zoom-into-kinsing-kdevtmpfsi/

> "This entirely separate attack exists therefor completely removing an entire attack primitive haves no value" It has value, but it's also true that trusting cloud providers serveless infrastructure introduces additional sets of vulnerabilities due to various reasons. eg: https://sysdig.com/blog/exploit-mitigate-aws-lambdas-mitre/ Reading your comments, I get the impression that you are used to dealing with clients whose infrastructure management skills are lacking, and they are making a mess of things. While serverless infrastructures certainly eliminate a range of vulnerability classes, it is adoption is unlikely to be sufficient to secure platforms that are inadequate for the threats they face. At the end of the day, someone has to put in the work to ensure that things are patched, safe, and secure, whether the computing model is serverless or not.

Adding to the parent's point. The docker hub is full of malware images and the docker devs could not care less.1.2.3. 1. https://www.bleepingcomputer.com/news/security/docker-hub-repositories… 2. https://sysdig.com/blog/analysis-of-supply-chain-attacks-through-publi… 3. https://www.bleepingcomputer.com/news/security/thousands-of-images-on-…

similar arch: https://sysdig.com/blog/sysdig-and-falco-now-powered-by-ebpf/

https://sysdig.com/blog/labrat-cryptojacking-proxyjacking-campaign/ Actual source

IMO, there is no simple answer, since many access control elements are involved and implementations vary between organizations. From my experience I would say that there may be different factors: - Every developer is an exception - IAM is challenging to scale - Lazy IT Teams? - Visibility of access controls are poor Some useful references: - https://sysdig.com/blog/identity-access-management-difficult-cloud/ - https://www.effectiveiam.com/why-aws-iam-is-so-hard-to-use - https://aws.amazon.com/blogs/security/iam-access-analyzer-makes-it-eas…

https://www.redhat.com/en/blog/selinux-mitigates-container-vulnerabili… https://sysdig.com/blog/detecting-mitigating-cve-2022-0492-sysdig/ https://danwalsh.livejournal.com/30565.html

That article is very interesting! Looks like a similar approach to: https://sysdig.com/blog/hiding-linux-processes-for-fun-and-profit/ I wouldn't feel bad about it. The article provides info for security experts about a potential attack vector that exists. That doesn't change if you unpublish the post. Keep it up!

https://news.ycombinator.com/item?id=6650987 https://blog.dijit.sh//don-t-pipe-curl-to-bash https://sysdig.com/blog/friends-dont-let-friends-curl-bash/ https://www.seancassidy.me/dont-pipe-to-your-shell.html

The article's "container lifespan" chart shows that 21% of containers live less than 10 seconds and 54% live <= 5 minutes. If the base data set has that many very-short-lived containers, it's probably not representative of containerized persistent services. How many services get CD updates every 10 seconds? Indeed, the original Sysdig report that the chart comes from makes this case: https://sysdig.com/blog/sysdig-2019-container-usage-report/ "Many containers need to only live long enough to execute a function and then terminate when it’s complete. Seconds may seem short, but for some processes, it’s all that is required. We believe the increased use of Kubernetes Jobs that run finite tasks like batch jobs contributed to this growth. In fact, we expect short lifespans to increase, especially on serverless platforms that are well-suited to running short term tasks."

There is a blog on how to detect this CVE using Falco that goes into more detail. https://sysdig.com/blog/cve-2021-3156-sudo-falco/