Code as Craft
codeascraft.com · Company blog · on HN since 2013
Code as Craft is Etsy's engineering blog, covering blameless postmortems, deployment, monitoring, HTTPS management, and engineering culture.
What HN saysHN commenters call it excellent, saying it taught them to monitor everything with statsd and showed Etsy's strong tech culture.
152HN people shared it
58linked it in comments
147stories submitted
2.3Kstory points
3 years agolast shared
2013people per year2026
Most-shared posts
- 1Blameless post-mortems and a just culture (2012)16 people · 105 points · first shared 2014 · 89 comments on HN
- 2Homepage14 people · first shared 2015
- 3Etsy’s experiment with immutable documentation10 people · 197 points · first shared 2018 · 62 comments on HN
- 4Etsy: Selecting a Cloud Provider7 people · 13 points · first shared 2018 · 1 comments on HN
- 5Etsy’s Debriefing Facilitation Guide for Blameless Postmortems5 people · 2 points · first shared 2016 · discussion on HN
- 6How Etsy Manages HTTPS and SSL Certificates for Custom Domains on Pattern4 people · 182 points · first shared 2017 · 22 comments on HN
- 7How Etsy Ships Apps4 people · 95 points · first shared 2017 · 7 comments on HN
- 8Improving the deployment experience of a ten-year old application4 people · 19 points · first shared 2021 · discussion on HN
- 9How Etsy Formats Currency4 people · 13 points · first shared 2016 · discussion on HN
- 10Etsy releases a tool for graph anomaly detection4 people · 3 points · first shared 2013 · discussion on HN
- 11Measure anything measure everything4 people · first shared 2014
- 12Etsy’s Journey to TypeScript3 people · 177 points · first shared 2021 · 152 comments on HN
- 13Developing in a Monorepo While Still Using Webpack3 people · 46 points · first shared 2020 · 15 comments on HN
- 14Transitioning to SCSS at Scale3 people · 40 points · first shared 2015 · 5 comments on HN
- 15API First Transformation at Etsy – Concurrency3 people · 20 points · first shared 2016 · discussion on HN
What HN people wrote when they linked it
whatismybrowser · 2016-02-25
Etsy's tech blog: https://codeascraft.com/ is excellent.
They got me on to monitoring EVERYTHING with statsd. Great stuff.
ajones · 2016-01-16
The tech and talent at Etsy would bring quite a bit of value to any company that were to acquire them. Their tech culture is top notch. I've followed their blog (https://codeascraft.com/) for a couple years now and have gained a ton of knowledge from it.
Some selected blog posts:
- Leveling Up Your Organization with System Reviews (Their latest post. https://codeascraft.com/2015/12/21/leveling-up-with-system-reviews/)
- Fault Injection in Production (http://queue.acm.org/detail.cfm?id=2353017)
- Q3 2015 Site Performance Report (Posted every quarter. https://codeascraft.com/2015/11/10/q3-2015-site-performance-report/)
jasonshen · 2016-02-01
Etsy | http://etsy.com | Brooklyn, NY ONSITE FULL-TIME
I'm a product manager here at Etsy and love it here. More than 20 million people buy from Etsy each year and we have 1.5 million sellers offering 36 million unique handmade, vintage, and craft items.
The people are really smart, friendly, a little quirky, and they care a lot about the product, the infrastructure, and the community. If you have read our engineering blog, https://codeascraft.com/, you know these folks know their stuff.
Feel free to reach out or comment with questions.
Jason
Full-Stack Software Engineer
You'll be helping us build new and awesome experiences for people shopping on Etsy. We're looking for full-stack engineers - you'll be dealing with the MySQL, Memcache, PHP, and front-end stuff (HTML/CSS/JS). The teams working on each feature are pretty small (a few engineers, a designer, and a product manager), so there's a lot of meaningful collaboration.
Because we're making user-facing features, we believe strongly in collecting and analyzing numerical data to help guide our decisions. To that end, you'll be able to use our in-house analytics tools to gather information and gain insights. You'll also be able to use Etsy's Continuous Deployment architecture to push code to production whenever you please (and you'll probably push on your first day here).
https://www.etsy.com/careers/job/oJoaWfwX
birken · 2015-06-13
I completely disagree.
If you work on a small engineering team and want to broaden your knowledge of technologies you might want to use, reading about approaches that larger companies use is a great source of information. Most cutting edge software engineering (not computer science), occurs in technology companies. So if you ignore "corporate" engineering blogs, then you are hugely limiting your exposure to how software engineering works in the real world.
Of course the companies are not blogging about everything, but that is irrelevant. Your choices are either a) not learn anything or b) learn stuff that might be dated (but probably also battle tested and rock solid).
And I think you are way off with the advertising thing. The reason most companies make these engineering blogs is to attract engineering talent, thus their incentive is to post cool stuff which is interesting and exciting. Look at Etsy's great engineering blog [1] which I've personally learned quite a bit from. Read a few posts... they aren't trying to sell arts and crafts.
----
1: https://codeascraft.com/, in the OP's list
mbesto · 2022-02-09
> we won't ever write articles
> Whitepapers.
I probably should have said "content" instead of blog because I agree 100% with this. Point still stands.
What the OP was referring to will likely become part of an engineering blog. a la : https://codeascraft.com/
jspaw · 2022-01-28
I coined this term (https://codeascraft.com/2012/05/22/blameless-postmortems/) and believe (productive and earnest) critique can be valuable when it comes to terms like this one. (See also “devops”, “agile”, “serverless”, “toil”, “technical debt”, and others)
Paul Reed has made a great case for ‘blame-aware’ (https://medium.com/@jpaulreed/why-blameless-postmortems-might-feel-wro…
Rein Henrichs also explores the topic very well through a unique lens (https://youtu.be/KXrsvLMqF1Q)
As Cipriani points out, there is the term and there is the concept(s) the term is intended to convey. It does seem to me more folks know the term than have read the origin (my post above from 2012). This is fine, as long as productive dialogue continues in the industry about what the term was made to convey.
lornajane · 2021-11-18
I'm a big fan of https://codeascraft.com/ which unashamedly long form writing about interesting technical problem solving. The authors do have their own voices but it's all at a really good standard.
quadrature · 2019-10-21
Documentation has a tendency to not keep up with what it is documenting, etsy has a great blog post about this and their shift in philosophy around documentation https://codeascraft.com/2018/10/10/etsys-experiment-with-immutable-doc… .
we've adopted something similar at work where we've necessitated that all support related questions be asked through https://www.discourse.org/ . This allows the most timely solutions to surface to the top and if a question hasn't been asked before it gets thrown into a slack channel where a subject matter expert can answer it. Making this one change has monumentally reduced the support burden of our teams.
cwt137 · 2019-05-29
You wont because they are a PHP shop https://codeascraft.com/?s=php
hoorayimhelping · 2019-02-24
John Allspaw applied concepts from The Field Guide to Understanding Human Error to software post mortems. When I was at Etsy, he taught a class explaining this whole concept. We read the book and discussed concepts like the Fundamental Attribution Error.
I've found it very beneficial, and the concepts we learned have helped me inn almost every aspect of understanding the complicated world we live in. I've taken these concepts to two other companies now to great effect.
https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/075464…
https://codeascraft.com/2012/05/22/blameless-postmortems/
https://codeascraft.com/2016/11/17/debriefing-facilitation-guide/
https://www.oreilly.com/library/view/velocity-conference-2011/97814493…
splatcollision · 2017-03-21
Etsy's blog post is a great primer (as usual) on how to accomplish this kind of thing: https://codeascraft.com/2017/01/31/how-etsy-manages-https-and-ssl-cert…
Corrado · 2017-02-21
Etsy is another company with extraordinarily talented writers. Their engineering blog "CodeAsCraft"[0] is superb and I invariably learn something everytime I read it.
[0] https://codeascraft.com/
johngalt · 2016-10-15
For actionable advice on how to be kind while still capturing the lesson. I recommend this article by John Allspaw on blameless postmortems.
https://codeascraft.com/2012/05/22/blameless-postmortems/
HealthyTree · 2016-08-18
A few that I follow on a semi-regular basis:
https://michaelfeathers.silvrback.com
http://highscalability.com
http://techblog.netflix.com
http://www.natpryce.com
https://8thlight.com/blog/
https://www.percona.com/blog/
https://www.smashingmagazine.com
http://blog.crisp.se/author/henrikkniberg
https://m.signalvnoise.com
https://dannorth.net/blog/
https://codeascraft.com
SEJeff · 2015-05-19
And Esty: https://codeascraft.com/
Netflix: http://techblog.netflix.com/
Facebook: https://code.facebook.com/posts/
Instagram: https://instagram-engineering.tumblr.com/
Pinterest: http://engineering.pinterest.com/
These are all excellent blogs to keep up on in your free time.
benihana · 2015-01-16
Blameless postmortems work phenomenally at Etsy which is a pretty low risk setting (after reading Sidney Dekker's book The Field Guide to Understanding Human Error [highly recommended] I would say that blameless post-mortems are even more important in a high-risk setting). Except failure isn't the correct word - the book makes the case that these are natural artefacts of complex systems.
1: https://codeascraft.com/2012/05/22/blameless-postmortems/
2: http://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/0754648…
vishalzone2002 · 2015-01-08
I like the approach explained by etsy engineers in the blog post https://codeascraft.com/2014/11/17/personalized-recommendations-at-ets…
adamkeeneetsy · 2022-01-06
Etsy is the global marketplace for unique and creative goods. With more than 100 million buyers, and over 5 million sellers, we are an ecommerce company at scale. As part of our mission to offer a best in class personalized search experience, our Search Engineering teams continue to grow. I include links to a couple of our live roles, and we welcome applicants from all across the US & Canada for these positions.
Engineering Manager, Search Platform: https://careers.etsy.com/global/en/job/JR1042/Engineering-Manager-Sear…
Senior Software Engineer II, Search Indexing: https://careers.etsy.com/global/en/job/JR605/Senior-Software-Engineer-…
More positions to go live soon, and these will be posted here at the next available opportunity!
For more information on our mission for search here at Etsy, please see this post from our engineering blog, Code as Craft: https://codeascraft.com/2020/10/29/bringing-personalized-search-to-ets…
sethammons · 2018-05-20
This reminds me of what we've been doing for a while, the "blameless postmortem." The technique is championed by Etsy, and you can read more about it in an article that introduces their debriefing guide:
https://codeascraft.com/2016/11/17/debriefing-facilitation-guide/
From the linked PDF from the article:
> “Adaptability and learning. We learn through honest, blameless reflection on lessons and surprises. We believe that traditional root cause analysis makes learning from mistakes difficult. Our blameless post-mortem process is a widely-cited technique that we believe is becoming best practice among organizations that value innovation. Blameless postmortems drive a significant percentage of our development as we analyze what about our production environment was less than optimal and rapidly make corresponding adjustments.”
(Etsy, Inc., 2015)
The idea, boiled down, is to inspect timelines, procedures, and actions and develop a narrative of how an incident came to be. The goal is for everyone to walk away with a (better) understanding of everything. With this, people are better armed to put into place solutions.
One example from the text is where an engineer pushes out a change because they thought the build system had zero failures. The push breaks the system and causes a regression that should have been caught in the tests. During the postmortem, the engineer says, "I thought the tests had zero failures. I guess I need to be more diligent in the future." Upon further…
kgraves · 2018-03-10
Etsy - https://codeascraft.com
bogomipz · 2017-05-18
>"AFAIK no one at Etsy has said that they run their own servers because it's more authentic, part of their "craft", etc."
I'm not sure I would agree with that based on a recent-ish post on “code as craft” the Etsy engineering blog:
"At Etsy, the vast majority of our computing happens on physical servers that live in our own data centers. Since we don’t do much in the cloud, we’ve developed tools to automate away some of the most tedious aspects of managing physical infrastructure. This tooling helps us take new hardware from initial power on to being production-ready in a manner of minutes, saving time and energy for both data center technicians racking hardware and engineers who need to bring up new servers. It was only recently, however, that this toolset started getting the love and attention that really exemplifies the idea of code as craft."
Source:
https://codeascraft.com/2016/02/22/putting-the-dev-in-devops-bringing-…
Tangentially related to the hardware discussion - a bit of dated hardware post:
https://codeascraft.com/2012/08/31/what-hardware-powers-etsy-com/
skl_ · 2017-04-21
conorgil145 · 2015-05-28
A few of my favoriates from my RSS reader (in no particular order) include:
- http://www.kalzumeus.com/blog/ I love reading anything by Patrick McKenzie because he writes about incredibly valuable things, usually focusing on how to make sales, lifecycle emails, running his companies, and other general start-up related information. You might recognize him on HN as patio11
- https://blog.cloudflare.com/ CloudFlare has really excellent content and often writes about SSL/TLS related topics, which personally interest me. They have good technical explanations of SSL/TLS vulnerabilities and discuss how they provide SSL to all of their customers. Also, they discuss how they handle massive scale and other interesting things.
- https://aws.amazon.com/blogs/aws/category/week-in-review/ I often find really good articles linked from the AWS blog. I like to stay current on the latest/greatest features and improvements which AWS puts out there and the Week in Review posts are often easy to skim and see if a linked article is worth reading in its entirety.
- https://codeascraft.com/ Etsy's engineering blog is fantastic. I am always blown away at the passion that comes through from their posts. They seem to have a really sharp set of devs over there and I just enjoy reading their stuff. I especially enjoy the quarterly performance review posts, which show graphs about latencies throughout their stack.
- https://strongloop.com/ The StrongLoop blog has some really great content for NodeJS…
iandanforth · 2015-01-06
Another good package to look at is Etsy's Skyline - https://github.com/etsy/skyline
Their introduction of the Kale stack (which includes Skyline) is a great read - https://codeascraft.com/2013/06/11/introducing-kale/
I spent a month or so evaluating anomaly detection systems and I can tell you a few things the twitter post fails to mention
1. You can get a long way with an ensemble of simple techniques. And it's always better than any single technique.
I wouldn't recommend trying to install skyline but re-implementing the ensemble of anomaly classifiers they have might take you a day or two and it will get you 90% of the way there.
2. The False Positive rate is probably your most important metric.
Detecting anomalies is good, but your ops team already has plenty of alerts to deal with. If you throw false positives at them from a new system they will hate you and ignore the new system. Most papers report ROC curve data as the metric for classification, that can be ok too.
3. Don't build something complex when a threshold will do.
If a point anomaly is obvious to a human, you absolutely should not build a complex system to detect it, just use thresholds. It's only when you want to detect anomalies before they cross a threshold that you should start on this kind of task. That leads me to
4. Almost all anomalies have a temporal component.
If your detector isn't ultimately looking at multiple sources of data and finding patterns that initially look like normal behavior…
bucky · 2014-03-01
Etsy - Brooklyn, NY. Some positions are accepting remotes (check descriptions).
https://www.etsy.com/careers
We're looking for everyone! Web/mobile apps/tools/testing engineers; Designers; Ops; Marketing; We have a lot of open reqs for managers all over the world as well.
---------------------
Etsy is the world's largest handmade, vintage and craft supplies marketplace. We are a values based company that believes that we can make a profit without violating our core beliefs [1]. We practice continuous deployment [2], and we measure everything [3].
I've never worked with finer engineers, product managers, designers, or data analysts in my life. I recently switched teams to work in a completely different domain because I wanted to scratch a different itch, and it was no problem for my manager or his managers. Highly recommended.
[1] (http://www.etsy.com/blog/news/2014/etsy-values-impact-annual-report-20…
[2] (http://codeascraft.com/2010/05/20/quantum-of-deployment/)
[3] (http://codeascraft.com/2011/02/15/measure-anything-measure-everything/…
danso · 2014-01-30
If you are in New York, you should subscribe for the Etsy Code as Craft speaker series, and not just because they always have free beer http://codeascraft.com/etsy-speaker-series/
Dan McKinley's talk on continuous experimentation of design is a must-watch http://mcfunley.com/design-for-continuous-experimentation
The last one I went to was a sparsely attended talk on using multiple CDN providers...sparse only because of the specific and technical nature of the talk, but it epitomized the scale and methodology of Etsy's testing operation...most non-massive techs are probably satisfied with moving to a reliable CDN...Etsy seemingly puts as much effort in testing CDN performance as smaller firms would in overall A/B testing. But I still found it to have some valuable insights for us small-operators, such as where the humans are involved in the monitoring process (given the investment they've made in automated systems)
http://velocityconf.com/velocityeu2013/public/schedule/detail/31536
bichiliad · 2023-11-05
Bit of a shameless plug, but I wrote about how Etsy migrated to TypeScript a few years back. I agree with the author generally: TS strict mode is the way to go, and you may find yourself regretting the alternative, even if it sounds appealing. It takes a lot of work, but most of the effort ended up being:
1. writing types for the big, central libraries that everyone uses (and make them "smart enough" to infer as much as possible).
2. teaching everyone at the company a brand new language without them hating it, and without their team losing (much) velocity.
https://codeascraft.com/2021/11/08/etsys-journey-to-typescript/
acemarke · 2021-11-17
I saw in https://codeascraft.com/2021/11/04/mobius-adopting-jsx-while-prioritiz… that you're using Redux Toolkit (which I created and maintain). I'm curious if you have any thoughts on using RTK in general and RTK+TS specifically.
p1necone · 2021-05-18
Working out the initial cause of this Covid outbreak is far far far less important/actionable than working out what we should do differently when (not if) another virus starts spreading.
This concept should be familiar to those in tech already: https://codeascraft.com/2012/05/22/blameless-postmortems/
mncharity · 2021-02-19
> facilitator to run a blameless postmortem
Here's Etsy's 2016 guide[1]. A bit old, but if it's been obsoleted by something, I'm not quickly finding it. Other suggestions? Its CareSets fork[2] saw a bit of work in 2020.
[1] https://codeascraft.com/2016/11/17/debriefing-facilitation-guide/ https://extfiles.etsy.com/DebriefingFacilitationGuide.pdf
[2] https://github.com/CareSet/DebriefingFacilitationGuide_FromEtsy