Eric Hammond

This is a great blog for all things AWS Lambda: https://alestic.com/ The author is popular in the AWS developer community, and he's a really good guy to work with.

http://alestic.com/ is the best thing I've come across (Ubuntu focused, but many general solutions as well). FWIW, we (at DuckDuckGo) ended up in much the same place: ditched EBS, avoid anything that relies on EBS, and multi-zone and multi-region redundancy (also for latency purposes). For ephemeral storage purposes, we end up mainly using xlarge machines since they have the greatest stability and speed (with 4 drives in RAID-0).

Amazon's AWS docs: http://aws.amazon.com/documentation/ And +1 for Eric Hammond's AWS blog, which you can find here: http://alestic.com/

Two reasons come to mind: * the failure rate of an EBS volume (assuming a bug-free EBS software stack!) should be less than any one spinning disk * these machines don't have disk space set aside for copying the root volumes to them from S3 to boot. (There's a 10GB limit, and to allow for this to be somewhere, you'd have to have another SSD, or partition the existing ones.) There's a good blog post about it on Rightscale: http://blog.rightscale.com/2012/11/02/new-ec2-instance-types-and-coord… esh's "You Should Use EBS Boot Instances" also lays out a number of good reasons to use EBS for boot: http://alestic.com/2012/01/ec2-ebs-boot-recommended You at least have two 120GB ephemeral SSDs to play with when you're running - a luxury you don't have on the new M3 classes.

I've recently been looking at it too, and trying to find clarification of the retrieval fee. It seems there's a lot of (potential) misinformation, though I'm still trying to verify that. My guess is the best idea is to upload to S3, using whatever tools you can for that, and then archive from S3 to Glacier using the AWS lifecycle rules. See previous discussion at http://news.ycombinator.com/item?id=4411536 and http://liangzan.net/aws-glacier-calculator/ - my concern is that they don't seem to match Amazon's Glacier calculator at http://calculator.s3.amazonaws.com/calc5.html http://alestic.com/2012/12/s3-glacier-costs also seems relevant. So be sure you know what you're getting yourself into, otherwise you could end up with thousands of USD in billing. If you get clarity on the retrieval fee, please let me/us know.

wow - $0.085 for free! they better give Ubuntu 10.10 public AMI ID. http://alestic.com/ still doesn't list one.

https://www.infoq.com https://www.heavybit.com/library/ https://alestic.com https://blog.deepracing.io https://www.haproxy.com/blog/ Podcasts: https://www.stitcher.com/podcast/offerzen-podcast https://www.youtube.com/channel/UCzo_FqWUbEH0AYs0NilUWpA https://podbay.fm/podcast/996111298

I don't think I explained that very clearly. What you see as zone A is not what I see as zone A (maybe). When you sign up for an account, AWS assigns your zone A to one of the 3 available zones: zone X (real) : your A (virtual) : my C (virtual) zone Y (real) : your B (virtual) : my B (virtual) zone Z (real) : your C (virtual) : my A (virtual) That's why people see different outage characteristics between the zones. Edit: one of the few articles (other than HN comments) that explains it https://alestic.com/2009/07/ec2-availability-zones/

I'm no experienced Lambda user either, but here's a couple helpful things I've found: * I think API Gateway can throttle requests. Not sure if that was part of your pipeline or if your lambda had a different trigger. * There is a Town clock SNS event https://alestic.com/2015/05/aws-lambda-recurring-schedule/

Availability zones are randomized per accounts; my us-east-1a might be the same as your us-east-1b. This is presumably done to stop everyone putting things into the same AZ by just choosing the first one in the list. There is some more detail here: http://alestic.com/2009/07/ec2-availability-zones

Yes, the Lambda function is run in an Amazon Linux environment. I published some explorations of that environment here: http://alestic.com/2014/11/aws-lambda-environment Let me know in the comments on that article what other commands you'd like to see output for.

Very cool. Out of curiosity, are there any plans to make it easy to provision/create AWS boxes securely via chroot via Eric Hammond's approaches at Alestic? Currently it is a very tedious process to make a secure image on Amazon so that no credentials leak into history and whatnot, it is a tedious process. Making this easy would go a long way to making it trivial to use vagrant to safely publish public boxes for EC2. More details on this problem here: http://news.ycombinator.com/item?id=4457336 Examples of Eric Hammond's process at Alestic: http://alestic.com/alestic-git/ https://github.com/alestic/alestic-git

You can use this, many people have reported it does well. I have not used it. https://cloudyscripts.com/tool/show/5 You can also use the APIs to do that S3 copy I mentioned. Here are one of the many write ups of that technique. http://alestic.com/2010/10/ec2-ami-copy

Those on EC2 might want to read http://alestic.com/2010/10/ec2-ami-copy for their own disaster preparations. (Though ideally you should have been on it before now...)

You can use the EC2 API and ec2-describe-availability-zones to find out which availability zone is having issues: http://alestic.com/2012/06/ec2-outage-availability-zone

Individual availability zones can be identified using the API. ec2-describe-reserved-instances-offerings --region will tell you what the zone's identifier is. After you list the permanent identifiers, you can match them up to find out if your us-east-1a matches my -1d. This Alestic article shows how to label them all. [0] "Matching EC2 Availability Zones Across AWS Accounts" http://alestic.com/2009/07/ec2-availability-zones

You can match them up through a quirk in one of the APIs. http://alestic.com/2009/07/ec2-availability-zones My us-east-1a is the affected zone, which is 3a98bf7d-126d-411a-a612-3a57a62dc688 using the incantation on the site. (Oh, and to note: my us-east-1a was also the affected zone during the massive outage last year, and I believe I remember another outage sometime between then and now. I almost feel like every Amazon outage affects my zone. I kind of wonder if that availability zone just sucks ;P.)

They pointed out the thinking behind it. If you check out Eric Hammond's blog at Alestic, it expands on what was mentioned here: http://alestic.com/2009/09/ec2-consistent-snapshot

http://alestic.com/2011/08/ec2-max-instances Somewhat out-of-date (Aug, 2011) but a useful estimate.

Note: your "zone d" is not my "zone d". AWS shuffles zone IDs across users. See http://alestic.com/2009/07/ec2-availability-zones

"In order to prevent an overloading of a single availability zone when everybody tries to run their instances in us-east-1a, Amazon has added a layer of indirection so that each account’s availability zones can map to different physical data center equivalents." http://alestic.com/2009/07/ec2-availability-zones

I hadn't considered raiding EBS volumes. One problem could be the ability to do snapshots. But here is a decent article on using XFS to make your own snapshots. (http://alestic.com/2009/09/ec2-consistent-snapshot)

Alestic does list an AMI for 10.10 now. http://alestic.com/2010/10/ec2-ubuntu-maverick

This article includes a script that will perform a consistent snapshot of an XFS-formatted EBS drive (in linux). http://alestic.com/2009/09/ec2-consistent-snapshot Consistent snapshots are needed when the disk contains multiple files that must be kept in sync with each other - such as a database's data files and the journal. If your DB gets its disk snapshotted while these files are out of sync, the DB will need to recover the missing data by replaying the journal. Not all DB's can do this reliably for all types of changes (e.g. MySQL's MyISAM engine is fragile in this regard). So you need to make sure that the files in the snapshot all reflect a consistent state of the disk.

What I like most about developing on EC2 is the ability to save an development environment as an Amazon Machine Image (AMI), and then spin up instances of it as needed. I can fine-tune the environment over time (just spin up a new instance, make changes, and save a new AMI), as well as have different environments for different types of development. I think you could do something similar with Slicehost/Linode using disk images and configuration profiles, but it doesn't seem they offer anything like Amazon's machine images. Correct me if I'm wrong, though. Of course, the flip-side of all this is that any individual instance of an AMI is inherently ephemeral, and you need to treat it as such. Even the reliability claims of EBS are fairly weak. [1] That means EC2 instance storage, even EBS storage attached to an instance, isn't truly persistent, and I haven't quite ironed out my system for dealing with this. You'll probably want to run your own Git server [2] and commit code as it's completed. That still leaves the possibility of data loss if your instance dies after you've written code, but before you've made a commit. You could just take periodic snapshots of the EBS volume where your in-progress code resides, but whatever period you choose for the frequency (like every 30 minutes) is equal to the amount of work you could lose. And that sort of constant backup isn't really what snapshots are meant for anyway. [3] I think the ideal solution would be to run Dropbox to get…

I know this is a little late for your case but you can migrate snapshots from 1 region to another. We spend sunday moving from US-East 1 to US-West 1 and took 2 hours to setup our entire infrastructure consisting of 20 servers & about 180 GB database There are several options 1. Manually - http://alestic.com/2010/10/ec2-ami-copy 2. Automatically via Scripts a) migrate-ebs-image.pl - http://search.cpan.org/~lds/VM-EC2/bin/migrate-ebs-image.pl b) CloudyScript Migrate SnapShot - https://cloudyscripts.com/tool/show/4 c) CloudyScript Migrate AMI - https://cloudyscripts.com/tool/show/5 3. Commercial Services a) http://ylastic.com/ b) RightScale * ylastic is really cheap at $25 / month * I used the perl script migrate-ebs-image.pl, super simple to install and use. * CloudyScripts has open sourced their Ruby gem so you can build on top of it. They also have a free web form you can just use (but may not be secure enough). But you can launch their AMI in your own instance which should be secure Hope that helps for next time!