Behind Enemy Lines

Be sure to also read the follow up for the actual solution that Ad-safe implemented here: http://www.behind-the-enemy-lines.com/2012/03/50-of-ads-are-never-seen…

Amazon's not amazing at announcing useful timelines for their products/services. There are presumably a bunch of internal factors at play, but spending limits aren't the only thing that's been "in the works" for ages. If you're holding out for something that may or may not be coming (eg DevPay in your region, hard spending limits, long-term-queues, scheduling, ability to cloudfront s3 'subfolders' etc) it's frustrating, but on the other hand, they do tend to release well thought out services that make sense. If it's any consolation, I have heard of individuals getting huge bills and having them cleared by Amazon when it became apparent that they hadn't willingly racked up that many fees (like the guy who included his S3 images in a google spreadsheet causing several TB of requests - http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self… ).

Have a read of this article: http://www.behind-the-enemy-lines.com/2012/04/feedback-unemployment-an… There's good reasons to believe that the portfolio that designers develop through 99designs is likely to be worth far more to them than the win itself.

I'm glad someone else is calling this for what it is: a way to get around minimum wage. Another solution would be to use worker reputation (something explored by Panos Ipeirotis @ http://www.behind-the-enemy-lines.com/ ) Because MT workers are unreliable, you often need to have several doing the same task. Better workers require fewer checks, and are therefore much more cost effective. Right now MT is a market for lemons. So the challenge then would be to let poor people build up a reputation to fix that market...

> What kind of AWS setup leads to surprise bills? You only need a single configuration wrong to get bankrupt overnight, and AWS has so many knobs that make a switchboard look like a toy. I remember things like [1]. It’s very easy for something you don’t control go rogue, and AWS offers no protection for such scenarios. 1: https://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-sel…

"TL;DR: There are about 100K-200K unique workers on Amazon. On average, there are 2K-5K workers active on Amazon at any given time, which is equivalent to having 10K-25K full-time employees. On average, 50% of the worker population changes within 12-18 months. Workers exhibit widely different patterns of activity, with most workers being active only occasionally, and few workers being very active. Combining our results with the results from Hara et al, we see that MTurk has a yearly transaction volume of a few hundreds of millions of dollars." https://www.behind-the-enemy-lines.com/2018/01/how-many-mechanical-tur…

... and has had security incidents http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self…

Or DDoSing http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self…

http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self… Discussion of the above: https://news.ycombinator.com/item?id=3890328

Possibly related: http://www.behind-the-enemy-lines.com/2011/03/uncovering-advertising-f…

This is about two years old: http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self… I would hope that Google is able to detect abuse of their infrastructure for (D)DOS.

someone already did it, and to himself :D http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self…

There was a case awhile back where Google docs helped a guy rack up a huge AWS bill. http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self…

If you have to watch the video for 15-30 minutes, I think it's not the persuasion aspect but probably loading different sites in an iframe to defraud advertisers: http://www.behind-the-enemy-lines.com/2011/03/uncovering-advertising-f…

There are some crazy clever schemes out there for click fraud. Check out this link - _WARNING NFSW IMAGES_ http://www.behind-the-enemy-lines.com/2011/03/uncovering-advertising-f… This was estimated to be making $500K a month before being found... and was a work of pure genius.

I imagine that must be their thinking. Clearly a lack of technical resources it's not. Moreover, Amazon has already shown [1] good will when negligence led to an unexpectedly high bill. I'm surprised (though empirically without much reason) not to hear stories from NewHotness et al about "how AWS burned five times our monthly budget in one day of frenzy". Even if those new visitors may very well be welcome for the business, I imagine there are many folks who would like some control (say, to double the limit), maybe at the risk of leaving the site offline for a few minutes. [1] http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self…

I decided to do some digging and found some demographics[1] of mTurk users. It's far more diverse than I thought. 1. http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mech…

http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self… Interesting post which showed how badly configured linkages between private google docs and S3 caused an accidental DoS.

Apart from the small sample size, if they collected data by mechnical turk then I wonder what sort of biases that introduces? In the paper you linked, Frances and Mialon state: "Samples of mTurk workers have been found to be more representative of the US population than in-person convenience samples, standard internet samples, and typical college samples" I am somewhat sceptical of this, and there seems to be some evidence to back-up my scepticism [1][2][3]. In particular, [2] states: "In sum, the MTurk sample is younger, more male, poorer, and more highly educated than Americans generally. This matches the image of who you might think would be online doing computer tasks for a small amount of money." Which are some of the factors that the marriage study itself seeks to examine. This looks like lazy data collection methodology to me. [1] http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-… [2] http://themonkeycage.org/2012/12/19/how-representative-are-amazon-mech… [3] http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mech…

Humans being the error-prone creatures we are, I wonder how it could work better to a longer word-based code, but allow for certain kinds of errors: for example, 1) word order doesn't matter, 2) it tries to correct a word not in the built-in word list to the "closest" word in it, 3) maybe you only need, say, seven of the eight words to be right to get in (then it might remind you of the real code). You can do a little calculation to see how many words you need to get to the desired number of bits then. Pretty different from passwords as we know them (#3 means you can't simply store a bcrypt/scrypt'd code--you could build something complicated tricks to square "allow one word wrong" with not storing plain PWs: each word is 10 bits and what you tell the user is really their secret code + a parity/ECC word, and correction happens before a convnetional password check--that rabbit hole goes deep, though.) But if all this gets average folks remembering more entropy than before, that makes it kind of interesting. On the study (which I admit just skimming), survey in 2010 suggested about half of Mechanical Turk users were located in the US (http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mech… who learned English at school would probably do better at memorizing a password in their native language. (I studied French many years in school, but I doubt I'd memorize a set of random French words as easily as English words.)

Turkit, Javascript code for achieving consensus among AMT workers, especially with multiple stages. See the Find-Fix-Verify pattern: http://www.behind-the-enemy-lines.com/2011/04/want-to-improve-sales-fi… By doing multiple stages of crowdsourcing, you can do complicated tasks with high quality, like: Improve the grammar of these reviews. Zappos did this and saw a lift in conversion. Crowdflower, which is good for achieving high-quality one-shot annotation, but not so much for pipelines of annotations. MobileWorks, which is good for achieving high-quality one-shot annotation, and says in the docs that it has pipelines. I haven't figured this feature yet out. CrowdControl, which supposedly solves every problem, but is priced as an enterprise solution. If you want to build something cool, implement pipelines of work. i.e. build a crowd-programming layer that has subroutines. Look at what CrowdControl says they are doing. [edit: If you also want to build something cool, implement a reputation system. Don't just assign workers a single number. Figure out what kind of tasks the workers are good as, and do a per-task reputation system. For bonus points, solve this correctly, by dynamically gauging the skillset and difficulty for each task, rather than simply grouping tasks into N clusters, where N is low.] Email me if you want to discuss. I've been thinking about this for a while.