Henrico Dolfing

My favorite Accenture story is when they charged Hertz $32 million for a website that never went live. Wonder if they will perform better with a smaller headcount. https://www.henricodolfing.com/2019/10/case-study-hertz-accenture-webs…

Reminds me a lot about the fascinating read of Knight Capital, and the $440M-in-28-min bug that lost them 75% of their equity. Reminds me a lot about Knight Capital bug that cost them $440M in 28 min. > Rogue orders seemed originated from the new RLP router code, but no one could pinpoint the bug.... they reverted to last-stable.... and even more trades executed than before. https://www.henricodolfing.com/2019/06/project-failure-case-study-knig… I remember a PDF that went into even greater detail, was a very good read.

This is a good time to remind everyone of the rules of SAP integration: 1) Your company processes MUST adapt to SAP processes. That's it. You can try customising SAP to fit your processes, but you will just waste time and hundreds of millions of money and then you will fail. Lidl spent almost a decade and 500M€ on its SAP project, but didn't follow rule #1: https://www.henricodolfing.com/2020/05/case-study-lidl-sap-debacle.htm…

What became universal in the industry is an item on the checklist saying that you have safety checks to prevent this. How seriously that item is taken, i suspect, varies quite a bit. In my experience, you aim for multiple extremely simple checks, with minimal logic and minimal calculation, so you can have confidence they won't have surprising behaviour in an unusual situation like this. The classic example is an order count limit - initialise a counter to some value at startup, and every time the machine sends an order, try to decrement it. When it hits zero, it can't send orders any more. Just throw an exception or return early or something. You display the value of the counter to human operators, and give them a button which resets it to the initial value. In normal operation, you are sending orders at a steady trickle, and humans will have to press the button every now and then. If something goes insanely wrong, as here, the counter will run down quickly, and then the humans hopefully won't push the button, because something is obviously wrong. It's a very crude safety, but it is a simple one. Another is a limit on message rate. You could use a token bucket filter. Does not affect normal operation, but stops a machine which is spraying out excessive orders. You could have it so that if the bucket runs out, it turns off until a human explicitly turns it back on. You have limits on net position too, to stop you running up huge positions in anything, but those are…

> Someone at Thoughtworks sold Dominion Energy on the idea that Thoughtworks... if things go well, they will get follow-on business and maybe even a whitepaper. ... > Market pressures don’t apply to internal teams Market forces are constantly applied against internal IT teams, particularly in non-tech companies. The market force is replacing them with consultants. Sometimes they will make those employees train their own replacements[1][2]. Or that there are consultants that can't be fired. The project is over schedule, but too much sunk cost to replace them and no internal devs. A $6M project becomes a $1.25 billion project and fails in slow motion over a decade.[3] Anyway, my point is that it's not as simple as "internal is always safe, consulting is always tougher". [1] - https://www.nytimes.com/2015/06/04/us/last-task-after-layoff-at-disney… [2] - https://www.mercurynews.com/2016/11/03/after-pink-slips-ucsf-tech-work… [3] - https://www.henricodolfing.com/2019/12/project-failure-case-study-quee…

[1] suggests that part of the problem is that Lidl's business processes were non-standard, and Lidl decided, rather than change their processes to match what SAP's solution supported, to instead customise SAP's solution to support their non-standard processes – and that was the point at which the project came undone, the SAP customisations couldn't scale to high turnover (but surely vanilla/uncustomised SAP Retail would not have had this problem) Lidl's non-standard processes may well be a big part of their business success. But maybe that was a sign they shouldn't have gone with an off-the-shelf solution like SAP, and should have stuck with their legacy in-house system, or tried to build a custom next generation in-house system. And their experience isn't necessarily transferable to another business whose processes might be more standard, or who might have more flexibility to alter their processes to meet the needs of an off-the-shelf solution. And if they'd gone for some other off-the-shelf solution instead of SAP (such as Oracle or Infor), they might have had just as many problems – the more customisation heavy an ERP implementation project is, the greater the risk of failure. [1] https://www.henricodolfing.com/2020/05/case-study-lidl-sap-debacle.htm…

Imagine if that was Meta that had over 1B users with their messenger desktop app update functionality that did just that. The loss would be in the hundreds of millions. > But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this. The lesson here is much better use of automated tests (The app likely has no tests at all) and proper use of basic testing principles like TDD would prevent such junior-level embarrassing bugs creeping up in production paid software. That is the difference between a $100 problem vs a $200M problem. See the case of Knight Capital [0] who lost $460M, due to a horrific deploy. [0] https://www.henricodolfing.com/2019/06/project-failure-case-study-knig…

In at least one case, a test routine activated in production created a terrible loss https://www.henricodolfing.com/2019/06/project-failure-case-study-knig…

There is hardly a company I trust less than Lidl in the realm of tech "Case Study 12: Lidl’s €500 Million SAP Debacle" (2020): https://www.henricodolfing.com/2020/05/case-study-lidl-sap-debacle.htm… They're already generating revenue so congratulations to them, but would I put my infrastructure in their hands for the long run?...

They add liquidity to the market, so when you (or someone indirectly on your behalf) buy or sell stocks, you (they) get a better price through narrower bid-ask spreads. They also sometimes generate comedy[0]. [0]: https://www.henricodolfing.com/2019/06/project-failure-case-study-knig…

Also, what's your approach to risk management? For example: 1. Do you have anything that limits the size of a single trade or position? 2. How do you measure and manage overall volatility/risk/VAR to your portfolio? 3. What kind of safeguards do you have to avoid catastrophic bugs? (For reference, see Knight capital and how a single bug brought down the entire company: https://www.henricodolfing.com/2019/06/project-failure-case-study-knig… Given how fast your system trades, I imagine it must be difficult to visually spot these errors. (You did mention paper trading, but I wonder if you have anything else you want to mention) Thanks so much for sharing your knowledge publicly. It's very much appreciated!

I feel for folks affected by layoffs, but accenture is a garbage company. [1] The sooner off-shore body shops are gone from this earth, the better. [1] https://www.henricodolfing.com/2019/10/case-study-hertz-accenture-webs…

> And avoid some code bug causing them to lose all their money Well, everything has a non 0 chance but you can code certain conditions that have to be met before an order goes through. IMO, you would have to have significant capital to even end up in a situation where a bug causes lots of loss. Here is an example of something you might be thinking of. https://www.henricodolfing.com/2019/06/project-failure-case-study-knig… > Not get lose all their money to some company's API fees It's not too bad these days. A lot of brokers offer "zero-fees" which really just means you only have to pay the SEC ones which aren't too bad.

> A seemingly innocuous engineering change led to some drastic consequences. If you think that this is "drastic" you should read "Case Study 4: The $440 Million Software Error at Knight Capital". https://www.henricodolfing.com/2019/06/project-failure-case-study-knig… tldr; At Knight, some new trading software contained a flaw that became apparent only after the software was activated when the New York Stock Exchange (NYSE) opened that day. The errant software sent Knight on a buying spree, snapping up 150 different stocks at a total cost of around $7 billion, all in the first hour of trading.

Knight Capital will forever haunt fintech engineers... https://www.henricodolfing.com/2019/06/project-failure-case-study-knig…

See also: "Lidl’s €500 Million SAP Debacle" https://www.henricodolfing.com/2020/05/case-study-lidl-sap-debacle.htm…

This article describes their efforts at streamlining their systems, which resulted in lawsuit against Accenture: https://www.henricodolfing.com/2019/10/case-study-hertz-accenture-webs…

This was in the case study linked elsewhere: > Lidl based its inventory management system on purchase prices. The standard SAP for retail software uses retail prices, and fearing the group could lose a competitive edge by compromising, Lidl declined to change, so the software was instead adapted. https://www.henricodolfing.com/2020/05/case-study-lidl-sap-debacle.htm…

I have a rule now that when designing a page, any "money screens" get at least 1.5X to 2X the estimate. I define a "money screen" as anything that leads a company to land a client or land a sale, things like checkout flows, signup flows, etc. Usually that extra time gets sucked up in A/B testing setup and setting up a staggered deployment per region that the biz operates. Whenever customers push back I tell them the story of Knight Capital [1]. You pay extra for extra assurance that you won't loose a shit load of money in the future. [1]: https://www.henricodolfing.com/2019/06/project-failure-case-study-knig…

Yes. UK 2018 TSB bank IT meltdown shutdowns all accounts for weeks: > With clients locked out of their bank accounts, mortgage accounts vanishing, small businesses reporting that they could not pay their staff and reports of debit cards ceasing to work, the TSB Bank computer crisis of April 2018 has been one of the worst in recent memory. The bank’s CEO, Paul Pester, admitted in public that the bank was “on its knees” and that it faces a compensation bill likely to run to tens of millions of pounds. https://www.henricodolfing.com/2019/03/case-study-epic-meltdown-of-tsb…

Don't forget crashing the first Ariane 5 on its test flight[1], incorrect floating point math[2] and losing $440 million in 30 minutes[3]. 1 https://www.esa.int/Newsroom/Press_Releases/Ariane_501_-_Presentation_… 2 https://www.cs.earlham.edu/~dusko/cs63/fdiv.html 3 https://www.henricodolfing.com/2019/06/project-failure-case-study-knig… The Error of Our Ways: https://www.youtube.com/watch?v=3YaI6lhn78g

Here's another famous example where the damage was thousands of person-years of salary: https://www.henricodolfing.com/2019/06/project-failure-case-study-knig…

This is small potatoes. When the UK government really puts its mind to it, it can screw up far more royally: How about 10 - 12 billion GBP (depending on who you ask) for a National Health Service computer system that never worked and was never delivered. https://www.theguardian.com/society/2013/sep/18/nhs-records-system-10b… https://www.independent.co.uk/life-style/health-and-families/health-ne… https://www.henricodolfing.com/2019/01/case-study-10-billion-it-disast…

This story makes me recall Hertz $32M Accenture website services bill. https://www.henricodolfing.com/2019/10/case-study-hertz-accenture-webs…