0x20


A blog about mobile, maker, and embedded development.


The Amazon Experience

Last year I left Amazon after being there for 7 years. When I read the recent (August 15th 2015) New York Times article on Amazon’s corporate culture it set off a period of reflection on my time there. Because I learned so much from the talented people I worked with I find it important to revisit those lessons with perspective and objectivity. This blog post is my own personal retrospective of my time at Amazon and a retroactive application of "vocally self critical" if my former colleagues will allow me the conceit.

Conclusion

Amazon teaches you to put your conclusion at the start of a document. It’s sort of their version of a TL;DR. Sorry but you’ll actually have to read this post if you give a shit. If it’s TL then just DR.

The Bad Year

I received one bad review in my 7 years at Amazon. It happened that this review period included the time my wife was in the hospital for over two months of bed rest while carrying our twins and the time afterwards when our children where in the NICU after being born fairly small. I, obviously, was juggling a lot between work and family at that point. Was it fair that my job performance was rated without taking that into account? I’m not sure. I admit I didn’t do my best work and in a vacuum the review was fair. But the review didn’t feel like a isolated analysis of my engineering output. It felt like a critique of my overall ability and ongoing value as an employee. It felt like a reproach for letting my personal life effect my work.

At the same time a colleague under the same manager was going through a divorce. He was "managed out". He was smarter than me and more qualified but he wasn’t working as many hours. That situation demonstrated an extreme lack of empathy from Amazon management. To be fair; this all happened over 5 years ago and the groups I worked for in subsequent years were a real joy. Still over the years there was a steady stream of people being placed on "performance improvement plans" (which always ended in termination) that seemed to be based more on a need to fill some hidden quota then actual problems.

Amazon is a big company and I was there for a (relatively) long time. I could provide a long list of points on both sides of the good and evil scale. In all I don’t think Amazon is especially "evil" compared to any other American firm but it does have a lot of areas it needs to improve on if it wants to hire and retain experienced talent.

The Good Years

During my time at Amazon I did work for many managers that I would rate as the best of the best. These were people who were empathetic and many had families of their own, yes; but they also were managers who used data, intellect, and wisdom to amplify the effect of dozens of people all working together towards the same end. My experience in these good times proved to me that the principals developed and adopted by Amazon can be interpreted in a way that makes an excellent and effective work culture.

My good years at Amazon were spent building things people wanted, that weren’t completely frivolous, while working for smart people with high but reasonable expectations. These good managers had a keen understanding of what was happening "on the ground" even as they were translating more detached directives from upper management. One of my current co-workers, John Ikeda, described this as being the "master sergeant manager." The highest ranking NCO, so-to-speak, that protects their platoon from the shit they take from the officers. Based on my considerable military experience with "Saving Private Ryan" and "Call of Duty" I find it an apt analogy.

What I learned from watching these managers ("You’re one of the good ones." "Some of my best friends are managers.") is to use Data as a way to avoid micromanagement rather than the inverse. Good managers watch the data flow and get to know it like the feel of an engine through the deck of a ship. They are able to stand back when the data "feels" right and only interfere when they feel a strange vibration coming up from the data beneath their feet.

Customer Obsession

Thinking about where groups within Amazon were good and where they were bad I find one key difference in how the group interpreted Amazon’s "god principal"; namely "customer obsession." Laser focus on the customer in all aspects and at all levels of the business is one of, if not the, key(s) to Amazon’s success. Interpreted by reasonable people when optimizing decisions and making hard calls this principal can reduce the amount of vanity and self-indulgent bull-shit a company engages in. Where it goes wrong is when this principal is used as an excuse for making people work long or unreasonable hours. The argument goes "customers want to be able to buy widgets at 1am and our widget service is down! You must stop everything else you are doing and fix this now!" Amazon needs to understand the inalienable '0' principal: "We don’t love our customers more than our families. We don’t love our customers more than ourselves." I don’t think any rational person would expect that any retailer be made up of individuals that love them more than life. Any organization that attempts to practice this sort of perverse customer worship is as much a cult as a business.

But this sort of religious zealousness wasn’t the most common abuse of "customer obsession" I saw. More typically the principal was perverted as an excuse to take shortcuts. It became a rug under which all manner of poor engineering practices were swept. The reasoning goes like this, "I can have my engineers spend the next week fixing a problem that happens once a month and is easily fixed by a human with 5 minutes of effort or I can have them build a new feature our customers want." Perhaps in isolation this tradeoff seems reasonable but compounded the "once a month" problems begin to add up until the systems owned by the group need constant human intervention to keep them running. The problems snowball and engineers leave the group in frustration further compounding the problem for those that remain. This is a pattern that Amazon repeats time and time again in its rush to build new services and features (For a concrete example, read this article). Even during the last months of my employment I became aware of a team that is doing just this; building systems held together with bubble-gum and duct tape in a mad rush to meet unreasonable schedules. This team was clearly at the start of the cycle I’m describing here and all the "customer obsession" excuses were deployed in typical fashion to justify the situation. Based on this experience I question the validity of the operational load metrics we were told showed a vast improvement in oncall rotations over the last few years.

In this operational death spiral the final insult comes when new employees, normally young college graduates, are used to keep the derelict systems running until they can be replaced. To do this managers use the fact that new hires are not allowed to move to another group for the first year after they are hired. Furthermore, if they do not get a good review they are not allowed to move until their performance improves. This means young developers are thrown into these grinders where they have to work with poorly designed infrastructure that often fails and is hard to fix. The engineers that built the software are gone and there is little documentation to be found. They are all but setup to fail which means a bad review and yet another year enslaved by a pager.

These quagmires can only be avoided by better strategic thinking. Rather than design software with only the first release in mind managers and engineers must first consider what the evolution of the systems could be. The immediate counter argument I hear when I say this is "we wouldn’t get anything done if we waited around until the future was clear." There seems to be this notion that agility and innovation require poor planning and short-sighted thought. Failed engineering organizations start like a chess player that thrusts out some random pawn without first devising a strategy. They are forever reactive and unable to respond to the demands placed upon them. In the face of uncertainty there has to be some evaluation of "possible moves" beyond the first one. A business, a service, a technology must be planned as both its first iteration and the possible changes that will come next. You don’t need to know every move in the game but you do need a framework that supports your strategy for winning. This planning means preparing for success as much as failure. Time and time again I’ve seen groups struggle because they were successful and weren’t prepared for the load that a robust user base placed on them. They weren’t ready for the next move even though they were given a fleeting opportunity to play offense.

The Conclusion at the End

Like I said in the first conclusion, I’m not drawing any simple lessons with this post. There are some like; ambition, hard work, and fearlessness actually can’t replace intelligence, planning, and craftsmanship when making things; that being kind is hard but valuable; that large companies are made up of individuals each with their own strengths, weaknesses, and habits; but in the end the last seven years were more about raising my two kids Huck and Esme and making a life for myself here in Seattle with my wife Kathleen. Amazon was work. Good work, great pay, and really awesome health insurance but in the end it was a job. If you are currently working at Amazon and it’s more then just a job that’s when you need to stop and evaluate your priorities. That’s when you can find reasons to work yourself into the ground and be miserable. Work hard, yes, but for god’s sake, if you’re not having fun you’re doing it wrong.