Infrastructure: to build, or not to build?

Without a doubt one of the most arguable subjects at any software company is:

"What is the right timing for building an infrastructure?"

This question can come up in all kinds of forms and shapes. For example, it can be a statement from the developers saying: "I'm sick and tired of handling bad code, from now on I'm building the best infrastructure there is!". Or, it can be a question raised by the management: "Why the hell didn't you build an infrastructure to support this?!".

No matter where I worked, being it a large corporation or small startup, there always comes a time for a discussion about the timing of creating the infrastructure for the product.

We are educated throughout our studies, as well as during our work, to invest in proper design of our code. We are encouraged to create good infrastructure in order to have the ability to make changes quickly and easily. We all know that the design should allow the code to be extendable, maintainable and testable.

But as time passes by, things such as funding, unreal timelines and lack of resources are starting to affect our technical judgment.

---

Perspective 1 - The Developer

Anytime. Prior to anything else.

Software developers are the core technical people that build and operate the product. They live and breath the code and its production. They sit at design reviews and fix bugs, but most importantly they are the ones that need to implement the customer requests. However, these requirements are often twisted and changed by the sales, management and product owners, adding unrealistic timeframes which makes the effort even harder.

This is the reason why developers turn to the most sacred thing they know, a proper design and a good infrastructure.

But when time is of the essence and the deadlines are near, they are forced to cut off many things from the development (more about this at: False Time Estimation Syndrome). One of these things is the investment in building infrastructure, which almost always seems to take too much time and have unknown ROI.

Although the developer's perspective is often true and have a clear vision of things to come, it almost always lacks the business insight.

The sad truth is that good infrastructure is never a guarantee of income, in fact I have never met a customer that bought the product merely because the CI/CD was so great.

So, spending two month on building some super crazy infrastructure or refactoring half the product so that you'll be able to support every future deviation of the development might seem like a good technical approach but it can also make the company go bankrupt.

Perspective 2 - The Development Manager

Yesterday. As long as it doesn't change the work plan.

Software development managers are developers at their core, they have the same perspective as their team members, but as managers they have a broader view of the company needs. They start to engage more and more in the business aspect, making them more thoughtful about the development time as well as the ROI of every single task that their developers do.

Having said that, they still expect the infrastructure to be solid enough so that they will be able to deliver any feature with as minimal effort as possible.

For them the implementation of a good infrastructure is something that goes without saying.

When I was just a junior developer I had an argument with my manager over a major feature implementation. I was claiming that we need to invest in the infrastructure otherwise we won't be able to support any changes in the future. His claim on the other hand was that infrastructure is important but getting the feature on time is more crucial so we need to find a way to do both.

This perspective is as predictive of the future as the perspective of the developers, but it doesn't provide the tools for it to work. It's one thing to expect that the infrastructure will be already implemented, but it's another thing to actually build it prior to everything else.

Perspective 3 - The Product Manager

Only when the product requires it.

Product managers come from various backgrounds, but most of them are people that started at R&D.

Most of them understand the developer's point of view, but unfortunately they are not developers. The most frustrating thing for them is hearing the developers, or their managers, say that in order to develop some new requirement they need to build some vudu infrastructure. Investing so much time in something that is the core product, just for the sack of having the ability to change something that haven't been yet defined, seems very odd.

Most product managers will ask a simple question: "How does this serve the product?" and usually the answer will be: "This serves the development by...", hearing such answer will irritate anyone.

Although both intend the same: having the ability to deliver the required product in the best way possible, this reflects the biggest difference between the perspectives. The first talks about the "product" and the other talks about the "development".

Perspective 4 - The Sales

Never.

Sales people are the key driver for bringing cash flow into the company. They are the front line to the customer. This puts them at the top of the ladder in the decision making about the product.

Steve Jobs once said:

"The company does a great job, innovates and becomes a monopoly or close to it in some field, and then the quality of the product becomes less important. The product starts valuing the great salesmen, because they're the ones who can move the needle on revenues, not the product engineers and designers. So the salespeople end up running the company."

The salespeople are as far from the developers as possible in terms of implementation details.

Their only goal is to sell. They don't care about how the product is built, as long as it's working and they can sell it.

It's funny, but a lot of the salespeople have never seen the product, but still are able to sell it.

For them, investing in infrastructure is pointless. It's not something you can sell, and even if the developers or support have to operate it manually, but seamlessly for the customer, this can work just as well.

Of course, selling a well engineered and matured product is by far easier. It's much easier to make a demo and establish a POC when the product has the infrastructure for it. But the sales cycle combined out of so much more than just a good engineering, that a good salesperson can close a deal without it.

Although it seems to be the correct thing from a pure revenue point of view, it can make the company go bankrupt just as easily.

---

The catch and the solution

The reality is that once you have customers you're on a tight journey to deliver the product. When features are piling up and you got to get them done, stopping everything for infrastructure is a bad idea.

So, there isn't a right time for building a good infrastructure!

Ok... this seems like a dead end. Fortunately it's not.

It's all starts from the beginning. When you're building a new product the start is filled with uncertainty. So, investing in huge infrastructure seems wrong, but this is exactly the time for you to put the foundations.

Looking at infrastructure as something huge is the key point that lies at the core of the differences between the perspectives, but this is also the key for the solution.

Infrastructure doesn't have to be big. In fact it can be as minimal as possible providing only the requirement for building your MVP. But once you lay down those foundations it'll be easier for you down the road.

Let's look at an example: when you start coding for your first POC or customer, after you've been prototyping and doing all market research for a while, the easiest way is to write the straight forward code, manually test it and manually deploy it. But at that point, the time it'll take to write small automatic tests infrastructure, creating the basic logging and metric infrastructure as well as a simple CI/CD will be infinitely small compared to the time that it'll take you even a month into the future.

But this is not enough! Building the foundations is rather easy, but it takes a very small effort to ruin it. No matter what, you have to invest in building the infrastructure in every feature. It shouldn't be a one time effort, or something that is separated from the product. It should be part of your design and code. Once it is, all the above arguments will fade and there won't be any ground for different perspectives regarding this matter.

---

Final thoughts

Although there is no right answer when it comes to software development or building a product, you should never treat an infrastructure as something that you do only once. It's an on going effort as much as your product is. It can help you with your journey or become the stone in the shoe that everyone in your company suffers from. The choice is really that easy.