Software Architecture Matters, Especially For Financial Projects

People sometimes ask me about my thoughts on various Fintech projects they come across. To provide an answer, I go through the features of each project, thinking about their relative complexity. I then try to find out how much it would cost the company to implement those features.The answer sometimes surprises me.

One such project is the now defunct Planswell. The company raised almost $14 million and hired over 60 people to build an automatic financial planning app. Considering how much they spent, I felt that they should have had more to show for it than what I saw when I signed up. While Planswell officially failed because of a sexual harassment suit, I have to think their lack of progress on the product made the decision to pull the plug easier for their financiers.

So what tends to go wrong with such projects? Having completed a fair number of projects, I feel I have a good sense as to why many of them don’t succeed. There are a number of reasons why financial projects stall or become failures, but today, I want to concentrate on just one: the lack of consideration for software architecture.

Software architecture is about how software developers organize code. Well architected code is easier to understand, more flexible to changes, and contains fewer bugs. But despite the benefits, I find that most non-technical clients tend to de-emphasize architecture. There are several reasons for this.

First, it takes time to craft good architecture. Firms are normally under tight deadlines to deliver a set of features, so they instruct programmers to ignore architectural considerations to move faster on immediate goals. Firms sometimes promise to devote time to fix the architecture later on, but they often fail to deliver on that promise when they set new goals to meet.

Second, it’s hard to measure the benefits of good architecture. It would be great if a firm could set up an experiment where one group of programmers devotes time to improving architecture, and the other group does not. Alas, very few firms would be able to justify spending money on such an expensive experiment.

Third, crafting good architecture is an art, not a science, and few developers possess the requisite skills. A firm may grant a programmer’s request to re-architect some code, and the programmer may try his/her best, but there’s no guarantee that the re-architected code will have better architecture than it did before. In fact, it’s quite easy for a programmer to “over-engineer” a piece of code and make the architecture worse.

Because of these reasons, even firms that initially had the inclination to invest in better architecture may stop investing in it later on, leading a software project down the path to poorer and poorer architecture. Unfortunately, this neglect of architecture eventually bites back with serious consequences in the form of lack of progress. Let me explain how that happens.

A financial software project generally involves a number of complex numerical algorithms. Let’s take the hypothetical example of a risk management system. Such a system may include the following steps, organized into separate modules:

• Importing data from databases
• Transforming data into forms that are more friendly to statistical analysis
• Cleaning and filtering out undesirable data
• Identifying and quantifying risk drivers
• Calculating potential loss estimates based on the risk drivers

A well architected system would try to modularize each step, so that each step is as decoupled from the others as possible. This allows a programmer to work on each module in isolation, without needing to understand every detail about every other module.

A poorly architected system, on the other hand, is built such that each module is tightly coupled with every other module. For example, a small change in the data import module may cause new errors in the data cleaning module, or a change in the calculation module may necessitate algorithmic changes in the data transformation module.

A poorly architected system therefore places much heavier cognitive burdens on the programmer. Not only would each module be hard to understand on its own, but the programmer would also need to understand the details of every other module in order to make progress. This dramatically increases the length of onboarding time before a new employee becomes productive, and it also shrinks the labour pool of developers who can contribute to the project in the first place.

But that’s not all. As even small changes in one part of the code base can have drastic effects elsewhere, collaboration among developers becomes that much harder. One developer may make a change in the risk quantification module, thereby changing the requirements placed on the data transformation module. But another developer working on the data transformation module may, at the same time, work on a task that conflicts with the new requirement.

These drawbacks from poor architecture become a bigger and bigger issue as the code base grows larger, until one day, progress on the project virtually stops. At this point, throwing more developers on the project doesn't help, as new developers struggle with onboarding and maintaining communications. This is not just a hypothetical scenario. The charts below illustrate this point from one real case study.

The moral of this story is that a firm should never neglect software architecture if it wants to create a project of any meaningful complexity. This means, unfortunately, that a firm should set aside more time and resources than it deems absolutely necessary to reach its immediate goals. However, there’s a silver lining in this story too.

Smaller firms may feel intimidated by the fact that there are big companies investing heavily in similar software projects as they’re undertaking. But as we’ve seen, pouring more money into a project doesn’t guarantee success if the architecture is poor. Given the propensity of big firms to fall into hiring the wrong people and underinvesting in architecture, there are good reasons to believe that small firms, if they play their hands correctly, can beat giants. After all, Google wasn’t the best funded search engine in the early days of the internet.