Scalability versus Feature

Learning from failures

Over the years I have tried to implement several systems that should – what I called it – revolutionize the perception on what software can do. Actually, I would have been pretty satisfied, if one of these systems would have grown out of its infant stages and so spread out to a wider audience.

In restrospect I now clearly see that – programming for one self – especially when having a broader perspective, may lead to software that is profoundly overengineered and so – caused by perfectionism and paranoia – is clearly violating the YAGNI principle.

As a consultant I rarely felt the urge to do something stunning, something exceptional, so – contrary to all statistics – all these projects were successful, but when I worked on private projects things started to grow over my head, squeezing out all motivation after a few weeks of work.

So I decided to create – even for myself – software, that is a little more boring, may bring some money and is just a little ahead of its time.

But the visions did not stop and the (revolutionary) ideas started again, which forced me to reevaluate, rethinking why my projects failed. Investing lots of work and time into a project that will surely have a dead end is not an option for me anymore.

One mistake, and that is the conclusion for this blog entry, is that I thought, to make something really “great” a lot of scalability dimensions need to be built in a software product, for example:

  • Number of users
  • Number of computers
  • Number of processors
  • Amount of data to process
  • Amount of memory / storage available
  • Automatic complexity reduction through caching
  • Automatic dependency tracking

And the list goes on and on. I knew that each of these scalability dimensions are possible, and sadly I even know how they need to be implemented.

Conclusion

A scalability dimension just does not compare to a feature. Whenever a new scalability dimension is considered, all software components need to be involved to keep up to the expectations.

A scalability dimension is so hard to integrate in an existing system that you need to define them in upfront before starting a software project. Try to keep the list as small as possible. It may never be possible to add one more later on. One single dimension may increase the software complexity of your project so much that it may fail.

A scalability dimension is not a feature, it is one of strongest cross cutting concerns that software developers must treat respectfully.

yours
armin

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

not adding another layer of abstraction

... or: why write a complex generic solution which can be reused for different scenarios, just in case, when you can write the same thing from scratch half sleeping in case you really need it :)