Everything has a cost

Too often we end up in a conflict while trying to explain how a minimally impacting function has no true business impact. If you are too detailed, no one wants to listen to you and you are wasting their time. If you are too high-level, they will challenge your evaluation because nothing costs nothing.

When we felt the cost

Everything that occurs on a computer has a cost. There is no secret sauce which allows computers to do work without utilizing physical resources and taking time.

Think back to the 1960s when time-sharing was first introduced or even 1995 when processors were single core and only about 33 MHz. The average cost of memory was about $100 per MB. In those days, every CPU cycle mattered and had a noticeable impact on the performance of the system. Now we have processors whose clock speed has stopped being a direct measure for processing power because of multi-core designs. Windows Server 2016 can support up to 12 TB of main memory!

Back in the day it was easy to communicate costs. Today it can be a challenge to explain the cost of implementing or enabling a feature.

Wide-necked bottles

I once had a difficult discussion with a technically minded business stakeholder. We were going to enable Transparent Data Encryption (TDE). I was asked, “what is the performance impact of enabling TDE?” This person knew enough about computing and encryption to realize that it is not free. I had a couple of responses available, off the top of my head.

  1. Microsoft officially declares a 3-5% performance degradation.
  2. None, the application won’t be affected.
  3. The benefit is worth the cost.

All three of those responses were true. #1 and #2 might seem to conflict but they do not.

A little background…

This system was new. It had only one year worth of production data and it was the prize jewel of the company. There was a lot riding on the success of the system and that translated into funding. When I designed the physical architecture I was allowed to buy hardware beyond anything the company had owned before. The result of this was that, with only one year worth of data, all the OLTP databases fit into memory (with SSDTs, as well) and there was CPU available for 5 years of 50% annual growth.

I knew all of this and understood exactly how wide our bottle necks were. Even though TDE was going to increase our CPU usage, I was talking to a business stakeholder. No matter how technically minded they were, they are concerned with noticeable impacts to the users and business processes. In this case, there was not going to be any impact at all. Weeks after this conversation, heavy load testing showed no measurable impact. The TDE impact fell into the small margin of performance fluctuations which occurred due to random user simulations. There was a minor up-tick in our server resource use.

When your wide-neck is really just a long-neck

In my example, I was intimately familiar with the system that I was talking about. This discussion becomes much more difficult when you are not familiar with the system. Maybe you are giving a presentation and making general comments about a feature, maybe you are a consultant who has only seen the system once so far, or maybe you manage too many servers and can’t be intimate with any one of them.

I attended Chris Bell‘s Hacking Expose at a SQL Saturday a year or two ago and he faced this same challenge. Without putting exact words into Chris’ mouth, he made a comment about enabling SSL on its performance impact. He was challenged by someone in the audience who fully understood the mechanics of SSL and disagreed with his comment about minimal impact. A short discussion occurred which temporarily disrupted the presentation. The vexing part is that both Chris and the attendee were correct. Neither presented any false statements but there was still a minor conflict.

I believe that these conflicts are partly brought about by differences in perspective. Cassy, the DBA, who works in a well-funded shop might not allow their servers to go above 60% CPU or always has 20% more memory than is required. Cassy might see a feature as non-impacting while Tiffany, who tunes queries daily to prevent the CPU from pegging, might see the same impact as completely unacceptable.

Do I have a solution?

No, I do not have a complete solution but I do have a couple of recommendations.

  • Draw pictures of your bottles. I find it easy to explain the non-impact if you show them that a few grooves in the bottle do not affect the through-put, if the bottle neck is smaller than the grooves.

Bottle-neck-compare

  • Speak about facts not impressions. If you do not have testing data from your system yet, rely upon Microsoft’s official documentation or credible sources who have published a performance analysis.
  • Separate impact to users / customers from impact the system resources. For example say, “this will have a 2-3% increase in CPU usage but zero impact to customers.”
  • Explain why there is no impact to users.
    • This only affects an asynchronous process which does not hold up the user.
    • We have 40% unused memory and this will only increase our memory usage by 5% leaving us 35% free.

What do you do?

Please let me know, in the comments section, how you tackle these discussions and achieve clarity without conflict.

 

Leave a Reply

%d bloggers like this: