Metricon Slides, and Viewed as PR

The slides from MetriCon 2.0 are all posted now. Many good talks in there; I'll probably comment on some more of them later.

One of the most interesting aspects was to see those with business experience try to explain to those who said "Just tell me what to count!" that counting isn't enough. If you want business managers and executives and board to pay attention, you need to say what your counts mean.

Chatting with attendees, it became clear some of them interpreted that latter as a call to make up numbers to match whatever you wanted to sell to management. Far from it. The point is to abstract your numbers and to describe them in terms of what they mean to the business.

Abstraction may be part of the problem.  For fans of Myers-Briggs, we seem to have quite a few ISTJs (security counters) being told they need to communicate with ENFPs (business executives). You don't even have to know Myers-Briggs to see that those types don't match at all. (Yes, I know I'm oversimplifying; any one of either of these groups might be of either of those types or another, but I think you'll agree that they'll each tend more towards the stated types.) In particular, people who work on details of security tend to have detail-oriented personalities (S). Abstraction to them can sound like fudging, and presenting with related content can sound like lying (see above).

So let me try an analogy instead. Suppose the VP of Marketing shows up at a board meeting and reports that his PR guy put out 5 press releases last month. So what?

OK, how about 5 press releases and 3 of them got picked up in a total of 500 stories worldwide. That's nice, but so what?

OK, and two new customers signed up who said they heard about the company's product through one of those stories. Now we're getting somewhere!

It would also be good to know how those customers compare in dollars they spent and number vs. others that signed up for other reasons, and how the PR budget compares to the income generated by those sales, among many other interesting questions. But this is enough to illustrate.

Now suppose the CIO shows up and says his security team fixed 5 vulnerabilities last month. So what?

How did fixing those vulnerabilities affect customers? How many more were not found, or not fixed? How does the security budget spent compare to related present or future sales or savings? Is finding and fixing vulnerabilities the best use of security money?

How many security teams or CIOs can answer questions like that? How many present to executives or boards including such meaning?

-jsq

Comments

[Alex]

One? :)

Did the reduction of vulnerabilities reduce risk? If not, I have a hard time seeing how the CIO in the example can express value for that effort.

[Iang]

huh, good points. I recall several classes in b-school about how to make the metrics mean something, be measurable, and simple all at the same time. They were boring classes, I guess I missed their importance.

OTOH, in the techie industry, making up the metrics is a time-honoured tradition. Is this limited to IT, I wonder?

I never connected the two together, but that's a good observation.

[John S. Quarterman]

Another thing I heard was several people calling for security people to get MBAs. From Iang's comment, it sounds like business people find this topic boring, too, and while some technical people think explaining numbers means making them up, some business people assume technical people just make up the numbers anyway.

What we have here is a failure to communicate.

And meanwhile the black hats are winning.

[Pete]

Being the one who said "just tell me what to count" it is clear to me that my intentions were misunderstood by you and (perhaps many) others at the conference. I don't understand why someone at a metrics conference wanting specifics about metrics (and speaking colloquially) could be seen as only caring about numbers. It has never been true. Numbers are a means to an end that is more objective and useful than the current state of affairs. Speaking as one with years of business experience, I have heard (and recommended) the "talk like the business talks" line for years and it is almost always true and meaningless at the same time when expressed as an excuse, for example, not to provide useful information to the executives themselves. And that was essentially what the panelist was saying. Did you happen to notice that we were at a metrics conference and there was not a single specific metric mentioned there that was strategic and business-like?

Pete

[John S. Quarterman]

Hi Pete,

If you'd been the only one who expressed such a sentiment, I'd go with your interpretation as the most useful one. However, it became clear in discussions with people after the break that there really is a disconnect between those who just want to measure and those who want information executives can use.

"Did you happen to notice that we were at a metrics conference and there was not a single specific metric mentioned there that was strategic and business-like?"

Why yes, yes I did. I think we're in agreement that that's the problem.

However, I don't see CEOs dictating a list of publications to use for PR and then spelling out exactly how to phrase the PR to get them to pick it up. In such a case, why bother having a VP of Marketing? Yet this seems to be the situation for security.

However, as I mentioned after their talk, I think at least one of the presenters was really close to strategic and business-like metrics.

More on that in a blog post.

-jsq