Expositus Procuratio

RE: So what can be done to tackle the network monitoring challenges?

stevecarl — Thu, 27 Mar 2008 03:15:25 GMT

My main problems around discovery as they pertain to this are scale and identity management.

90/10 works on relatively small networks, but as the enterprise gets larger the numbers start to need to verge closer to 100%. For example, with 20,000 end nodes (not even counting all the plumbing it takes to hook together that much stuff) 10% is 2000 systems, and that is going to take someone a very long time to deal with manually.

Now, add in security policy: The larger the enterprise, the more likely the security policy requires passwords on end nodes and infrastructure are fluid enough to avoid compromise. Say they change every 90 days in our supposed 20,000 end node network. It is highly unlikely one can effectively deal with 2000 systems every 90 days, and even worse, the discovery mechanism itself has to be re-educated every 90 days to be able to keep all the maps up to date. The right thing to do of course would be to have the identity management tool tap the discovery tool on the shoulder whenever critical passwords change that discovery needs, but as far as I am aware, no one has done that.

RE: So what can be done to tackle the network monitoring challenges?

ldimeglio — Wed, 26 Mar 2008 15:00:11 GMT

You're right, the overall picture is much more complex. All vendors, including my company, struggle with understanding the full landscape at any given customer. If only everyone implemented their network the same way, things would be much easier! We're watching the OMC carefully and looking forward to the solutions that you guys come up with that everyone can benefit from. Thanks for the work in these areas.

RE: So what can be done to tackle the network monitoring challenges?

mberkay — Wed, 26 Mar 2008 10:42:58 GMT

Hi Louis,

Welcome to the discussion. In my experience, layer 2 topology discovery has been more problematic. Most solutions that I'm aware of (big 4) etc. can't do it right, require a lot of custom work etc. bridge mib does not always have the information, CDP is just for Cisco and increasingly turned off thanks to rising dominance of "security" contingent in the enterprise. topology information is often in proprietary mibs, etc. and heuristics need to be used to process the data and infer the topology.

But layer 2 is only part of the story. Similarly, you'd need routing (BGP, OSPF, etc.) topology, vpn topology, etc. and the dependencies among these layers.

When people talk about silos, they typically refer to application, systems, network etc. in broad terms. But without this topology, we essentially have several silos within the network. Domains where the monitoring systems provide information that cannot be easily related to each other at best, false and misleading at worst.

RE: So what can be done to tackle the network monitoring challenges?

ldimeglio — Wed, 26 Mar 2008 03:50:37 GMT

Sometimes I think that you have to look at the 90/10 rule. In the case of discovering network topology, if you can cover discovery of connected devices through the BRIDGE-MIB and perhaps supplement that with CDP and/or LLDP you should be able to discover the Layer 2 topology of a network in most cases. It's the method that my company uses in our product and in almost all of our customer environments gets up to 90% or better accuracy in the discovery. The last 5-10% can then be corrected/supplemented by hand and there you have it.

RE: open vs open source management. What does open mean in IT management?

ldimeglio — Wed, 26 Mar 2008 03:46:07 GMT

I know I'm a little late to the commenting game, but I wanted to say that I think that you make some great points. The network and systems management appliance company (that shall remain nameless) that I work for embraces much of what you're talking about by not creating our own agents and instead supporting everyone elses, supporting as many standards as we can for data exchange (XML/SOAP, SNMP, ODBC, sqlnet, etc) and sharing just about everything with our customers (including in some cases the source code). I will say though, that in a for profit company, when you are sharing something as sensitive as source code, you better protect it with an NDA. I never understood companies that required NDAs for public interface information (eg Canon wanting an NDA to release their MIBs), but in some cases you DO have to protect what you've got!

RE: So what can be done to tackle the network monitoring challenges?

stevecarl — Thu, 28 Feb 2008 21:54:34 GMT

One of the things I keep thinking about as it pertains to issues like this is "Has this problem been solved before?". Related to that, I think "The more things change, the more they stay the same"

Ehh?

We have seen problems like this in the past, and from where I sit it seems to me that when an idea occurs, it races out the door and no one sits back and thinks about the problems it might have... or might cause... long term.

it could be one of several things, but as it related to this I think "Gee, when device / protocol X awere designed did they not think about how it might be instrumented up front? I relate this (beacuse I am high mileage) to the way IBM used to roll out new stuff. A new device or a new network subprotocol would come with its measurement capabilities built right in. A new device controller would have a fully architect'ed control blocks book available, and would already be tied into the newest version of the monitoring (EREP) and measurement (SMF) tool.

3rd party vendors made money making these tools easier to use, but the basic infrastructure was thought out and designed in.

When I was a subcontractor at IBM, I used to hear hallway talk basically laughing at the idea that anyone would leave something like VTAM behind for TCP/IP. VTAM was stable, measured, architect'ed, stateful, and manageable. TCP/IP "didn't even guarantee packets would be delivered!".

VTAM got slaughtered.

Yet here we are, trying to figure out how to make devices and protocols monitorable and manageable.

Of course, the way that IBM was able to do what they did was simple in concept. They controlled the standard. They controlled the releases. They made the devices. The reports took freaking experts to read, decode, and understand. Shoot, you had to be unbelievably careful about making sure you only enabled just the things you needed to monitor, or your whole computer / device / line was tied up with measuring itself. But IBM was incent'ed ed to do this. To not do it meant that they, the single throat to choke, would not be able to figure out what was wrong when things went wrong. Customers would lynch them for downtime / outages / poor performance. PC software had not yet lowered the expectations of the customer as to what they could expect from computers.

So, we have been here before. People are starting to realize the real costs of downtime, and are raising their expectations again. But the stack is not longer made by one vendor. There is not one throat to choke. It is easy to point the finger across the way and deflect blame.

For this to work, the incentive has to come back around so that everyone is on the line. Everyone can get choked if we do not all work together.

I don't think anyone wants to go back to the monolithic days of yore, except for maybe those that yearn for the says when they could just beat up one vendor. That had its limits too of course: as long as they were the only game in town, they could just take their lumps.

As long as their is no incentive to design / architect this stuff in from the get go, we are going to stay in the Tower of Babel.

RE: Network monitoring is a commodity myth

mberkay — Wed, 27 Feb 2008 22:24:32 GMT

Steve, "Network Management by Airline Magazine" I'll have to remember to use this one! I gave up hope in standards by comity a while ago. I'm no longer sure it's even realistic, given how fast technology changes and how little real pressure is on vendors. I still hold some hope for grassroots driven standards, it management version of rest vs soap perhaps ..

Cote, I think the big 4 recognized the garbage in garbage out problem to a degree, as a result startups with application discovery technology were acquired by the big players quickly. Now I don't know how effective these solutions are as they are a bit of a black box (often literally), but there does not seem to be anything similar in the networking layer. My guess is that application layer is easier sell than networking. In the mean time, networking layer becomes more and more crucial everyday. You're right, monitoring needs a marketing makeup, a new name, a former model spokesperson may do ?

RE: Network monitoring is a commodity myth

cote — Wed, 27 Feb 2008 20:16:43 GMT

I'm glad you like the slogan.

As background, every time I talk with the high level, "dashboard" type people I ask them how they get all the fancy data to put on those displays. The more marketity people just talk about collecting it and then move on. The more technical people will give me that sizing up look, see that I'm OK, sigh, and then say something like, "yeah."

At that point I usually say, "yup, it's like the old adage: garbage in, garbage out." The technical people get this of course, and always express a bit of concern that they're dependent on NOT getting garbage from the lower levels. At the same time, as you say, the lower levels are being ignored (seemingly, at least), meaning they're more prone to become garbage.

This is why I like all the open source IT management people running around now: they're actually paying attention to and innovating at the lower levels. Hopefully, the Big4 and friends will feel threatened and jump back on that as well. But, I suspect that lot will always be focused on things with more fancy sounding words than "monitoring." Maybe they'll partner up.

I am pretty biased though as my roots in IT Management are monitoring, so I probably get too wistful about things like MIBs for my own good.

RE: Network monitoring is a commodity myth

stevecarl — Mon, 25 Feb 2008 19:27:05 GMT

This so echoes so many things that have been bothering me lately. Thanks for drawing it into focus like that. I look forward to your next post.

I can not tell you how many times every day I get the "Why is this not done yet: this is all commodity now" kind of statements. "Network Management by Airline Magazine" (tm)

It would be fair to say that some things are easier and more mature now than they were. That I can monitor more things with less effort than I could before. That is a "Good Thing"(tm). But we are so far from plug and play, and increasing complexity and lack of Open standards is most definitely not helping.

RE: open vs open source management. What does open mean in IT management?

mberkay — Sat, 05 Jan 2008 10:39:36 GMT

Agreed! Well said. Many promises and initiatives have long been forgotten, so much that no one believes the vendors anymore, no? We've all turned into skeptics and rightly so.

To add to this, I also don't think the community/customers do not demand this strongly enough from the vendors. While working for a vendor I was surprised how shallow the inquiries related to integration were. Most often we just got asked "do you have an integration (adapter) with X?" or "do you have an API?" not much more went to it, so it is easy enough for the vendors to say they are "open".

This may be due to the fact that customers don't have the established criteria, vocabulary to ask the right questions. I may be wrong. "Open source" is one such criterion that is easy to ask so it has at least that advantage, but I think we need more granular definitions. As I work on integrations a lot, I had attempted to identify some seed questions to evaluate integrations in a blog post (http://www.ifountain.com/blog/Top8questionstoaskwhenevaluatingintegrationsolutions) I'm sure we can do much better than that collectively.

We should be able to more questions that will allow us to gauge the openness of the products, and better yet, should do this out in the public (domain rather than sales meetings) so that everyone can see the answers and scrutinize them. We need this conversation of openness to happen in the public domain (hence the need to do away with NDAs) to create awareness. If there is sufficient demand from potential customers, vendors would respond, otherwise it will not go anywhere (even if some people in the vendor organizations also push it, they also need backup.)