Expositus Procuratio

RapidCMDB, an open source federated CMDB solution for the enterprise

mberkay — Wed, 09 Apr 2008 21:29:00 GMT

About a month ago, I had posted about our open source plans. It took longer than anticipated but I'm happy to announce that we've launched the development site and released RapidCMDB as an open source project.

So what can be done to tackle the network monitoring challenges?

mberkay — Thu, 28 Feb 2008 14:22:00 GMT

In the Network monitoring is a commodity mythI argued that network monitoring is far from being a commodity and on the contrary needs innovation to cope with the increasing complexity.

As cote mentioned in the comments of that post, there has been some fresh blood in the IT management industry. Several open source companies/projects are tackling the monitoring problem, which is a good thing, yet I feel we're still missing some pieces. AFAIK, most of the monitoring solutions seem to be following existing paradigms :

monitoring the devices (nodes) through SNMP agent
synthetic transcations to determine the status of services running on nodes

The understanding of the network topology is missing in both paradigms. In other words, nodes are what's being monitored. Not the network. The network topology (except layer 3) is largely unknown. This limits the effectiveness of the monitoring. Monitoring tools (or rather functionality offered by the tools) can be categorized broadly as the following:

Polling the devices: Most common approach in IP networks. Most IP networking devices have an SNMP agent that supports at least MIBII so basic availability and performance information can be obtained. For more detailed information however, use proprietary MIBs is needed. Many IT management guys spent long hours trying to understand these MIBs, which data is where, compile them to be used by their monitoring tools, etc.
Listening for exceptions: Not every network device has an available agent that can be polled, especially in lower layers below IP. And when available, ability to listen for information is useful as it can be more immediate. In IP networks, these are typically SNMP traps or syslog events. In others, there are often element managers that convey messages. Again, IT management folks spent countless, often frustrating hours, trying to make sense of the traps, syslog events, etc. normalizing them, translate them into human language, identifying what is important and what's not etc.
Listening to the pipes: It is possible to learn a lot by listening to what goes on the network. Flow tools (Netflow and its kin cFlow, J-Flow, netstream, sflow, etc.) generate end to end traffic statistics based on flow of data through the network device that support it. Another approach seems to be analyzing the traffic going through a device using a span port. Although it seems this method is popular to analyze application traffic. I don't have a lot of personal experience with these tools so I'll leave it to others to explain it better or correct me. From what I see these tools often require hardware distributed throughout the network to get full visibility which may be a hurdle for adoption.

IMHO, all of the approaches I've tried to summarize above have some shortcomings. As far as I can see, the situation may improve in two ways:

someone may come up with a new technology, a clever way to monitor the network and identifytthe problems, may be discover & represent the network etc. IMO, this can only happen if some of the investment and attention in tools that target “business users” with sexy, shiny UIs flow back to the muck. When the payoff is so low (who wants to tackle a “commodity” problem?) significant investment is not likely.
The power of the community is harvested to solve tedious problems once and share rather than each user struggling to solve the same problems over and over independently. There are already some examples of this splunk is attempting to create a repository of log events and what they mean. ZipTie open source project is working on solving device configuration through collaboration of vendors and customers (how come they are not a member?)

There is a lot more that can be done in the monitoring realm, if we can manage to setup the right collaboration platform (commercially, legally as well as technically) to facilitate sharing, which is sorely lacking in IT management for whatever the reasons may be.

From what I can see, ZipTie model is particularly interesting and suitable. Ability to collaborate and share is potentially a major competitive advantage for open source projects. I believe there are opportunities here for collaboration among open source projects/companies and their users/customers.

For example, in the case of discovery and representation of the network topology, how to get the topology data out of vast number of different type of devices is can be shared. If a common model can be defined to represent the topology, adapters to populate the model for each device can be developed.

In case of trap and event log processing, the knowhow of what each trap may mean, what the varbinds are can be shared. And again if a commong model can be defined to represent the traps/events, adapters to convert the traps into the common model can be developed.

I think these activities are naturally conducive to be solved through collaboration, and the life in the trenches would improve significantly if we were tackling them together instead of drowning in them alone.

Network monitoring is a commodity myth

mberkay — Sun, 24 Feb 2008 20:04:11 GMT

There is a persistent meme in the industry that states (network) monitoring is now a commodity. This meme is so persistent that it seems it's no longer even disputed. There are lots of different monitoring tools, many of them are open source and/or free, and they've been around for a long time, hence the thinking goes, monitoring is now a commodity.

It is quite puzzling to me how terribly wrong this meme is. How can we be so wrong? IMHO, network monitoring is not a commodity. Far from it. Network monitoring is still largely an unsolved problem. The tools we have to monitor the "network" are largely inadequate.

Network is a complex beast, and level of complexity is increasing by leaps and bounds as well as the criticality of it.. It has layers over layers and only limited set of people understand it all. Our monitoring of the network is mostly limited to what we understand the most: the nodes in the network. We don't really monitor the network itself which is a complex distributed application running on these nodes.

This reminds me a famous Nasreddin Hodja folk tail where he looses his ring in the basement of his house but people find him looking for it outside, on the road. When asked why he is looking for it outside, he says that the basement is too dark, and he can't see anything there.

It seems to me that somewhat like Hodja, we're monitoring the nodes in the network since we can, and not monitoring the network because, well, we can't. The problem is largely related to instrumentation. More or less standard instrumentation SNMB MIBII, etc.) to monitor the status of a device and its ports & interfaces has been available for quite some time but very little instrumentation is available to determine the network topology, and whatever is available is not standard.

Without the understanding the network topology and the role of the nodes in that topology, the value of monitoring of the nodes is quite limited. We end up collecting a lot of information that does not necessarily helps us determine what's wrong. This is also largely the cause of the disconnect between the users and IT organizations when talking about availability reporting. IT reports on availability of the nodes in the network which does not necessarily equate to the availability of the services that run on the network.

As an alternative when the services are monitored directly, we may be able to determine whether the service is up or down, but cannot determine what the cause of problem may be by looking at the monitoring tools.

The focus in IT management market has moved up to stack so to speak to “business level” where tools which shiny user interfaces that provide “executive dashboards” are all the rage. IT departments have hell of a time justfying an investment in better monitoring tools but have easier time investing in tools that address the higher level. Ironically, the higher level tools rely on the information provided by the lower level tools such as the monitoring tools hence without solving the monitoring problem, it's not feasible to have meaningful dashboards.

Beating up the IT organizations has become such a popular sport that no one seems to listen to what they have to say. As a result, IT management discussions increasingly risk loosing touch with reality. I confess to be jealous of cote's blog biline “one foot in the muck, the other in the utopia” as I believe is the right philosopy to solve any problem worth solving. Network monitoring is in desparate need of innovation and attention, but that is not likely to happen if we start paying more attention to what the people in the muck are saying and kill this false meme of monitoring is a commodity

I don't have the answer to how to solve this problem, but I think the community may well have. In the next post, I'll lay out not what I think may be an answer but what I hope may trigger some thoughts on what can be done to tackle the problem of “network” monitoring.

Road to open source

mberkay — Mon, 11 Feb 2008 19:14:46 GMT

I'm excited to announce that iFountain has decided to go down the road not so less traveled and embrace the open source model. Being open has always been in our DNA, and we've agreed that embracing the open source model is the best manifestation of this objective.

As part of being open and transparent, I'll try to journal the road to open source for us. We'll try to get this done right. No doubt that it would not be difficult to post all the source code somewhere, stick on the GPL license, make a press release and declare the mission accomplished. But outcome of that alone would be no good for anyone. It takes more than declaring a license to turn closed source products into useful open source projects. Based on lessons learned from other projects, here is what we intend to do:

Open development infrastructure

It's clear that we need to implement an infrastructure to support open development where (most) developers are not in the same room. We need to make better use of the available tools to facilitate productive work and establish the methodology. Atlassian products JIRA and Confluence seem to be the defacto development/collaboration tools in many of the open source components we use, hence we intend to use them as well. We will also move from CVS to Subversion as the version control solution as it alleviates some of the problems we currently have.

I'm bummed that JiraStudio is not yet available (in closed beta) as I think it'll be a great solution for any distributed team. We'd love to use the JiraStudio instead of having to implement and integrate these tools ourselves. We'll keep and eye on it anyway. One can always hope

Solution architecture

As we transition our existing products into open source, we intend to rearchitect them, using a more loosely coupled component based approach that will facilitate plug-ins as much as possible. I think this is essential to make the code really “available” to people. Decoupling different components, will make it possible for others to use the components in their solution if they choose to. It should not be necessary for someone to understand the entire code base to develop additional functionality or modify/enhance parts of it.

Since our projects will inherently have significant integration requirements, plug-in structure should make it easier for others to create integration modules to serve their needs without having to master the entire solution.

Product design and messaging

Open source is an ecosystem. I think it is wise to take the existing landscape into account and address an area that ecosystem is in need of, rather than piling on an area where there are already existing open source solutions. I believe our solutions will fill a gap in IT management field and has very little, if any, overlap with existing projects and companies in the open source field. Hence, I hope that we will find significant synergies with other projects.

To emphasize this we will shed parts of the products where others do a better job (and use what's available instead) and focus on the parts we believe we're filling a gap. As can be imagined, this is not an easy feat, and we'll work on this heavily next couple of months. As always, any feedback will be more than welcome, to guide us to the right pastures

So what's the next? We will launch the development site (ifountain.org) with Jira and Subversion and release first version of “RapidCMDB”, an inherently fedarated CMDB solution for the enterprise by next week. Stay tuned to get more details on RapidCMDB ...

More investment flows into open source IT management companies

mberkay — Wed, 23 Jan 2008 17:31:39 GMT

OMC member Zenoss has just announced $11 million new investment so congragulations is in order.

Tarus of Opennms has a blog post where he somewhat criticizes the VC financing of Zenoss, Hyperic and Groundwork. The post reads as an attack against these open source competitors (though Tarus does say that it isn't), and I felt compelled to comment on some of the memes in this post as I saw them elsewhere before. To get it out of the way, I don't have any kind of relationship business or personal with Hyperic, Zenoss, Groundwork or OpenNMS.

"Since we are trying to position OpenNMS against products like Tivoli and OpenView, they really aren’t who we are trying to compete against, and since we don’t have millions of dollars in investment it could come across as if I was a big whiner, boo-hooing about our poverty."

Like it or not, OpenNMS is competing against other open source IT management players. Customers decide who the competition is and the "Little 3" are competition for OpenNMS as much as Openview or Tivoli, may be more so. Tarus's statement seems to be more of a "desire" than fact. Implication that they are not open source enough, because they have some proprietary components and OpenNMS is the only "true open source" project is all over the post. Not the first time I saw this type of mud slinging.

Companies have to find a way to make money, and nothing wrong with having an enterprise version. It is in the interest of their community for them to be successful so that they can continue to develop the open source solution. Their open source versions stand on their own and welcome contribution to the field. If people decide they are not, they simply won't use them.

What is open source now will stay open source by definition and even if they stopped developing the open source parts, someone else can always continue. From what I understand this is sort of what happened with openNMS (someone else continuing part)?

"they accepted some VC funding, developed some code, and released it on Sourceforge. They were promptly sued by their board to the tune of US$50 million for releasing trade secrets."

So some VCs did some really bad things. All VCs are evil anyway, hence companies (Hyperic, etc.) took VC money are tainted? Tarus would only take money from investors who "shared his vision" but they took money from the dark lord, so they must have sold their soul. The fact is, we don't know what conditions they took the money, or what they plan to do with it. This is pure FUD. Not cool.

"So how will such a project respond to a user who implements the “enterprise” functionality as open source? I don’t think they could accept the code and commit it while at the same time satisfy shareholders who expect software revenue. They’d have to make up some excuse about why they couldn’t commit it, and unless the independent community was strong enough to fork the code, nothing would change."

This is ill informed speculation at best. The model for contributions is established in several projects, including MySql. A recent example of it is between Google and MySql, where Google donated significant chunks of code to MySql. If someone wants to contribute they don't have to make any excuses. They just have to sit down and make a deal.

"I have to wonder. If acquisition is the main exit for these companies, why would someone like IBM or Sun want to pay US$100 million or more when they could pick up, I dunno, something like OpenNMS for half that? (grin)"

The answer is simple: same reason why Sun acquired MySql and not PostgreSQL. The one with the biggest community and largest customer base wins. If Zenoss or Hyperic manage to get 10 times as many customers as OpenNMS, IBM or someone else may be willing to pay many times more to acquire them as Sun did with MySQL. Also note that, MySql "owns" its products just like Zenoss and Hyperic. AFAIK no one owns OpenNMS so it can't be acquired.

We need more business models that make open source viable, not less. More ways to attract investment, not less. Throwing mud at other business models is simply not productive, and not a good way to compete.

open vs open source management. What does open mean in IT management?

mberkay — Tue, 01 Jan 2008 17:28:54 GMT

What is Open Management Consortium?

The about page states the primary objective as "Create awareness of open source management tools in the market", so the focus is open source mangement tools. Fair enough.

But open management need not mean open source management. There is a lot more to openness than seeing the source code. In my experience ability to see the code is not even highly sought after by the customers ( I work with). I think the term "open source" has come to embody a lot of things that we've been longing for: interoperability, integration, transparency which are also somewhat mentioned in the objectives. I think the "Open Management" as a term is a better embodiment of these principles.

This is not just a play with words. The nuance is important. There are already calls for the large management vendors (loosely referred as Big 4) to open source their products. I don't this approach is neither realistic nor productive. I think we ought to demand them to be more "open", and this does not mean they have to open source their products. There are many other steps that are much less controversial, yet may even be more useful for the industry. IT management vendors (as most software vendors) are typically very "closed" organizations. What do I mean by that?

How many of you have signed an NDA with a vendor? It's pretty much demanded by every company I dealt with so far that restricts what I can share in public. NDAs are used routinely in the industry. You want to have access to software, you have to sign an NDA. This may sound trivial but I think it illustrates the attitudes and the problem. Tendency is guard information, not share it.

Can you go to the websites of the Big 4 (or to any of the other large management tools vendors) , download the product you're interested and give it a spin to see whether it meets your needs? Overwhelming majority of the vendors do not even have evaluation copies available. Transparency. Do you feel like you can participate in the direction the product? Can you even see where the product is heading? And pricing. Can you tell how much the price of the product is without putting a gun on the account manager's head?

I think "open source" products are on the rise, not necessarily because we can see the code (most of us can't care less) or we can contribute to the development (most of us are not devleopers) but if a product is open source, it is assumed to be "open". We can take the open source product,s evaluate/use as long as we want, learn from experiences of the others in the community, and earn our say on where the product is heading, well, not always, but may be most of the time.

In short, I believe we should value emphasize the open in open source more.

I think one reason we are relying on open source as the litmus test for openness is that the other criteria for open source is easier to establish than criteria for being open. Not having an established way to measure openness, it is easy to descend into subjective "I'm more open" pissing match. So I wonder whether it would be possible to come with the criteria to measure how open a company is. I've already hinted some of my criteria, I'm sure there are other better ones.

1. Access to the software. There is no reason why potential customers and partners should not be able to download and evaluate the software, without being harrassed by sales people first.

2. Published APIs and developer programs. Almost all companies claim some sort partner program, but few has active ones, and there are lots of barriers. No reason why the API and documentation should not be available to any interested party, along with the software. The process to become a partner or use the APIs should be simple and transparent. Software vendors should take a page from the book of companies like Google, Yahoo and Amazon in creating APIs and developer ecosystems. The process to use APIs should be straight forward both from technical and commercial perspectives. This is essential for integration and interoperability.

3. No NDAs to silence customers and partners. Let people share their opinions and knowledge as they like. In all these years in the industry I've signed many NDAs, I don't think I ever knew anything worth protecting. This approach is simply poisonous to sharing and collaboration.

4. Available communication channels for the community to participate. Having couple of product managers talk to couple of important customers simply won't do. To ensure the products stay relevant and useful, the best option is to let the community have a stong voice and provide guidance. This is probably harder to quantify than others.

5. Transparent pricing. The game of hiding the prices, having very high list prices and offering big discounts is getting old. Why not publish the prices?

If the software vendors did all of the above but kept the source closed, I'd be more than happy. I'd wager that most companies would not score all that well using these criteria.

What do you think? Are these reasonable? Other criteria to quantify openness of a company?

ITIL and ITSM still matter in a world with external providers

mberkay — Mon, 31 Dec 2007 14:40:38 GMT

John Willis asks whether ITIL still matters in the world of Amazon and Google (what I once referred as "[best in class infrastructure providers|http://www.ifountain.com/blog/Arebestinclassinfrastructureprovidersaviablealternativeforcompanies]"). ITIL skepticism is not new; there has been skeptics since the beginning for variety of reasons; some more valid than others. John is raising the issue from a different perspective. He stipulates that ITIL may not be required if majority of the services are provided by external giant service providers like a utility.

From my perspective, the availability of these services from the likes of Google/Amazon make ITIL and ITSM more relevant and necessary in the enterprise, not less. One of the core ideas of ITIL/ITSM is to have a service perspective and managing the dependencies of the services to the infrastructure used to provide these services. The fact that some of the infrastructure components are provided by external providers who supposedly have great availability numbers does not change the fact that enterprises still have to manage the "service".

As I stated in a previous post the question we should be asking is how we can you end to end management of a service when the infrastructure for the service relies on combination of multiple internal and external service providers.

ITIL/ITSM offers some guidelines on how to cope this complex world. How should the enterprises troubleshoot problems? What should the service desk processes be like? Business/end users have never cared much about the availability of the servers, they care about the service. The services as perceived by the users are rarely provided in their entirety by a single provider. Most of the mission critical services have multiple components provided by different internal and external entities. What should be the operational processes to manage these services?

A typical scenario that exposes the cleavage between different silos in the enterprise is the "blame the network syndrome" where users complain about the performance of an application, and every group (silo) blames another and the network group gets stuck with proving their innocence. How do you "convince" all parties involved -the connectivity providers (LAN/WAN/Security, etc), application providers, platform (server) providers (internal or external) - to cooperate in order to resolve problems quickly? This has always been difficult, and rise of giant service providers don't alleviate the pain. Processes are still needed, guidelines are still needed, learning from the experiences of other still needed.

This is not to deny the significance of the change in the field . No doubt the game is changing as stated by John, but the implications of these changes are not so apparent. The rise of service providers that promise 99.99% availability may mean enterprises will more and more use the services provided by these external providers, instead of keeping them internal. If that's the case, enterprises will need to learn how to manage services that are not under their direct control. It may also mean that if they do keep them internal for whatever reason, management of these services can no longer be an afterthought as it often has been.

The game is changing and we must figure out how to adapt. Unfortunately, enterprise (IT management) users are not out on the web sharing their thoughts with each other in mass. As the web 2.0 culture infiltrates the enterprise, who knows may be the enterprise folks come out to play and we can come up with an ITIL that is developed like an open source application, out in the open with participation of hundreds. Who knows, may be OMG will be the catalyzer for wider discussions, once can only hope ...