LSB Reloaded

For probably 10 years or more I have been involved with the LSB  Working Group. When I initially got involved I was working on the application side of the world and in the ISV world we needed to answer the "do you support distribution X" question on a more or less frequent basis. Often the answer was "no" and sometimes the answer was "how big of a check are you willing to write?" At that time penetration of Linux into the commercial world, what we now call the "Enterprise market" was a fraction of what it is today. Enterprise distributions existed but were not as entrenched as they are today. Support of Linux for our application suite was  maybe 2 or 3 years old, at best. Wow, that's a long time ago, I am getting old..... well at least I am in good company with that problem. Anyway, from an ISV point of view the LSB at the time offered a great value proposition. Build once run on many distributions, use these tools for application checking, and there are also some build tools you can use. If practicable, this would of course drastically change our answer to the question, "do you support distribution X?" and would help our application with some customers. Thus, the idea of certifying against the LSB and simply stating that all LSB certified distributions was just the ticket. I had the support of Development management to pursue the goal. Putting the LSB tools to work on the application made the tools fall over relatively quickly. Dealing with symbols in hundreds of shared libraries with a large surface area of interfaces and analyzing dependencies and extracting the external dependencies is not necessarily something that can be considered easy. Tooling was eventually fixed and what was left was a list of interfaces that were not in the LSB. Therefore, the original idea was not tenable with the LSB release that was current at the time. This is probably in the LSB 3.0 time frame, I remember buying the LSB book that was published at the time.

Fast forward, I am still involved with the LSB Working Group, but now looking at the world from the distribution side of things. The LSB in principal still holds the same value proposition for ISVs than it did 10 years ago, and for those ISVs that use the LSB it works really well.

The world around us has changed tremendously and the LSB itself has grown significantly. The number of interfaces that are covered by LSB 5.0 is very large compared to the number of interfaces that was covered when I first got involved. At the same time the Enterprise distributions delivered by SUSE and RedHat have established themselves as the distributions to use in a commercial environment. With a large certified application catalog, which is extremely important to end customers it is difficult for other distributions to make a dent. The establishment of two primary vendors in the Enterprise world has led many ISVs to simply test on both distributions, call them supported and rarely consider other distributions. Taking a peak into the crystal ball as to why this may be the case produces some insights:

• There is a very large number of applications that has migrated from UNIX to Linux and thus it is not surprising that ISVs brought along their "tried and true" mental model. In the UNIX world ISVs used to build for Irix, True64, HPUX, AIX, and Solaris and thus, a new world where you build on one Linux distribution and test on two is a tremendous improvement and saves a chunk of money.
• The "tried and true" thinking stands in the way of the paradigm shift at the heart of the LSB value proposition; build on one distribution, test on an LSB reference implementation, and then support all LSB certified distributions. Although on the surface this is a money saver as well, such a change in approach makes people feel extremely uneasy. In a way this discomfort is understandable. Support costs are in general high and claiming support for a distribution that was not actually tested, only indirectly via reference platform, just sounds like a crazy idea. Having had many conversations about this topic over the last 10 years I have come to the conclusion that this is partially a case of support phobia, without looking at actual data, and partially there exists justifiable fear.
• Even if the "test against a reference platform" idea does not make the hair on the back of ISVs necks stand up, there is still cost associated with having machines with many Linux distributions available just in case something does go wrong and bug chasing is required. Thus, some of the savings the LSB provides get eaten up by indirect costs.

I could ponder this some more but that's not actually the topic, I do want to get to the reloaded LSB soonish, and I will, I promise.

Another issue for ISVs is of course that the LSB can never have enough libraries and interfaces defined. Basically the problem that I had way back when still exists despite us adding thousands of interfaces over the years. There is always this one missing interface for ISV X.

Therefore, as we enter the endgame for LSB 5.0 the next steps were not immediately obvious. There is always the option of, "continue what you are doing and hope for the best." Quite frankly this was from my point of view the worst possible direction. Because the LSB has so many interfaces and the target, Linux distributions, is extremely fast moving it is more than difficult for the Working Group to produce a specification that is not behind when it is released. For example, LSB 5.0 contains Qt4 while both Enterprise distributions that are expected to release this year will have Qt5 available. The Enterprise distributions are of course the slow moving part of the field. One potential solution to this problem would be to find more fingers for keyboards, but the contribution model is rather tricky and lets face it, "formal standards" work is not very sexy. Basically even with a better documented and easier way to contribute we would probably still get little attention from those needed fingers. In an effort to be more responsive the LSB could certainly dump a large number of interfaces from the specification, but this would increase the "cannot certify because interface Y is missing" problem again,. Therefore, going in the wrong direction. Over the years distributions have moved closer together in the surface area that the LSB covers today. The times were a particular distro releases its own compiler that happens to be incompatible with what everyone else releases are behind us. I claim, that today for the most part distributions are closer together than they have ever been before, in the areas that matter for applications.

During the LSB working group annual face to face meeting at the Linux Foundation Collaboration Summit we took a long look at all the various angles, those describe above and a few more, and came to the conclusion that getting LSB 5.0 out the door is top priority, that of course should not be a surprise and is not news worthy. Nor would this decision in and of itself get the world's most infrequent blogger, me, to write a blog. Getting LSB 5.0 out the door ASAP will allow the current crop of distributions to certify and provide a long transition period into a world where the LSB as we know it today will no longer exist. The leading Enterprise distributions are both set to releases in 2014 and at the current pace new releases can probably be expected in 3 to 4 years. This constitutes the transition period into a world without a formal LSB specification and certification. LSB 5.0 will enter maintenance mode once the release is out the door. This implies bugs will be accepted and we'll try to get them fixed in a reasonable amount of time. Accumulation of fixes may result in releases of LSB 5.0.1, 5.0.2, and other minor releases. What we will not do is make grand plans for LSB 5.1 or 6.0 specifications. Instead the LSB working group will change it's focus. That's how we left it at the end of the day after basically spending all day on just this topic.

Going forward the Working Group wants to focus on real world problems that makes live for ISVs and system administrators difficult. A big part of Linux penetration into the commercial world is that for the most part ISVs and administrators can treat Linux as one platform, no matter who the distribution vendor is. Yes, especially on the admin side there are some differences but in the over all picture they are rather minor, and that's a good thing. We at the LSB work group would certainly like to think that with the work we have done over the years we have contributed to the current state of the art. As a work group on "neutral ground" we would like to become the place where distributions can work together to resolve cross distribution issues. In order to facilitate this conversation and problem resolution we started to create new infrastructure on Github. So far we have started by defining a work-flow, providing an explanation of what's going on and some guidelines about how we see the contribution stuff working. There is also a first "Problem Statement" that can probably use some additional polish and examples, hint hint. Things that people think the LSB work group should tackle should be filed as issues in Github. Bugs in the existing standards should still be filed in the existing Bugzilla.

There is much to be worked out of course. The LSB as we know it today has valuable parts, there are thousands of tests locked up in the current LSB certification framework that should be preserved and be made easily consumable by distributions and or upstream projects. There are valuable tools such as the app-checker and some accompanying backend database stuff that is really helpful to ISVs. We also have to see how we can "transport" ISVs that answer the "do you support distribution X" question with "use an LSB certified distro" into a post certification world. There's no clear cut answer about the meaning of a Linux platform when there is no formal specification that promises at least a certain set of interfaces, nor are we certain whether this actually matters at this point in the progression of Linux as a platform. As fingers diligently scrape away the remaining lumps on LSB 5.0 we are gearing up to step into a different role and I personally haven't been as excited about the LSB as I currently am in a few years.

So, get involved. Lets make the LSB a happening place again. Bring your cross distribution issues to the fore, add them to the Github issue tracker. If we can all keep pulling in the same direction and keep the core distribution bits close together ISVs and admins can continue to treat Linux as one platform which will grow the cake for everyone involved.