Friday, September 23, 2011

p2 and equinox "office hours"

You have beginner or experienced-level questions about equinox or p2? You want to contribute to equinox or p2? Just want to chat with us?
Join us on our weekly office hours every Tuesday at 1pm EST on the #equinox-dev channel on IRC (http://wiki.eclipse.org/IRC).

During these meetings equinox and p2 committers, contributors and aficionados will answer questions on any topic you come up with. Questions to be answered on a first come, first served basis.

See you there.

Ps: This is inspired from the office hours held by the Jenkins team

Monday, September 12, 2011

More on p2 and OBR

When I set out to write my previous blog post about why not OBR, the intent was to give, with a catchy title, an historical perspective on p2 (thus the recent title update) but also to answer people who have been asking for years this question

(and yesterday night another one of those came in).


The point was not to do a comparison of p2 and OBR as they are today, and especially not to claim that OBR is not good today. Both technologies have pros and cons. My statements about OBR were a reflection of the situation as it was ~5 years ago. To be honest, I have not been following the recent development on the spec, just heard some rumours.


When I wrote the previous post, I also wrote two additional paragraphs that I had not published. One was about our absence of involvement in the spec process and one about going forward. So here they are:


Could we have collaborated on the spec and implementation?

Maybe, but at the time the interest around OBR was rather low at the OSGi alliance, we did not had the bandwidth to drive improvements to the specification since we were focusing on shipping running software (the first release may not have been the best thing, I agree), and my management at the time was not up for the work, distraction, and frustration it could have represented. It needs to be remembered that some of us had been involved in the OSGi spec process to drive changes (adding fragments, bsn, require bundle, etc. Yes all these mostly were driven by the equinox team) and it had been a painful experience that left the people involved with scars, and made some of them the ongoing target of criticisms because of the changes they pushed for.


A couple years later (~2009?), OBR got brought back the forefront in the context of the OSGi Enterprise Expert Group. At this point, I have been involved in the spec work even to a point where I was the lead. Because of my lack of time, and the problems scoping the work, my contributions have been limited and then I stopped being involved as Sonatype withdrew from the OSGi alliance.



Is there a way forward?

Until a publicly visible draft of OBR is available, I can't provide a definitive answer. However implementing OBR on top of p2 has definitely been discussed several times and is not out of question especially if the specification stays focused on providing a resolver API and a repository API.


Now time, energy and contribution will tell...


PS: there are things that need to be corrected from Neil's post, but my other commitments are calling so it will have to wait.


Sunday, September 11, 2011

Why not OBR? - An historical perspective on p2.

Many times I've been asked why p2 got created when there were other solutions around, namely OBR. To understand why p2 got created, it takes a little bit of understanding on the requirements that drove the work on p2, but also some history.


So let's start with the requirements.

The work on p2 started in 2006 / 2007 at at time where companies contributing to Eclipse were wondering how to increase their ROI. For this, they were all sure that having an easy path from the free Eclipse distress to their cool add-ons or full suite of products would be the solution. The term coined for that was "on-ramp". Nowadays what best embodies this is the Eclipse Marketplace and its client.


It is in this context that IBM spearheaded an internal effort to replace the aging update manager. The requirements were to provide a "provisioning platform" that would allow to install the complete Eclipse SDK from scratch (not only bundles and features like the old solution was), and also deal with more native things like dlls, .net assemblies, rpms, etc. and provide desktop integrations. In addition to that, the solution was expected to:

- allow both fine grained provisioning (just want to install a bundle) or large grained provisioning (install a complete product) -- this is why we used the term "right-grained provisioning" to describe a solution that would let its user decide the right granularity for himself.

- provide a "transactional" update, meaning that a failed or cancelled install could not leave the system in an unknown state

- allow to install and uninstall without having the provisioned system be up and running

- separate the metadata from the payload

- support mirroring

- …


With those requirements, I spent a fair amount of time looking at what was being provided in the Java and OSGi ecosystems, as well as what could be offered from the linux package managers and other esoteric technologies. This was a phase of research were I was harvesting as many ideas as possible, comparing them with each others, matching them with my requirements. It is really this phase that defined much of what p2 has become. Without being exhaustive, here are some of the things that I looked at: SDD (http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=sdd) from oasis, OBR from OSGi, Deployment Admin from OSGi, JSR 124 (http://jcp.org/en/jsr/detail?id=124), another similar JSR for embedded device (can't remember the number), Maven, debian, conary (http://wiki.rpath.com/wiki/Conary), RPM, numerous Java installers, etc. From their I went down the path of research papers for dependency resolvers, looked at constraints programming, and was lucky to come across the paper from Tucker that studied the usage of a SAT solver to resolve install problems (unfortunately his work never made it to production).


So why not OBR?

OBR was not up to the task. OBR was a repository format, a resolver, and a downloader but each of these element was far away from being sufficient to build what we needed on top.

- OBR focus was bundle centric. No "group" (aka features), no native files, …

- The repository format was fixed and was XML only, no possibility of extension.

- Each bundle listed in the repository was referring to the payload (the actual jar for the bundle) using a fix URL rather than abstracting the artifact location in a coordinate -- like Maven does with its concept of GAV or like p2 does with "artifact keys" -- which makes mirroring just plain painful (I know it is possible to provide URL handlers, but come on…).

- The provided API was not allowing for transactional state change (some bundles could be installed, but not all the required ones if the download failed)

- The metadata did not accommodate the expression of what needed to happen to the bundle (e.g. should it be started, set the start level, etc.) and thus even less to other things.

- The resolver always insisted on installing all the fragments available (which in complex repos like Indigo means that installing JDT will bring in parts of WindowBuilder since it has some fragment to JDT UI iirc)

- …



As you've seen, a fair amount due diligence to OBR and plenty of technologies had been done. Rather than being a thing that just came to be by itself, p2 claims lineage from pretty much all these technologies, and the whys could be repeated and inverted at will :).


PS: Short lineage: InstallableUnit and Touchpoint terminology comes from SDD, parts of the transactional aspect of the p2 engine is inspired from OSGi deployment admin, the list of phase is from debian, the metadata content is inspired from OBR and OSGi Manifest, Maven for the idea of coordinate system...


Update: Changing the title to reflect the intent of the post