Simon had a short paper at the Working Conf. on Reverse Engineering (WCRE) last October, presenting his first study on open source Java applications. The paper was quite well received, from what he told us. Simon’s talk was the only non-invited talk the WCRE steering committee chair twittered about, and it was mentioned the next day by Nicolas Anquetil during his most influential paper talk.

Simon has meanwhile enlarged the depth and breadth of the study: he did the analysis at method level (before it was at class level), he compared naming quality with more code quality metrics, and he also compared it with a readability metric. He also did a statistical analysis to see if some naming problems (e.g. not using dictionary words) could be used as a light-weight test to point to methods with readability and quality problems. The paper we submitted to the European Conf. on Software Maintenance and Reengineering (CSMR) was accepted and today we uploaded to our institutional open access repository the camera-ready version that, from past experience, will take some months to show up in IEEE Xplore.

In our paper at the 2008 International Conference on Software Maintenance, my colleague Yijun Yu and I presented an analysis of the evolution of the plugin-based Eclipse SDK architecture. The aim was to see whether some systematic architectural change process was followed and if commonly prescribed design guidelines (like increasing cohesion and decreasing coupling) were followed. Towards that end we collected several metrics (number of plugins, number of dependencies, etc.) and plotted them over various releases of different types (major release, maintenance release, etc.).

We have now analysed a few more Eclipse releases, did more measurements (on extension points) and refined others (distinguishing between forced and unforced changes), and decided to make the data public, by putting the metrics on Google spreadsheets and using Google Visualization to show the evolution of the measurements. One spreadsheet (HTML, CSV, Excel, OpenOffice) contains the metrics for major and maintenance releases from 1.0 to 3.5.1, the other (HTML, CSV, Excel, OpenOffice) has the metrics for the milestones and release candidates of releases 3.2 and 3.3. You’re free to use the data for your own purposes, provided you acknowledge where you got it from, e.g. by providing the URL of this post. Our Javascript code to obtain the charts from the spreadsheets is open source: just use your browser’s ‘show page source’ feature! In that way you can reverse engineer what the headings in the spreadsheets columns mean, but an easier way is to just contact us We next present the results, but they will only make sense if you first read the paper.

We use mostly stacked bar charts, with each segment showing a particular subset of the total number of items (plugins, extension points or dependencies). The segments are stacked, from bottom to top, as follows: unforced deletions, forced deletions, items kept since the first release, items kept from the previous but not the first release, forced additions, unforced additions. In general, a change is considered unforced if it is by choice, and forced if it is due to another change. For example, the unforced deletion of a plugin forces the deletion of all its extension points and dependencies. Clicking on a bar segment will say precisely what value is being plotted for what release and what kind of change.

We use line charts to show ratios or other values. Again, it is possible to click on a data point to obtain its precise value, instead of guessing it from the y-axis.

The first charts show the evolution of Eclipse’s size, as measured by the number of plugins. We consider all additions and deletions as unforced, because they are architectural choices. We can see that the set of plugins remains constant during maintenance releases, i.e. additions and deletions are only observed at major releases. A look at the interim milestones and release candidates reveals that most of those changes occur in milestones, although some also occur in the later release candidates. (Note that in this case, the number of kept plugins is with regard to the first release in the sequence, i.e. 3.1.)

The next charts plot the changes to complexity, measured as the number of dependencies between plugins. We distinguish between static and dynamic dependencies (see the paper for what those terms mean in our context). The overall complexity is the union of both, i.e. a dynamic and a static dependency between the same pair of plugins count as one overall dependency between that pair. A forced addition or deletion of a dependency is associated to the creation or removal of at least one of the involved plugins, i.e. the addition (resp. deletion) of a dependency between two plugins is called unforced if both plugins already existed (resp. still remain). Again, we see that dependencies change mostly during milestones and remain the same during maintenance, except for a few deletions in 3.3.1.

Overall, the biggest changes were in releases 3.0 (almost doubled the complexity while retaining the size) and 3.4 (almost doubled the size). Only 3.1 presents a decrease in complexity. If we now compute directly from the spreadsheet the cohesion, measured as the dependency-to-plugin ratio, we note it’s remarkably constant except for the increase at 3.0 and decrease at 3.4.

The next chart shows the changes to coupling, measured as the number of external dependencies, i.e. between an Eclipse SDK plugin and a 3rd-party component. All external dependencies are static. We can see that coupling was drastically reduced in 3.0, with all existing external dependencies being replaced by new ones. The paper explains what happened.

We also measured the number of extension points provided by the Eclipse SDK and how it changed over time. The addition (deletion) of an extension point is considered forced if the plugin that provides the extension point has been created (resp. removed) simultaneously. We first show the absolute, then the relative number of extension points. We note that most extension points are added to already existing plugins. Hence, new plugins have few or no extension points, leading to a decrease in the average number of extension points per plugin.

Also interesting is to see how many unused extension points there are, i.e. whether the SDK is ‘eating its own food’ or providing extension points mostly for 3rd-party plugins. The addition of an unused extension is considered to be forced if it is due to the deletion of a dynamic dependency (and hence the extension point became unused). The deletion of an unused extension point is forced if due to the deletion of the corresponding plugin. We note that most additions are unforced and for external plugins, because the percentage of unused extension points is constantly rising. Release 3.5 removed a large part of the plugins with unused extension points.

I’m very pleased with Ireo’s efforts and with the strategy followed to get a publication out of his work. Our MSc students are all part-time and off-site (i.e. they don’t come to campus). Hence, it’s a bit difficult to get them to write a paper once the dissertation is submitted. They go through various intermediate assignments, which build up to the final dissertation submission in March. The last assignment consists of the last dissertation draft and of the first version of an extended abstract of the thesis. The word limit imposed on the abstract is rather constraining for a paper. Moreover, the abstract’s second and final version is submitted after the actual dissertation. Asking a student to then expand the abstract into a paper is therefore pointless, from their perspective, because the work is over and there is no academic incentive (e.g. a better grade) for them to write a paper.

The strategy we followed was to have Ireo send me versions of the extended abstract well before the last assignment, and gradually expand it into a paper. This helped him stengthen and streamline the narrative (context, method, results, conclusions) for his last dissertation draft. The deadlines of the workshop were also quite convenient, with the submission date two weeks after the last assignment, and the notification date well before the dissertation submission. In this way, the paper could benefit from what had been written for the last assignment, and the final dissertation and extended abstract can benefit from the reviewers’ comments. I like when everything falls into place!

We illustrate the approach by relating developers and Eclipse bugs and then showing the hierarchy of bug fixers and discussants, aggregated by component. Developers towards the top of the hierarchy are generalists (involved in more components), those at the bottom are specialists (involved in just one component).

The reviewers had some questions about technical details of our example, and about the potential of the approach — good points to take care for the final version. Meanwhile, you can read the submitted version.