• Autoupdate and native packaging has to stay separated. It is not possible for autoupdate to modify or upgrade packages that have been installed by native packager. Both methods can coexist under the assumption that they both update completely different files/directories.

• Any user-mode module management when using Web Start. If you click on the .jnlp file, you get some enabled modules; take it or leave it. If you want a different config, set one up - there should be tools available for power users and sys admins to easily create customized NB configurations for JNLP.

• Explicit module management (Tools -> Options -> Modules) as part of the core UI. This should be a feature of Auto Update, if that is installed (i.e. on Windows or Unix in evaluation mode), and properly integrated with the update functionality (so that Delete really uninstalls NBMs, see issue 20323, etc.).

Solution

Because of the requirement to have different products based on NetBeans we have choosen a solution based on multiple installation directories (for discussion see issue 27151). Different products are composed from a set of clusters. There will be a NetBeans Platform, NetBeans IDE, NetBeans Enterprise, and NetBeans Branding clusters. They will then be composed into related products:

• NetBeans IDE Product - contains the base NetBeans Platform cluster, two clusters providing the basic IDE functionality - NetBeans IDE cluster and NetBeans Enterprise cluster. Moreover it will have its own branding and other non-library parts in its own NetBeans Branding cluster.

• Native Product - contains NetBeans Platform cluster and the NetBeans IDE cluster but it does not include NetBeans Enterprise cluster as it does not need the functionality at all. The own additions of the product are in its own Native cluster.

• lib: contains native JNI libraries and basic boot files. The naming policy of the libraries is in hands of the modules, as that it has worked well enough till now. If it proves unsufficient following conventions shall be obeyed: There are subdirectories under the lib directory that reflect the architecture (e.g. i386, obtained via System.getProperty ("os.arch")) and possibly also subdirectory identifying the system (e.g. i386/linux, obtained via System.getProperty ("os.name").toLowerCase ()). When the system is looking for a particular library it scans the deepest directories first and then the shallow ones (lib/i386/linux, lib/i386, lib).

• core: contains JAR files that will be placed into the dynamic system classpath when the NetBeans based system is starting.

• modules: contains directories and subdirectories with modules. Usual convention is to name module according to its code name base (e.g. org.netbeans.modules.html will be in JAR file org-netbeans-modules-html.jar) and if the JAR uses Class-Path entry, then put such extension into ext/ subdirectory.

• config: is a place where configuration data are being stored like Modules/ directory, etc. A view of this directory is visible as Repository.getDefaultFileSystem() from inside running NetBeans system.

• docs: a directory dedicated to documentation, release info, credits.

The NetBeans IDE product should provide its own executable /usr/bin/netbeans which will locate the $nbplatform/lib/nbexec (currently implemented here for unix and here for windows) executable, read their own configuration files, find out locations of additional clusters and pass --clusters /pathto/nb4.0:/pathto/ide4:/pathto/enterprise7 and --userdir $HOME/.myproductuserdir to it.

The Native Product should have its own launcher called netbeans-native and read its own settings, specify its own user directory and list of clusters (probably --clusters /pathto/ide6:/pathto/web9 as the Native product is extension of the base NetBeans IDE without the enterprise part) and pass that information to the generic $nbplatform/lib/nbexec.

Configuring the Launcher

• --clusters - specifies the paths to clusters that shall be placed on the execution stack (config, modules, configurations, etc.). Individual elements are separated using java.io.File.pathSeparator so for example on Windows it is ; and on Unix :

• --branding - gives the name of branding that shall be used

• regular parameters - these are end user oriented parameters printed when invoked with --help option:

  Usage: bin/../platform4/lib/nbexec {options} arguments
  
  General options:
    --help                show this help
    --jdkhome <path>      path to Java(TM) 2 SDK, Standard Edition
    -J<jvm_option>        pass <jvm_option> to JVM
  
    --cp:p <classpath>    prepend <classpath> to classpath
    --cp:a <classpath>    append <classpath> to classpath
  
  Core options:
    --laf <LaF classname> use given LookAndFeel class instead of the default
    --fontsize <size>     set the base font size of the user interface, in points
    --locale <language[:country[:variant]]> use specified locale
    --userdir <path>      use specified directory to store user settings
      

Compatible development

• $nbnative cluster (in the Native Product 1.0) needs debugger major (incompatible) version 1 and minor x.y.z (as it does not matter for this example)
• $nbweb (in the Web Product 7.1) needs debugger major version 2 and minor a.b.c
• debugger module is provided by the base $nbjava, version 4.3 delivers the debugger major 1 (minor u.v.r where u.v.r >= x.y.z so it satisfies the requirements for Native Product 1.0) and version 5.1 provides debugger major 2 (and some specification version, which is not important)

Requirement A: It is necessary to install more than one incompatible version at once.

Note: Version 3.1 of module A is incompatible with version 2.8 if there exists a module B that depends and runs with version 2.8 of A and does not run with version 3.1 of module A (more).

A possible solution is to encode the version (or at least major version) of the module in the JAR file. So instead of text.jar the module would be named debugger-1-1.32.jar. Then multiple versions could be installed at once. Another possible solution is to place the JAR files into different directories. Which means to encode some version number in the directories itself. Again this satisfies the requirement that multiple incompatible versions can be installed at once.

Packaging into RPMs

• An RPM package m.rpm contains some files and also debugger major version 1 and is delivered as part of the Native Product 1.0.
• Another RPM package n.rpm (does not matter whether it is just a different version of m.rpm or has completely different name) contains the debugger major version 2 and is delivered as part of Web Product 7.1.

Observation 1: Regardless of the size of native packages, an incompatible change in one module means that other parts of the package has to change location (pretend incompatible change) as well.

This implies that if we package whole cluster into one RPM and there is an incompatible change in one module, which happens nearly every release, all files in the RPM has to change location in order to allow coexistence with the RPM of previous release. Well, if we do this, then it is better to include the version number directly in the name of the directory, than uselessly mangle names of the modules.

RPM for Product and RPMs for Clusters

Can this launcher script be part of the same RPM that contains the debugger module? Due to observation 1 it would have to change its name or location if debugger or other part of the RPM would change incompatibly.

That either means to include a version name in the launcher script (netbeans35) or keep it in separate RPM which only depends on other cluster RPMs.

Better seems to have own RPM for Product that just declares dependencies on RPMs for clusters, as this allows separation of compatibility versioning (nbjava35-1.0.rpm and nbjava36-1.0.rpm) that appear to the RPM management system as two different packages, the user visible versioning like NetBeansIDEProduct-3.5.rpm and NetBeansIDEProduct-3.6.rpm that will be treated as updates with proper notification. Of course this means to keep compatibility of the launcher script but that shall not be as hard nor too important.

Impact on Module System

As the compatibility is more important than any possible benefits from new incompatible updates, it is better not to do any tricks in module system and leave the decision on writers and packagers of the system.

External Contributions

There is another related problem to sneaking - uninstallation. If user installs external modules in the $nbjava dir created via package installation, then when uninstalling this cluster, this may create some problems. Uninstaller will not be able to properly clean up the product/cluster installation directories. Some files will be left behind. As the installer does not know if its safe to delete these dirs or not.

Restrictions

The transitivity problem is unwanted, uneliminatable effect. But it is easily preventable if developers are careful enough. The only necessary thing is to produce an incompatible version whenever any of clusters we depend on changes incompatibly. So in the example above, the IDE Product should include new incompatible $nbide 4.9 instead of pretending compatible evolution.

Conclusion

• As not all clusters can commit to do compatible development for now and ever (reality of life)
• As two or more versions of the module has to be installed at once (discussed here)
• As RPMs carrying incompatible variations have to able to coexist (discussed here)
• As having the two versions of a module would complicate the module system (discussed here)
• As there shall be a Product and Cluster packages (discussed here)
• As we need extensibility of clusters (discussed here)

It is up to the individual clusters to decide whether they want to develop in compatible or incompatible way. Both is possible with all its pros and cons. The ability to deliver compatible releases highly depends on correct partitioning of clusters.

The cluster is packaged into one or more RPMs that are designed to keep external compatibility as much as possible - for that reason they contain the name of the version in their name (nbplatform5 vs. nbplatform6), thus allowing different incompatble versions of an cluster to be installed at once. It is possible that the same directory is filled with a content of foreign RPMs that just want to provide some extensions to the base functionality of such extension.

There is a product RPM containing just the launcher script and possibly private product cluster, having references to all clusters that it needs. This product RPM does not include compatibility identification as it will evolve in compatible way (also due to the fact that nobody is supposed to depend on it cluster) and thus new versions of the product are easily recognizable by native package managers and offered for update. Such update will then update all needed clusters with their appropriate versions. Removing of useless cluster native packages that are not needed anymore would be nice, but is more a task for the native packager that should realize that there is a package used by nobody.

Good Practices: Do not package things with different compatibility lifecycle together

Good Practices: Separate API modules and GUI modules

If separated, then the API module can become autoload and can be deprected after a while (but still offered for a compatiblity) and after another while completely removed. The GUI module can meanwhile change inside out, with a possiblity to introduce new API autoload module that replaces the old deprecated one.

Good Practices: API modules should not register anything

Good Practices: Do compatible development

Breaking compatibility of a cluster every release means that products depending on your cluster will never get an updated version or in the best case, you will have to produce patches and updates for critical bugs to more incompatible versions of a cluster at once. Which is bigger price than one really wants to pay.

Windows

Appropriate execution scripts are supposed to be installed into the c:\Program Files\NetBeans\bin directory by the installer of actual Product.

Each product installer will mark and register each installed cluster into windows or JDK1.4 registry, so other installers are able to find the previously installed clusters and reuse them.

Solaris - workstation install

• The $nbcluster/config, $nbcluster/lib, $nbcluster/share and $nbcluster/docs are placed under /opt/netbeans/cluster6

The native package for each Product will provide launcher scripts and additional icons. Executables shall be installed in /opt/bin.

Solaris - compound install

Example: One installs NetBeans into /server/for/clients/soft/netbeans/ on server and mounts it as /opt/netbeans on clients. The directory /opt/netbeans/platform-3.5 than contains everything config, lib, boot, modules, docs.

The default locations of other needed clusters in launcher script shall use relative paths, so if a set of clusters ($nbnative, $nbenterprise, $nbjava, $nbxml) is installed into one server location (/server/for/clients/soft/netbeans/) than there is a need of only one NFS mount to make all of them available on client systems. Moreover the relative paths will continue to be valid, whereever the components are relocated to. Of course these default paths shall be overridable by specifying arguments or some environment variables to allow usage of clusters in non-standard locations.

The native package for each Product that provides executables shall install them to bin directory under the NetBeans server installation directory. E. g. /server/for/clients/soft/netbeans/bin in the example in previous paragraph.

Note: For Solaris server-based installations, it is better not to edit the launch scripts at installation time. The path to $nbjava may be different on the desktops. Each desktop should mount $nbjava and $nbplatform to the default location, or each desktop should use an environment variable, or .... Also it is better not to complicate the installer by requiring different kinds of installations for servers. The installer shall not need to know if it is doing a server or a local install. No editing of scripts at install time.

Linux

Linux Product RPMs will normally install additional things into e.g. /etc/X11/applnk/Applications, /usr/share/pixmaps, etc. Typically each Product (i.e. the RPM that included the product launcher script) would create a set of such files along to the command line launcher.

ZIP file structure

netbeans/bin/netbeans
netbeans/platform4/boot
netbeans/platform4/modules
netbeans/platform4/config
netbeans/ide7/modules
netbeans/ide7/lib
netbeans/ide7/config
etc.

• modules, docs: have the same meaning as in the common installation. The user can install additional modules here (usually by AutoUpdate because we do not want to mix the native packager installed modules with those installed by AutoUpdate).
• config: contains user modifications to files found in shared config directories as well as those in module's layers
• var/cache: contains pre-cached information about things that can be found elsewhere (mostly info about enabled modules, etc.)
• var/log: directory to write logs to. Will contain more than one log file, at least messages, and older archives messages.1, messages.2 like the /var/log directory system does.
• var/cache: directory to keep discardable caching files in. Right now used only by platform to cache informations about enabled modules.
• There maybe additional directories provided by different clusters in the structure

AutoUpdate

When a native package installation is used, AutoUpdate is restricted from accessing the directories/clusters under the control of native package manager. Which usually means to update just user directory.

Localization of updater must be possible. The proposed solution is to pass the branding and cluster directories from a product launcher directly to updater so it can in its early stages know the right language or branding variant.

Detailed Functional Analysis

1. Regardless of how the product is installed, user is allowed to autoupdate to his user directory (if the autoupdate is present, e.g. it is not in webstart version). If that happens, the modules in user dir takes precedence over system one (if it is newer).
2. If the product is installed from RPM or solaris package and possibly also from WindowsInstaller, the autoupdate is not able to update those clusters. This is signaled by existing $nbcluster/.noautoupdate file. In such case the autoupdate must not touch the directories.
3. If the product or its part is not installed using native installation method (packages), the autoupdate is technically able to update these clusters. It will recognize them by non existance of $nbcluster/.noautoupdate (this is likely to be created by RPMs and other native packagers). In order to work correctly the AutoUpdate needs to decide which cluster the to be installed NBM belongs. If the NBM represents an update, it should be put into the same cluster where the original NBM was. If the NBM is new and user would like to install it globally, the module is installed into the "highest" - e.g. private cluster (in case of NetBeans nb4.x), as it represents the product private cluster and thus the influence of an NBM installed this way on other products is minimal - more likely zero.

Hotfix Example

First thing to do is to create an NBM file and make it available on one of our AutoUpdate centers. Users running on any platform, using any kind of installation method (but WebStart) can see it and without any restrictions install in their user directory. As the user directory takes precedence over shared install locations, the new editor module will be activated for those users. This is exactly as it works in version 3.6.

Users that used windows or ZIP installation style, will have a chance to select global installation of the NBM file and replace the editor module in ide4 cluster (as it was already installed).

Those that are using UNIX installation based on native packages will not be able to install NBM file globally, as it would clash with native RPM databases. Instead they will connect to their favourite RPM distribution server (urpmi, apt, yum - something like AutoUpdate for RPMs) and download and upgrade the native package that contains editor module. This requires those who are producing NBMs to also produce RPMs and Solaris packages or patches and make them available on suitable distribution channels.

Module System