Lexer Migration Guide

Please note that this tutorial is obsolete. Go here instead:

http://wiki.netbeans.org/How_to_create_support_for_a_new_language

This document shows you how to migrate from an implementation of org.netbeans.editor.Syntax to an implementation of org.netbeans.api.lexer. At the end of this document, you will have a NetBeans module that provides syntax highlighting for Manifest files. You will have tokens for the 'name', 'colon', and 'value' in manifest entries and each will have a distinct color, as illustrated below:

Highlighted manifest file

The starting point of this migration guide is the NetBeans Manifest File Syntax Highlighting Module Tutorial, which implements the org.netbeans.editor.Syntax class. To migrate from an implementation of that class to an implementation of org.netbeans.api.lexer, you will need to replace the files listed in the left column in the table below with those listed in the right column:

Pre-Lexer
Lexer
ManifestEditorKit.java extends NbEditorKit ManifestEditorKit.java extends LexerEditorKit
ManifestSyntax.java extends org.netbeans.editor.Syntax ManifestLexer.java extends org.netbeans.spi.lexer.Lexer
ManifestTokenContext.java extends org.netbeans.editor.TokenContext ManifestTokenId.java extends org.netbeans.api.lexer.TokenId
ManifestEditorKit.java Not needed.
ManifestSettingsInitializer.java Not needed.
org.netbeans.modules.manifestsupport.options Not needed.

The following topics are covered in this document:

  • Migrating the Module
  • Building and Installing the Module
  • Exploring Further
  • Once the software is installed, the migration can be complete in 15 minutes.

    For more information on creating NetBeans plug-in modules, see the NetBeans Development Project home on the NetBeans website. If you have questions, visit the NetBeans Developer FAQ or use the feedback link at the top of this page.

    Setting Up

    Before you start writing the module, you have to make sure you have all of the necessary software. In addition, you might want to play with the sample before building it yourself.

    Installing the Software

    Before you begin, you need to install the following software on your computer:

    • NetBeans IDE 5.5, together with the Lexer modules provided by the Update Center (to come) or Milestone 5 (or later) of NetBeans IDE 6.0
    • Java Standard Development Kit (JDK™) version 1.4.2 (download) or 5.0 (download)

    Installing the Sample

    Take the following steps to install the sample:

    1. Download and unzip ManifestSupport.zip.
    2. In the IDE, choose File > Open Project and browse to the folder that contains the unzipped file. Open the module project. The files that we will deal with are in the main package, org.netbeans.modules.manifestsupport:

      Final Projects window

    3. Right-click the project node and choose Project Properties. In the Sources panel, change the source level to 1.5. In the Libraries panel, add dependencies on Lexer and Lexer to Editor Bridge and Plain Editor Library.

    Migrating the Module

    After taking the preparatory steps above, you need to change four files and delete several others. The files to be changed are listed below, with explanations after the code listing and, in some cases, within the code listings themselves. After that a list of files to be deleted is provided.

    Migrating the Tokens

    Do the following:

    1. Migrate the file. Refactor ManifestTokenContext to ManifestTokenId. Replace the code with the following:
      package org.netbeans.modules.manifestsupport;
      
      import java.util.Collection;
      import java.util.EnumSet;
      import java.util.Map;
      import org.netbeans.api.lexer.Language;
      import org.netbeans.api.lexer.TokenId;
      import org.netbeans.spi.lexer.LanguageHierarchy;
      import org.netbeans.spi.lexer.Lexer;
      import org.netbeans.spi.lexer.LexerRestartInfo;
      
      /**
       *
       * @author Administrator
       */
      public enum ManifestTokenId implements TokenId {
      
          // The token ids may be assigned to categories and the coloring information
          // may then be assigned directly to a tokenId.name() e.g. "NAME"
          // or to a token category e.g. "separator"
          NAME(null),
          COLON("separator"),
          VALUE(null),
          END_OF_LINE("whitespace");
      
          private String primaryCategory;
         
          private ManifestTokenId(String primaryCategory) {
              this.primaryCategory = primaryCategory;
          }
          
          public String primaryCategory() {
              return primaryCategory;
          }
      
          private static final Language<ManifestTokenId> language = new LanguageHierarchy<ManifestTokenId>() {
      
              protected String mimeType() {
                  return "text/x-java-jar-manifest";
              }
      
              protected Collection<ManifestTokenId> createTokenIds() {
                  return EnumSet.allOf(ManifestTokenId.class);
              }
              
              protected Map<String,Collection<ManifestTokenId>> createTokenCategories() {
                  return null;
              }
      
              protected Lexer<ManifestTokenId> createLexer(LexerRestartInfo<ManifestTokenId> info) {
                  return new ManifestLexer(info);
              }
      
          }.language();
      
          public static Language<ManifestTokenId> language() {
              return language;
          }
      
      }
    2. Understand the file. This Java class specifies a token for each item in the Manifest file with which we want to work. Each distinct item in a Manifest file is a token: 'NAME', 'COLON', and 'VALUE'. In addition, there is also a token for the end of the line, because we need to work with the end of the line—the end of the line determines where a value ends and the next name begins. We can also assign tokens to categories and then bind colors to categories. For example, above COLON is assigned to the category 'separator', while END_OF_LINE is assigned to the category 'whitespace'. If the example was not so simple, we would probably assign multiple tokens to the same category, which is superfluous here.

    Migrating the Lexical Analyzer

    Do the following:

    1. Migrate the file. Refactor ManifestSyntax to ManifestLexer. Replace the default code with the following:
      package org.netbeans.modules.manifestsupport;
      
      import org.netbeans.api.lexer.Token;
      import org.netbeans.spi.lexer.Lexer;
      import org.netbeans.spi.lexer.LexerInput;
      import org.netbeans.spi.lexer.LexerRestartInfo;
      import org.netbeans.spi.lexer.TokenFactory;
      
      public class ManifestLexer implements Lexer<ManifestTokenId> {
          
          private static final int EOF = LexerInput.EOF;
      
          // Lexer internal states - preferably small integers for more compact token storage
          private static final int INIT = 0;
          private static final int AFTER_COLON = 1;
          private static final int AFTER_NAME = 2;
      
          private LexerInput input;
          
          private TokenFactory<ManifestTokenId> tokenFactory;
          
          private int state;
          
          public Object state() {
      		// autoconversion uses Integer.valueOf() which caches <-127,127>
              return state;
          }
      
          public ManifestLexer(LexerRestartInfo<ManifestTokenId> info) {
              this.input = info.input();
              this.tokenFactory = info.tokenFactory();
      		this.state = (info.state() != null) ? (Integer)info.state() : INIT;
          }
          
          public Token<ManifestTokenId> nextToken() {
              int c = input.read();
              switch (state) {
                  case INIT:
                      return nextTokenInit(c);
                  case AFTER_COLON:
                      return nextTokenAfterColon(c);
                  case AFTER_NAME:
                      return nextTokenAfterName(c);
                  default:
                      throw new IllegalStateException();
              }
          }
              
          private Token<ManifestTokenId> nextTokenInit(int c) {
              switch (c) {
                  case ':': // ":"
                      state = AFTER_COLON;
                      return token(ManifestTokenId.COLON);
                  case '\r':
                      input.consumeNewline(); // continue to '\n' handling
                  case '\n':
                      //state = INIT;
                      return token(ManifestTokenId.END_OF_LINE);
                  case EOF: // no chars -> finish lexing by returning null
                      return null;
                  default: // Name follows
                      return finishName(c);
              }
          }
      
          private Token<ManifestTokenId> nextTokenAfterColon(int c) {
              switch (c) {
                  case ':': // ":"
                      state = AFTER_COLON;
                      return token(ManifestTokenId.COLON);
                  case '\r':
                      input.consumeNewline(); // continue to '\n' handling
                  case '\n':
                      state = INIT;
                      return token(ManifestTokenId.END_OF_LINE);
                  case EOF: // no chars -> finish lexing by returning null
                      return null;
                  default:
                      return finishValue(c);
              }
          }
      
          private Token<ManifestTokenId> nextTokenAfterName(int c) {
              switch (c) {
                  case ':': // ":"
                      state = AFTER_COLON;
                      return token(ManifestTokenId.COLON);
                  case '\r':
                      input.consumeNewline(); // continue to '\n' handling
                  case '\n':
                      state = INIT;
                      return token(ManifestTokenId.END_OF_LINE);
                  case EOF: // no chars -> finish lexing by returning null
                      return null;
                  default:
                      throw new IllegalStateException();
              }
          }
      
          private Token<ManifestTokenId> finishName(int c) {
              while (true) {
                  switch (c) {
                      case ':':
                      case '\r':
                      case '\n':
                      case EOF:
                          input.backup(1);
                          state = AFTER_NAME;
                          return token(ManifestTokenId.NAME);
                  }
                  c = input.read();
              }
          }
      
          private Token<ManifestTokenId> finishValue(int c) {
              while (true) {
                  switch (c) {
                      case '\r':
                      case '\n':
                      case EOF:
                          input.backup(1);
                          state = INIT;
                          return token(ManifestTokenId.VALUE);
                  }
                  c = input.read();
              }
          }
      
          private Token<ManifestTokenId> token(ManifestTokenId id) {
              Token<ManifestTokenId> t = tokenFactory.createToken(id);
              return t;
          }
          
          public void release() {
          }
      
      }
    2. Understand the file. This Java class tells the IDE which part of the text is which token. It does this by starting in an initial state and sequentially looking at each character in the text and deciding if it stays in that state, moves to another state, or announces that a token was found.

    Migrating the Bundle.properties File

    Your Bundle.properties file, which is in the same package as the previous two classes, should have this content:

    #Layer.xml entries for fonts & colors in Options window:
    NetBeans=NetBeans
    NAME=Name
    VALUE=Value
    COLON=Colon
    separator=Separator
    whitespace=Whitespace

    Migrating the XML Layer

    This is how your new Lexer implementation should be registered:

    <folder name="Editors">
        <folder name="text">
            <folder name="x-java-jar-manifest">
                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.myorg.manifestfiletype.Bundle"/>
                <folder name="NetBeans">
                    <folder name="Defaults">
                        <file name="coloring.xml" url="resources/NetBeans-Manifest-fontsColors.xml">
                            <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.manifestsupport.Bundle"/>
                        </file>
                    </folder>
                </folder> 
                <folder name="CityLights">
                    <folder name="Defaults">
                        <file name="coloring.xml" url="resources/CityLights-Properties-fontsColors.xml">
                            <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.manifestsupport.Bundle"/>
                        </file>
                    </folder>
                </folder>
                <file name="language.instance">
                    <attr name="instanceCreate" methodvalue="org.netbeans.modules.manifestsupport.ManifestTokenId.language"/>
                    <attr name="instanceOf" stringvalue="org.netbeans.api.lexer.Language"/>
                </file>
                <file name="EditorKit.instance">
                    <attr name="instanceCreate" methodvalue="org.netbeans.modules.lexer.editorbridge.LexerEditorKit.create"/>
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.lexer.editorbridge.LexerEditorKit"/>
                </file>
            </folder>
        </folder>
    </folder>
    
    <folder name="OptionsDialog">
        <folder name="PreviewExamples">
            <folder name="text">
                <file name="x-java-jar-manifest" url="resources/ManifestExample"/>
            </folder>
        </folder>
    </folder>

    Finishing Up

    Do the following:

    1. Delete ManifestEditorKit. You do not need this anymore, because of the LexerEditorKit registered in the XML layer. Right-click the ManifestSettingsInitializer.java file and choose Delete.
    2. Delete ManifestSettingsInitializer. You do not need this anymore, because initizialiatio is done through the XML layer. Right-click the ManifestSettingsInitializer.java file and choose Delete.
    3. Change the installer. Previously, you used a module installer to call the ManifestSettingsInitializer. You no longer have it, so there is no need to call it. In RestoreColoring.java, delete the addInitializer() method and remove it from the restored() method.
    4. Delete the Options package. Delete org.netbeans.modules.manifestsupport.options.

    Building and Installing the Module

    The IDE uses an Ant build script to build and install your module. The build script is created for you when you create the module project.

    Installing and Testing the NetBeans Module

    1. In the Projects window, right-click the ManifestSupport project and choose Install/Reload in Target Platform.

      The module is built and installed in the target IDE or Platform. The target IDE or Platform opens so that you can try out your new module. The default target IDE or Platform is the installation used by the current instance of the development IDE. Note that when you run your module, you will be using a temporary test user directory, not the development IDE's user directory.

    2. Choose File > New Project (Ctrl-Shift-N). Create a new Java application project or a new Plug-in Module project. Open the project's Manifest file in the Source Editor and notice the syntax highlighting.



    Next Steps

    For more information about creating and developing NetBeans modules, see the following resources:


    Versioning

    Version
    Date
    Changes
    1 5 November 2006
    • Initial version.
    2 12 December 2006
    • Removed editorkit. Not necessary anymore. LexerEditorkit is declared in layer XML.
    • Added migration of Bundle.properties and XML layer.
    • Changed token ID class and Lexer class, to reflect latest state of the implementation.
    3 13 December 2006
    • Removed the Options package. (Forgot to do that yesterday.)
    • Small tweaks.

    Project Features

    Project Links

    About this Project

    Platform was started in November 2009, is owned by Antonin Nebuzelsky, and has 149 members.
    By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20160708.bf2ac18). © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
     
     
    Close
    loading
    Please Confirm
    Close