Commit Rules: Pushing To The NetBeans Mercurial Repository

Having push access to the NetBeans Mercurial repository is a privilege but also a responsibility: responsibility to the quality of the code and responsibility to other people working on NetBeans. In order to ensure that the quality of the code base is high and that you will not disturb anybody else's work, here are a few guidelines that will help you to validate your changesets before you push them into the NetBeans Mercurial repository.

1. When your modifications are ready to be pushed do a clean build to make sure that it does not negatively influence other modules. Execute

   ant clean

   to clear the sources and

   ant

   to compile the NetBeans IDE. Currently we are building with JDK 1.5, so you can use all its methods, but be careful not to depend on anything newer.
2. Execute commit validation test suite which is designed to test the basic functionality of NetBeans and shall affirm you that your integration is not completely broken. Execute

   ant commit-validation

   Of course no test shall fail otherwise there is something wrong - probably with your integration. You should evaluate why it fails.
3. Execute unit tests that might be influenced by your changes. For example when committing into the form module its test shall be run by

   ant -f form/build.xml test

   and shall not fail.

4. If you use Linux, you can run a premade script that does the clean build and test suites for you. Try nbbuild/misc/full-build-linux.sh (see the shell script comments for usage instructions).

5. If you really want to verify everything that we verify on continuous build, run:

          ant -f nbbuild/build.xml commit-validation
              -D"xtest.userdata|com.sun.aas.installRoot"=<glassfish install dir>
              -Dwtk.dir=<path to wtk22_win.zip or wtk22_linux.zip>
              -Dxtest.config=commit-validation-all

   For detailed description of these tests look at this document.

6. If the IDE compiles, tests pass you are ready for integration. Check what you have changed by evaluating the output of hg out -p and if the output really contains what you have modified do hg push.

Evaluate commit validation results

Once you have executed commit validation test suite you have to evaluate its results. If you run it from command line, at the end it is printed out summary information about pass ratio of the suite. There is also provided a link to HTML presentation of the results. If something fails, you should follow this link and check error messages and log files to determine what is wrong with your build.

Main page of the results is split into two frames. The left navigation frame shows you links to results of particular modules and also list of modules which failed.  The right frame shows summary of results in several levels.

The shortest way to see the reason of failure is to click on module name in Failures report in the left navigation frame. Then you can see name of failed test case and error message. If you need more details, click on its name. You will see test cases for whole test suite and failed test in red color. To see exception stack trace follow the link in Status column. There might be also additional proprietary log files in working directory of the test case. In that case there should be "Yes" link in Workdir column. For UI functional tests (module ide) are useful jemmy.log with trace messages of performed operations and screen.png containing B&W screen shot at the time of the failure.

If you are not able to identify reason of failure, or you think failure is not caused by your changes, please write an email to .

How to quickly find useful things from results main page

• messages.log and similar
  
• click on module name, then testbag name - there should be link IDE Logs which points to messages.log
• ant logs
  
• follow the link Logs from build scripts - you can find there compilation errors etc.
  
• test case error message
• click on module name, then testbag name, suite name - you will see list of executed test cases and failed one in red color with error message
• exception stack trace
• click on module name, then testbag name, suite name, link in Status column  - you will see stack trace of exception which caused that test failed
  
• jemmy.log
• click on module name, then testbag name, suite name, link in Workdir column, jemmy.log file - it contains trace messages of performed operations and it can be useful to know the last operation before the failure
• screen.png
• click on module name, then testbag name, suite name, link in Workdir column, screen.png file - it contains B&W screen shot at the time of the failure
  

Add a new test case into commit validation suite

To add a test case into the commit validation suite you have to modify module's tests config file and also xtest's master config if module is not already included in the suite. If you need more information about XTest, please visit the http://xtest.netbeans.org/ site.

Reasons to add a test

The commitValidation suite shall serve as a global check suite, that will try to catch possible regressions spread over wide range of functionality. For example a modification in org.openide.text.Line can affect the correctness of debugging of JSPs. It would be the best to write correct unit test of the Line but if that is not possible, than the JSP module developer is encouraged to add a test for the correct JSP debugging functionality into the commitValidation tests of the JSP module. That way he will help the core developer recognize that there is something wrong before the commit is done.

Another example might be related to correct garbage collection. The whole system is designed to free itself from memory when not used, but small mistake in any module can prevent this feature to function correctly. As this can be broken by any commit in any module, this is a reasonable test candidate for a commitValidation suite.

In no way the commitValidation is a replacement for unit tests. Each module is responsible for its own set of tests for basic functionality that shall be executed before any change of the module code. The commitValidation suite cannot contain such tests as it would not scale and we need it to execute quickly as it shall be run before every commit. Think about that before extending it - if it is not reasonable to provide validation of an important functionality on a level of unit tests, then commitValidation is the right place. But try to minimize such cases for the sake of scalability.

Modify module's config file

Create a new test bag with attribute commit and add test cases in it which you want to be executed in commit validation suite. The test bag should look like this:

    <testbag testattribs="commit" executor="code" name="Before Commit Validation">
        <testset dir="unit/src">
            <patternset>
                <include name="org/openide/loaders/*Folder*.class"/>               
                <include name="org/openide/windows/*Test.class"/>
            </patternset>
        </testset>     
    </testbag>

Above example is taken from openide/test/cfg-unit.xml in openide module. If test bag with commit attribute already exists in config file, you only need to add your test cases by adding an appropriate include element.

Modify XTest's master config file

You have to modify master config file only if your module is not already included in commit validation suite. In that case simply add a line to config named commit-validation-nb. Master config is located in xtest/instance/master-config.xml and commit validation part looks like this:

  <config name="commit-validation-nb">
      <module name="core" testtypes="unit" attributes="commit"/>
      <module name="openide" testtypes="unit" attributes="commit"/>
      <module name="ide" testtypes="qa-functional" attributes="commit"/>
  </config>

Tips and tricks

• If you participate in a development, you should be subscribed to . Failures in a daily builds as well as continuous ones are notified there and if your integration breaks something, you can react quickly. If you are going to fix such problem please reply to the mail, so others know that it is being solved.

• Test and test tools are also just a software this means if you find bug in them please fill it. If you don't now which Issuezilla module it is then go to http://testtools.netbeans.org/ where you can find more info,

• If you don't know how to evaluate commit validation suite results, read above.
  

• The automatic tests opens the IDE and popup and close various windows which prevents me from work. How can I prevent it? Easy on Unix - start new X server (Xnest is useful) and execute the validation test there. Using the full-build-linux.sh driver script, just pass spawndisplay=yes.

• Usually one is commiting change that has a record in Issuezilla (bug, some task, enhancement). In such case mention the issue number in the commit message.

  hg ci -m '#31043: Fixing the screen resolution'

  and include the changeset hash of the commit into the description field when resolving the issue. Then one can it is possible to find out more about the commit in the Issuezilla and it is possible to find the sources that were modified to resolve the issue.

Common mistakes

• Do not check in debug code that depends on a modified API with something made public that isn't Solution: Always do a sanity diff

• Check in a class that also contains a change that's part of a different bug fix Solution: Always do a sanity diff