SIMPLIFIED METHODS FOR THE NIPS 2003 FEATURE SELECTION CHALLENGE

Radford M. Neal, December 2004


This directory contains the programs and command scripts for a set of
post-workshop entries to the NIPS 2003 feature selection challenge.
The datasets and results of the feature selection challenge can be
found at

    http://www.nipsfsc.ecs.soton.ac.uk/

Two of these entries contained here, New-Bayes-nn-small and
New-Bayes-nn-large, are analogous to my original BayesNN-small and
BayesNN-large entries, which placed second and third overall in the
original ranking.  The new methods are somewhat simplified, and
modified in other ways.  The winner in the original ranking was the
BayesNN-DFT-combo entry, a joint entry with Jianguo Zhang, which was a
combination of some of the BayesNN-large methods with methods based on
Dirichlet diffusion trees.

Both the original and the new methods will be described in a chapter
in a volume on the feature selection challenge to be published soon.

These methods use my ``Software for Flexible Bayesian Modeling'',
which can be downloaded from my web page, at

    http://www.cs.toronto.edu/~radford/

This software is written in C, for Linux/Unix systems, though it
should also work on Windows machines using the Cygwin emulation
environment (see http://www.cygwin.com).  I used the version of
2004-11-10.

The preprocessing programs are written in the R language, free
implementations of which can be downloaded from

    http://www.r-project.org/

The version I used was R 1.6.0, but newer versions are probably also
fine.


Contents of this directory.

This directory contains various the following files:

    README              What you're reading now

    *.r                 Various R functions of general use

    make-sub-files.c    A C program used to create the files for submission 
                        to the challenge web site. 

    make-tar            A shell script for making a tar file for submission
                        to the challenge web site.

    all-pred            A shell script for running all the prediction
                        shell scripts for all the datasets.

    pred-copy           A shell script for copying prediction shell scripts. 
                        Can be ignored.

    setup-data          A shell script for setting up the data files to
                        begin.

    ndiffs              A shell script to count how many differences there 
                        are in the predictions in two .resu files.

There are also two subdirectories.  The "datasets" subdirectory contains
subdirectories for each of the four datasets.  Each such directory contains
the data files for that dataset, the R scripts for processing the dataset,
the shell scripts for running the methods on that dataset, and the files
of predictive probabilities as produced by the methods.  The "submissions"
subdirectory contains subdirectories for each of the methods.  Each such
subdirectory contains a "grab" shell script that produces the files to be
submitted to the challenge web site for that method, and a "desc" text file
with a brief description of the method.  These subdirectories also contain
the result files produced by "grab".


Reproducing the results.

The directory comes with all the results files that were produced by
the methods, but not with the "log" files from the MCMC runs of my
flexible Bayesian modeling software (which are big, and are
machine-architecture-specific).  To reproduce these result files, you
will need to do the following:

  1) Download the data files from the challenge web site, placing the
     data files (eg, arcene_train.data) in the corresponding subdirectories
     of the datasets subdirectory.

  2) Run the "setup-data" shell script in the main directory.

  3) Download R and my Software for Flexible Bayesian Modeling, if you
     don't have these already.

  4) Compile the make-sub-files program with the command

         cc make-sub-files.c -o make-sub-files

  5) For each of the five datasets, do the steps below.  These are
     illustrated for the "arcene" dataset, but the same is done for
     the other four datasets.

     a) Change into the directory datasets/arcene.

     b) Start up R, and enter the commands

            source("arcene1.r")
            source("arcene2.r")

        Then quit R, saving the workspace.  Note that, as commented in
        various places, some parts of some of these R scripts may take
        quite a long time (hours) to run and/or require lots of memory.

     c) Run all the .cmds shell scripts in the directory except the 
        *-red*.cmds scripts.  Some of these take a long time (days).

     d) Start up R, and enter the command

            source("arcene3.r")

        Then quit R, saving the workspace.

     e) Run the *-red*.cmds shell scripts.  These also can take a long 
        time.

  6) For each of the methods:

     a) Change to the directory for that method, in the "submissions" 
        subdirectory.

     b) Run the "grab" shell script.

     c) Run the ../../make-tar shell script.  This produces the 
        results.tar.gz file that was submitted to the challenge web site.

If you want to experiment with modifications of my methods, you can
produce new versions of the various .r or .cmds files and redo steps
(5) and (6) above as necessary.

Note that the R scripts like arcene2.r produce various plots.  During
development of the methods, I looked at these, and on that basis, and
on other examinations of the results, decided on the numbers of
features to select, which appear as constants later in arcene2.r.
Similarly, the plot produced by arcene3.r was consulted when deciding
how many features to choose later in arcene3.r.  So the final version
of these R scripts reflects results obtained with earlier versions.
