In this assignment you will implement and experiment with an interactive image matting tool. This tool was demonstrated in class in Lecture 1 and will be discussed in detail next week. Apart from implementing a specific matting technique called Triangulation Matting, the interface you will work with will also serve as an image manipulation front-end for future assignments. In fact, each assignment will add more functionalities to this front end.
Right now, the goal of this assignment is two-fold:
- the FLTK user interface design library and the FLUID user interface builder software
- using the CDF Linux machines to write user interface code
- using the VXL image processing library to read/write images, display them on the screen through the user interface and access their pixel information
- solving linear systems using some basic numerical routines in VXL
While the due date for the assignment is 2 weeks away, it is strongly advised that you try and complete Part A.1 of the assignment within the next few days and use the rest of your time for Part A.2 and Part B. It will take some time to familiarize yourself with the FLTK programming model, the VXL library, and the helper code that I have already written, so start early!!! Once you "get the hang of it," the programming part of the assignment should not be that hard and there is very little coding for you to do. But it will take you quite some time to internalize exactly what you have to do, and how.
NEW: At the bottom of this page you will find information about how to work on your assignment in VisualC++.The technique we will implement is based on a paper by Smith and Blinn that appeared in SIGGRAPH'96. You do not need to read or look at this paper (it's not particularly well written...).
We are providing you with a full-blown implementation of the tool in executable form, along with a suite of test images, so that you can run it yourself and see how it works (see the executables partA-2/bin/VisComp_full.exe and partA-2/bin/viscomp_full in the assignment tarfile). The program should work for a variety of image formats (tif, jpeg, etc). I suggest you use very small images to test things out first. In fact, the full-blown executable will only accept images whose largest dimension is less than 512 pixels. When you write your own version, you can change the size of the maximum displayable image in the supplied source code (there is documentation about that in the code). Also, if you are running the tool on CDF but working remotely from a home machine, the tool will not run if your X server does not support the GLX extension.
Briefly, the tool works by first loading pictures of an object in front of two different backgrounds (they can be found in the partA-2/test_images sub-directories):
You can think of these photos, called Composite 1 and Composite 2, as being the superposition of two different background images with a single "foreground" image of the object.
Next, we load two more photos, called Background 1 and Background 2, that show only the backgrounds:
From this set of four photos, the matting algorithm computes two images: an Alpha Matte, that tells us which pixels in Composite 1 and Composite 2 correspond to the foreground object and also estimate the "transparency" of those pixels and an Object photo, that is a photo of the foreground object only (ie. all background information has been removed):
The Alpha Matte is a grayscale image, where the grayscale value indicates transparency (0 is fully transparent, 1 is fully opaque). For example, the pixels of the glass vase have small grayscale values, indicating that the vase is semi-transparent, while the flowers have alpha close to 1, indicating that the flower petals are not transparent at all. You can zoom in to get a better sense of the alpha values for different parts of the image by changing the zoom settings in the UI.
Once the Alpha Matte and Object photos are computed, you can create new image composites through superposition of the Object photo with a new background photo, according to the computed Alpha Matte:
Note how the background image appears to "show through" the vase.
The supplied program can be run in interactive mode (ie. UI windows are displayed and user works through menus and/or keyboard accelerators), in batch mode (ie. user supplies a set of command-line options specifying the input and output images names) or both (ie. some or all command-line options are specified and, after images are loaded, the UI windows are displayed as well). Type 'viscomp -help' to see the command line options.
For this assignment you will have to do the following:
I suggest you use the above order in tackling these tasks. Specifically:
The updated helper code is packaged into the tarfile matting.tar.gz. If you got a copy of the started code prior to Monday Jan 13 at 10pm, the binary viscomp_full will not run, and partA-2 will not compile. Please download this new version. The following sequence of commands creates a directory called CS320 in your home directory on CDF and unpacks the code:
> cd ~
> tar xvfz matting.tar.gz
> rm matting.tar.gz
This will create the directory ~/CS320 along with files and subdirectories needed for the assignment. All the code that you turn in should be in those directories as well, exactly as specified in the details below.
This part of the assignment will help you get familiar with the FLTK toolkit and FLUID, a program for interactively creating user interfaces.
- Dot file initialization
In order to use any of the toolkits required for the course, you need to set up your Unix environment. To do this, you will need to modify your .cshrc file so that it correctly defines certain environment variables. A sample file can be found in directory 320/Matting/partA-1. You can take a look at it and then use it modify your .cshrc file, located in your home directory, accordingly.
After you have modified your .cshrc file, execute the following commands at the shell to make the new settings take effect:
> source .cshrc
> rehashTo test whether your paths are now set correctly, try running the command
> fluid
You should now see the main window of fluid on your screen. If you get an error instead (e.g., "command not found"), your paths are still not set up correctly. In this case, it is strongly advised that you replace your own .cshrc file with the sample file and try again. If things still don't work, make sure you are working on a CDF machine running Linux.
- The FLTK and FLUID documentation
One of your tasks in this assignment is to get as familiar as possible with FLTK and FLUID. Fortunately, there is a wealth of on-line documentation about these tools. The starting point for the documentation is here.
Note that there is A LOT of documentation here, so it is important to take it easy and be selective about what you read. As you become more and more familiar with these tools throughout the course, you can venture into some of the more advanced topics covered in the manual.
The first thing to read are Chapters 1 and 2 of the FLTK documentation. In Chapter 1, ignore all sections about "Building and Installing FLTK Under ...". All software has already been installed so you don't have to worry about all this.
You should become thoroughly familiar with the "First FLTK Program" discussed in Chapter 2. It is strongly advised that you copy the program into the directory 320/Matting/partA-1 (call the program hello.cxx), compile it, and run it.
To do the compilation, you can use the Makefile located in that directory simply type
> make hello
This should run the compiler with all the necessary flags and create an executable that you can run from your shell. While you are not required to turn in anything related to this program, it is highly unlikely that you will be able to complete the rest of the assignment if you cannot get through this stage. (Note: if you want to see what the command 'make hello' really does, type 'make -n hello' and this will display the sequence of compilation steps that are executed.)
UPDATE: The supplied makefile in the directory /CS320/Matting/partA-1/ is designed for the CubeView program. To compile hello using the same makefile you must add the following section:
hello: hello.o
$(CC) -o hello hello.o $(LDFLAGS)
which is quite similar to the linker section of CubeView (see file for details). The key is to place the libraries after the executable and the object file names (common practice, details here).
- Building a sample interactive application with FLUID
Now that you've completed Chapter 2 and ran the sample program, you are ready to dive into fluid, which is the main tool we will be using to create user interfaces. You will learn the basics of fluid by creating, step-by-step, a simple interactive application that is described in detail in Chapter 9 of the FLTK manual.
At this point, you should skip Chapters 3-8 of the FLTK manual and go straight to Chapter 9. The chapter contains a brief tutorial that guides you step-by-step through the construction of a simple graphical viewer called CubeView.
The main task in Part A.1 of the assignment will be to create a fully-functional version of the CubeView program. To help you with this process, we are providing a partially-completed version of the program that you will have to finish up yourselves, guided by the tutorial in Chapter 9.
Step 1: Notice the files CubeView.h, CubeView.cxx , CubeMain.cxx and Makefile in the "partI" subdirectory. These files provide the partial code you will need to get a complete implementation of CubeView running. Compile them by typing
> 320/Matting/partA-1
> makeand make sure that the only errors generated are because of the CubeViewUI.cxx file that you haven't created yet.
Step 2: Run 'fluid' from the 320/Matting/partA-1 directory and go through all steps in the Chapter 9 tutorial (you can skip the section called "The CubeView Class" for now). Note that all the code under the heading "The CubeView Class" is included in the helper files you downloaded so you don't need to type it in yourself. Basically, your main task will be to run fluid and use it to create the CubeViewUI class and associated files, as described in the chapter. Once you are done following all the instructions in the tutorial, be sure to save your work by selecting File->Save As, naming the output file CubeViewUI.fl, and saving this file in the partA-1 directory. Naming is important so be sure to save the file with this exact name. To generate the code implementing the CubeViewUI class, select the File->Write Code menu option in fluid. Again, make sure that all the files you create go in the partA-1 directory.
Once fluid has generated the code implementing the class, you need to compile it by running 'make CubeViewUI.o' and making sure that it compiles correctly.
Step 3: You are now ready to create an executable version of the program. To do this you need to link together the CubeMain.o CubeViewUI.o and CubeView.o object files to create an executable called CubeView. The supplied Makefile should do this for you by running 'make' again. You are done!
- What we will be looking for in the 320/Matting/partA-1 directory:
- The CubeView executable
- That the CubeView executable and related object files can be compiled/created by just typing 'make' at the Unix prompt while in that directory
- The file CubeViewUI.fl that you created with fluid
The goal of this part of the assignment is to implement the triangulation matting algorithm to be discussed in class. The algorithm takes four RGB images as input (Composite 1, Background 1, Composite 2, Background 2) and produces two images as output---a grayscale Alpha Matte, and an RGB Object image.
viscomp -no_gui -mback1 <back1> -mback2 <back2> -mcomp1 <comp1> -mcomp2 <comp2> -mobj <obj> -malpha <alpha>
this will save the results in two images, <alpha> and <obj>.
-mnewback <newback> -mnewcomp <newcomp>
The last step in PartA of your assignment is to extend the "incomplete" user interface of your starter implementation so that it matches the functionality of the full version. The starter implementation has the following components missing (you can run both versions to verify what needs to be implemented):
Both of these functionalities should be implemented through fluid by running fluid on the file 320/Matting/partA-2/src/VisCompUI.fl. As stated in the README.1st file, any callbacks you implement must be placed in the student_callbacks directory.
This part of the assignment is meant to assess your detailed understanding of the matting algorithm and to have you perform some matting experiments of your own!!
Use your own digital camera to capture a set of images suitable for the triangulation matting procedure, and run the algorithm on those images.
Specifically, you need to
Note that the question of how you acquire your 5 input images will require some thinking/planning on your part!! The number of points you get for this part of the assignment will be based on how good your matting results are or, if they aren't, on your assessment of why it was not possible to get better results.
If you do not have a digital camera and cannot borrow one from a friend, let me or the TAs know RIGHT AWAY and we will try to make suitable arrangements.
The triangulation matting algorithm does not always behave as one would expect from a quick glance of the photos at the beginning of this assignment! Here you need to answer a set of questions that examine what can go wrong and, more importantly, why.
You can find the questions in the file 320/Matting/partB/Written.txt (also available here). Write all your answers in that file.
Once you are done with the above, edit the file 320/CHECKLIST.txt (also available here) to specify which components of the assignment you have completed, along with notes about parts that you were not able to complete, if any.
Pack up your portion of the code with the following commands:
> cd ~/CS320
> tar cvfz assign1.tar.gz Matting/CHECKLIST.txt Matting/partA-1/* Matting/partB/{WRITTEN.txt,*.jpg} Matting/partA-2/bin/viscomp Matting/partA-2/src/{VisCompUI.fl,Makefile,ADDITIONS} Matting/partA-2/src/matting Matting/partA-2/src/student_callbacks
Finally, you should use CDF's assignment submission system to submit your assignment:
> submit -c csc320h -a Assign1 assign1.tar.gz
Note that the system has been configured (1) to accept only files whose name is assign1.tar.gz and (2) to not accept submissions that are more than 4 days late. Just do 'man submit' at the unix prompt to get more info on this process.
In evaluating your assignment, the first thing we will look for are the files CHECKLIST.txt and WRITTEN.txt.
Then we will make sure that your code compiles (just by typing "make" in the partA-1/ and partA-2/src directories).
Then we will run your code on test examples in both the interactive and the non-interactive modes.
Finally, we will look at your code. It must be well commented: if the TA has doubts about the correctness of a specific implementational feature and there is not enough documentation/comments for the TA to decide, you will lose points.
In this course, we will be using the VXL library for representing and manipulating images. This is a library developed by dozens of computer vision researchers over many years, so it has very broad coverage in terms of functionality. As a consequence, the documentation is quite extensive. The best place to start is to look at Sam Hasinoff's old 320 page (from a previous instance of the course). There you will find introductory notes on VXL, pointers to the top-level documentation tree, as well as pointers to the two VXL modules that are directly relevant to you for this assignment: the vil library for representing images and the vnl library for numerical computations. Your TA will cover the basic structure and organization of VXL, and go over simple image manipulation functions. Another good source of information is, of course, the starter code!
If you want to work in windows, and you got a copy of the starter code prior to this announcement, you must re-download the starter code. The new one has updated options that work with the new libraries.
However, your implementation MUST run on the Linux CDF machines and can only be submitted for linux.
Both fltk and VXL have been ported to windows, making the choice of operating system completely transparent: the code I am providing compiles without any changes on both Linux and Windows 7 (using VisualC++ 8). You should, in principle, be able to work in Visual C++, copy your code to your CS320 directory on CDF, compile using the supplied linux makefile, and run. Give yourself some extra time if you choose to do this. Ssometimes, bugs that do not seem to affect a program in VC++ cause coredumps on linux (and vice-versa).
If you plan to work with VC++ download the file 320CodeLib.zip (1.8GB !!!, almost 10 GB uncompressed!!). It contains the entire compiled VXL and FLTK distributions for 64-bit machines. This should work on your Windows computer with VC++ 8.0 (2005) installed, but I will not offer any support if you encounter problems with it. Alternative you may download sources for both VXL and fltk and compile them yourself. This distribution contains the latest libraries but older ones have worked. Once you decompress this library, add a system variable called 320CodeLibPath that points to the root of the decompressed files (instructions below) and the Visual Studio project should compile without issues. The provided code may work with more recent releases of Visual C++/Visual Studio and with Windows 8 but there are no warantees on that end either.
To create a user system variable hit the "windows" key and type the word 'path' (without the quotes). This will open the Environment Variables window. Then press new (green box in the image below) and enter the name '320CodeLibPath' (without quotes) and the path where you decompressed the libraries.