View source for CIBC:Seg3D:Documentation:Future

== Seg3D Future Work ==

Our goal is to get people making segmentations as soon as possible.

While Seg3D is in active development and the number of users continues to grow, our programming resources are limited.  As such, we will take an incremental user-based approach to software development.  We will continue to make frequent releases that add capabilities and fix bugs without rewriting major portions underneath.  We strive to make each release strictly better than the last one.  Make it work.  Fix it.  Make it easier.

The most important thing is to add the basic segmenting capabilities so that users can do things they could not otherwise accomplish.  It may be a little more time consuming than a fully automated approach, but they go from being unable to accomplish a task to being able to get something done with a reasonable amount of work.

The next major task is to fix any bugs and data integrity issues.  It doesn't help that the user can do something if the results go away or are incorrect.  Nothing wastes more time than doing a large manual segmentation and having the program crash an hour into it.

Another goal is to make the case studies/workflows more efficient and easy.  This means adding documentation, training, and clarifying UI elements and workflows so that new users can get up to speed more quickly and experienced users can segment more rapidly.

With these goals in mind, our priorities (in order) are:

* Fix all the known I/O limitations so that people can get their data in and out properly.
* Fix crash bugs
* Keep memory usage down.
* Fix non-crash bugs, including bad UI elements.
* Add in whichever filters would be appropriate for making specific segmentation tasks easier.
* Add builds for 64 bit OSX and Windows.
* Update the documentation and tutorials.


The tenative feature list for 1.10.1 is as follows:
* Add a GUI to Brightness/Contrast tool.
* Show Threshold Tool results in Volume Rendering window.

The long term 2.0 feature list:
* Fix the default transforms and add a 'Set to 3D View' button to the flip tool.
* Volume Rendering Improvements
** 1D colormap minimal hardware shader
** 1D colormap editors for both curve based and discreet segmentation color values
** Support multiple volumes
** Support masking
** Add the data + segmentation shader
* Improved Volume Rendering Tool integration
** Polyline in the volume rendering window for Osirix style clipping (and make labels)
** Move seed points in Volume Rendering window
* Convert the filters into a plugin architecture
* Move the layer editor to wxwidgets
* Support non-float data volumes (this may be a bandaid, really need 64 bit OSX and windows builds).
* Add segmentation objects (data volumes with discrete values)
** Add the morphology filters for preproccessing segmentations before meshing.

== 1. Current bug list ==

DB - NEU) There had been a problem in an earlier version that was fixed but that seems to have re-appeared in 1.10.0. Sila describes it as follows:

When I load the series of images an error window pops up.


ARS - UU) Conversion of DICOM -> ITK -> NRRD transform and spacing does always happen correctly. Is the problem in the DICOM -> ITK or ITK -> NRRD

== 2. Feature request list ==

=== On Going ===

BD - NEU) Any new features have at least a one-line explanation on the wiki --- I think she was looking at the intensity correction filter, which sounded like it might solve 2) above but she did not see how to use it?

  ARS - I have found the documentation to be lacking and plan to work on it as an on going project.

=== Completed ===

JB- UU) Flip Coronal (Flipping a view vs flipping the data). When we load in images, we always have to flip in the coronal plane to get the same view that other dicom viewers automatically give us.  This would be fine, but when we save out our segmentation, it is now flipped in the other dicom viewers.  I believe it was determined that the operation was acting on the wrong level.  Ideally there would be two scales, one for visualization, and another for modifications to the data

        ARS - I have added buttons to flip each of the orthogonal views. This flipping is strictly a view flipping and does not affect the data.

JB - UU) Flipping data via the spacing what should this really do?? Flip the data???

       ARS - Flipping the data just re-orders the data in memory. 

       ARS - Using a spacing that is negative re-orders the data via the view but does not affect the memory layout.  This is the same as flipping the view.

JB - UU) Preferences for a view. Rather than having to flip a view each time. Have preferences set to do it automatically with each data set

       ARS -  This would require the ability to save/change state which with the current GUI is not an easy task. So for now no state saving
                    in the session files or via preferences. Most other apps do not save the session state so this should not be a major blockage.

JB- UU) Data type (and value) in = Data Out. It appears that pixel values of segment data change from their original value.  For example, voxel (320 X 210 X 5) will have a value of 45000 in the raw data.  But when that pixel is masked out and saved as a dicom, it becomes 450.   Would you mind checking to see if you see a similar shift?  Also, I know that Darrel is experimenting with the dicom reader, and may have some insight into this problem.

      ARS - The nrrd write is correct but not the DICOM. The data is quantized because the internal rep is floats. I have added an option to use
      either the image min max or the max range 0-65535 for dicoms only.

JB- UU) Grayscale Arithmetic. With some image volumes that come naturally aligned, it would be nice to, for example, subtract one volume from another to enhance features.  This would aid in segmentation as well.

    ARS - This has been added in and allows all nrrd arithmetic operations (Same as those in SCIRun).

JB - UU) Invert black and white filter - many of the filters have a difficult time identifying regions of dark pixels.  For example, livewire does not appear to work when the seeds are placed along a black line.  If the image colors were inverted, would this allow the seeds to be placed on the white line and function normally?

    ARS - This has been added in and allows non labeled volumes to be inverted. The values are inverted within the min max bounds. So to do the operation invert the colormap, perform the operation, and then invert again.


DMW - UU) Look at the session file and the associated intermediate files. Preferences - zip everything, save intermediate files.

   ARS - We can do this but then what?? Maintaining state with the GUI is problematic which is why nothing but the current files are saved. Lets look at this later.

=== Completed ===

Colormapping/Histogram issues

DMW - UU) Ability to adjust the opacity of layers - especially for gray scale images which do not have a colormap. ARS -comment I would like propose that each layer have a colormap pop up this would also address the contrast - brightness issues as well as the histogram issues.

    ARS - I have added in an Opacity button in the layers. It will adjust the opacity for the orthogonal views only. 
    I will look into the colormap issue later.

DB - NEU) report out a numerical range (in terms of the image intensity value) when changing contrast and brightness. In other words, as you scroll to change these settings, have a window available to see what the current range is as it changes.  In addition to this being generally useful, we have a collaborator who works with CT data and in the CT world these numbers, the Hounsfield units, are well understood ---so much so that the device manufacturers like GE put presets on their workstations so that you can push a button to get the right presets (they call them "window and level" in that community) for lung or bone or liver or whatever ..... Longer term, it would be really nice to have a histogram with an interactive interface to do histogram stretching, but in the short term just telling the user where they are in terms of the range being visualized is essential.

     ARS - When changing the window/level via the mouse the new bounds are reported in the status bar.

     ARS - I have added a window/level tool that displays a histogram and allows the user to interactively set the window and level.

JB - UU) Histogram - There already exists a histogram equalization function, but there is no way to display is simple histogram.  A histogram would be very useful in conjunction with the threshold tool under the tools menu.  I am thinking of an indicator on the histogram that would display what values are being masked out.

   ARS - This has now been added. A histogram is displayed based on using 200 bins between the image min/max, the lower and upper bounds are displayed within the
   histogram.

JB- UU) Histogram. The threshold tool works great to isolate bright, or dark features within an image.  However, it can be difficult to choose a threshold.  Having a histogram of the image (excluding black pixels) often helps in determining where to set the threshold.  Would it be possible to display a histogram in association with this tool. or otherwise, that would show the characteristics of the image.  Ideally, a marker would indicate the location of the threshold on the histogram.

   ARS - Same comment - This has now been added. A histogram is displayed based on using 200 bins between the image min/max, the lower and upper bounds
   are displayed within the histogram.


=== Completed ===

Filter issues

DB- NEU)  finally, both our CT collaborator and a dermatology clinician had the same comment: once you have done a segmentation, they found it annoying to have the fill lines over the data. They wanted to be able, as they progressed, to go back and re-examine how they had done the segmentation, and the presence of the fill lines obscured the data enough to be an impediment to them. Of course, they do want to keep a visual representation of the *border* of the segmentation.

As I understand things, seg3d works in terms of label maps not boundaries, so even if using a boundary-oriented tool like the polyline segmenter, once you do the fill to label a region, the polyline boundary is gone. If that is a correct understanding, then this request sounds like it might be difficult to implement directly. But maybe there is some way to control the opacity of the fill with a slider or something, so that you could turn it down and up to better see the data in the segmented region ? or most likely you guys will have an even better idea of how to do this.

   ARS - There is now an opacity for the layers that for the most part addresses this issue.

JB- UU)  2D - 3D Filter Issues. The problems we are experiencing arise because our data is not acquired as cubic voxels.  Particularly when we run the binary dilate/ erode filter we notice that the filter expands too rapidly in the z direction compared with the XY plane.  We had discussed implementing a 2D / 3D switch that would allow the user to decide how the filter would be implemented.  This would likely be a useful feature on many of the filters already implemented.

    ARS - This has been implemented for the Dilate and Erode Filter. If others are useful we will need to redesign the whole filter class.

JB - UU) Zero crossing filter - many of the element for edge detection approaches already exist.  Adding this functionality would take us that much closer.

    ARS - I have added in the Canny Edge Detection which wraps everything into one filter.