imstat bug

Jason Quinn wrote on Jun 20, 2008

The imstat task gives incorrect answers when combing the upper and lower parameters with clipping. Here is an extreme example:

PACKAGE = imutil
   TASK = imstatistics

images  =              dev$pix  List of input images
(fields = image,npix,mean,stddev,min,max) Fields to be printed
(lower  =                   0.) Lower limit for pixel values
(upper  =                   0.) Upper limit for pixel values
(nclip  =                    1) Number of clipping iterations
(lsigma =                   3.) Lower side clipping factor in sigma
(usigma =                   3.) Upper side clipping factor in sigma
(binwidt=                  0.1) Bin width of histogram in sigma
(format =                  yes) Format output and print column labels ?
(cache  =                   no) Cache image in memory ?
(mode   =                   ql)

gives

#               IMAGE      NPIX      MEAN    STDDEV       MIN       MAX
              dev$pix    262144     108.3     131.3       -1.    19936.

With upper and lower set to zero, there should be no pixels left to do the statistics. You could even set upper to be less than lower and it will produce the same. The output is correct if the number if clipping iterations is set to "0".

I'd imagine the behavior that is desired when mixing upper and lower with clipping is that data points outside of upper and lower are immediately thrown away, then afterwards the iterative clipping routines are applied to whatever data remains.

My guess is the problem arises because of "one-off" style errors when computing histogram boxes because as you narrow the upper-to-lower window the output appears to make sense (that is, the number of pixels used for computation decreases) until you hit some threshold and all of a sudden the number of pixels "jumps up".

Jason

---

Mike Fitzpatrick wrote on Jun 20, 2008

Hi Jason,

I was gonna let Frank handle this one but he is out this week. You should hear from him when he returns.

-Mike

P.S. I'm leaving some of your other questions for Frank as well, sorry for the delay.

---

jcadien1 wrote on Jun 20, 2008

Jason,
One aspect of imstat I have noted over the years is the effect binwidth, has on the median estimate. I see that yours is set to the default 0.1. I always set mine to 0.0001. Smaller will work, but the computaional time becomes extreme. Try diiferent values for binwidth and you will see what I mean. Whether this will help your particular problem I am not sure.
Jim

---

Francisco Valdes wrote on Jun 20, 2008

Hi Jason,

Thanks for pointing this out. The source of the problem is that the clipping step resets the limits without regard to the user supplied limits. The correct method is to bound any clipping limit adjustments by the user supplied limits. This potentially has a wider impact than the case you reported where the limits simply eliminate all pixels. You are correct that the expected behavior should be that any pixels outside the user specified limits should never be used in the statistics.

This has been fixed for the next release.

Yours,
Frank Valdes

NUMBER: 561
MODULE: imstatistics, mimstat
SYSTEM: V2.13
DATE:   Tue Jul 15 09:59:20 MST 2008
FROM:   valdes

BUG:    The user supplied upper and lower data limits may be exceeded if
        clipping is used.  For instance, if the limits exclude all pixels
        the clipping with behave as if no pixels are excluded.  If not
        all pixels are excluded and the clipping thresholds computed
        (number of sigma * standard deviation) exceed the limit then
        the clipping threshold is used.  The only workaround is to not
        use clipping if the limit values are supplied or check first
        with no clipping to see if all pixels are excluded.

STATUS: Fixed for the next release.

---

Last post on Jun 20, 2008