Overscan subtraction: line-by-line vs fitting
Frank Winkler wrote on Mar 10, 2008
For Arcon data especially, there are low-level frame-to-frame variations in the overscan which vary significantly from line to line. In bias frames, these variations seem consistent all across the lines from a particular amplifier. If doing overscan fitting, it would require an absurdly high order to take them out, since they occur on very short spatial scales. I think the right thing to do is simply to average the overscan on a particular line, and subtract that from the data, on a line-by-line basis. I'm pretty sure I've done this in the past, but forget how to do it. One might expect an option in the "functio" parameter in quadproc (or ccdproc) of "none" for no curve fitting, but that doesn't seem to be among the choices. Can you help please?
Thanks!
Thanks!
Francisco Valdes wrote on Mar 10, 2008
Hola Paco,
The option you want is "minmax" which you set in the function parameter for CCDPROC, etc. This takes the overscan pixels for each line, throws out the high and low as possibly cosmic rays and transition, averages the reset, and subtracts that number for that line. This works well for the type of behavior you mention. You may still see a little stripeyness in the low count level data (zeros) because of the low statistics but it should be insignificant for science data. You are completely right that you don't want to use a high order fitting for this type of situation.
Yours,
Frank Valdes
The option you want is "minmax" which you set in the function parameter for CCDPROC, etc. This takes the overscan pixels for each line, throws out the high and low as possibly cosmic rays and transition, averages the reset, and subtracts that number for that line. This works well for the type of behavior you mention. You may still see a little stripeyness in the low count level data (zeros) because of the low statistics but it should be insignificant for science data. You are completely right that you don't want to use a high order fitting for this type of situation.
Yours,
Frank Valdes
Francisco Valdes wrote on Mar 10, 2008
For posterity ...
The line-by-line overscan options are not available for quad format data. There is a simple workaround described below. The final disposition is that in the next release of IRAF and the QUADRED package the capability will be present using the QUADPROC task though not the quad version of CCDPROC.
Frank Valdes
===== Try not to get confused by the two parties both being Frank =====
===== The following discussion is in reverse chronological order =======
Hi Frank,
Well I guess the quad format at CTIO will be around for a while longer! I am adding support for
line-by-line overscan as well as potential other features found in the current CCDPROC but not
found in the quad version of CCDPROC.
I looked at updating the quad version of CCDPROC, which was written with this in mind. However, the changes were too extensive and not simple to merge with the quad modifications.
So I changed things more along the lines of the workaround. So now QUADPROC is no longer as "obsoleted" since it uses the current CCDPROC on each piece to get new features from
CCDPROC. There is no need, of course, to require users to do the "redefine" workaround. I also made a modification so the log output shows the overscan function used. Note that I didn't remember to mention all the possibilities earlier in that there are actually three options for the line-by-line overscan (mean, median, and minmax). So you could have used a median but I still highly recommend minmax.
Regards,
Frank
On Mar 12, 2008, at 4:15 AM, Frank Winkler wrote:
Hi again Frank,
Just wanted to let you know that I processed all the data from our run through last night using minmax, and it worked like a charm. The subtraction looks significantly better than when I was trying to fit it. I suppose that at some point you will make the version of ccdproc that resides in quadred match the updated version (in ccdred?) so the re-definition will not be necessary?
One final point: the logfile, and image header, have lines that look like this:
OVERSC11= 'Mar 11 22:38 Overscan section is [1055:1108,1:1023] with mean=699.33'
i.e., no explicit indication of how the overscan subtraction was done (minmax or otherwise). This does not matter a whit to me, but if you think it does to someone, thought I'd let you know.
Many thanks for your help!
Frank W
On Mar 11, 2008, at 4:13 PM, Frank Valdes wrote:
Hi Frank,
That sounds good. The log file should have an indication that overscan was subtracted with "minmax". I had never thought about interactive=yes for this algorithm which is non-interactive. You probably should have interactive=no unless you really want to fit the overscan.
You should read the help page for ccdproc, of course. But to give you the direct answer, the minmax overscan subtraction does not use any other parameters related to the overscan fitting. So once you set function=minmax the setting of the other parameters doesn't matter.
minmax is simply the following. At each line take all the pixels in the overscan for that line, through out the highest and lowest, average the rest, and subtract the average from every data pixel on that line.
Frank
On Mar 11, 2008, at 12:22 PM, Frank Winkler wrote:
OK, here goes.
Even though I have set interactive = yes, and when it asks if I want to fit the overscan for the first quadrant interactively I hit yes, I do not get the icfit graphics window coming up. Not surprisingly for this algorithm, since there is really nothing to fit.
And, a few seconds later, it finishes normally, and ... apparently success!!
The line-by-line pattern noise has now largely disappeared.
So I think you've done it! Now, just a couple of questions about parameters in quadproc. Do "low_rej" and "high_rej" now refer to how many high and low pixels to reject from each LINE? Etc. for "grow?" And "niterat:" if set to more than 1, does it make a second (or more) pass through rejecting even more pixels? With no interaction, it's hard to judge these; what's your recommendation about how to set them? Whatever it did with the default settings seems pretty good.
Thanks very much!
Frank
On Mar 11, 2008, at 1:30 PM, Frank Valdes wrote:
Hi Frank,
Well it turns out that quadproc actually calls the version of ccdproc in quadred and that version is older than the current ccdproc which has the "minmax" capability. So indeed my suggestion did not really change anything.
I have an idea which has a good chance of working but you will have to be my guinea pig to test it since I don't have an easy way to do it. The idea is quite easy to try!
1. Log into IRAF and load the quadred package as you have been doing.
2. Type the following
quadred> redefine ccdproc = quadsrc$x_ccdred.e
3. Now run quadproc as before.
What is happening is that quadproc is actually a script that breaks up the Arcon format and calls ccdproc on each piece and finally puts things together again. The change should then be to let quadproc run the latest ccdproc but using the parameters in the quadred package. The only thing that could go wrong is that there is a mismatch in parameters. I don't think this should be the case but if it is I think we can make an easy fix.
Let me know how it goes.
Yours,
Frank
On Mar 11, 2008, at 4:01 AM, Frank Winkler wrote:
Hi Frank,
Well, I am sad to report that quadred.ccdproc seems to behave in precisely the same way as quadproc. Does not seem to recognize function = minmax either. I do see from the "help quadred" page that quadred.ccdproc is now preferred over quadproc, so I'll plan to use that in future. But that does not solve my little problem. (If it matters, I'm running IRAF 2.23beta2 on my Mac; have not gotten round to installing 2.14).
It is not a very big deal, since the sort of line-by-line variations I see are less than the read noise, but since I can see it there in the images--at least for bias frames--I'd like to get rid of it.
Thanks very much,
Frank
On Mar 10, 2008, at 11:51 PM, Frank Valdes wrote:
Hi Frank,
QUADPROC was something I did not write. I do try and support but there are things that don't work quite like you expect. There is a better task to use. You load the QUADRED package as before. But in that package is a version of CCDPROC which is designed to work with the funky Arcon quad format. In that task you should be able to use function=minmax. Try typing "help quadred" to see the recommendation.
What Chris was remembering has to do with quick "on-the-fly" calibration. It is not something for reductions nor does it work with the quad format.
Good luck and best wishes,
Francisco/Frank
On Mar 10, 2008, at 12:10 PM, Frank Winkler wrote:
Hi Frank,
Thanks for your quick answer. This answer is very involved, so I'm doing it off-line from the forum; let me know if it's better to do it the other way.
I guess I am a bit confused about what's going on here. First, the "function" param for quadproc does not even mention "minmax" as an option, but it does accept it in eparam. But then running quadproc/ccdproc interactively puts me into the usual icfit graphics window, where some value for each line is plotted, as a function of line number (y) across one quadrant of the chip. And there is some sort of an attempt to fit that shown by a dashed line. And the parameter list at the top shows that it's fitting with a "spline3." I have done an "unlearn quadproc" a couple of times, re-entered minmax as the function type, and the same thing keeps happening. The result in the end appears virtually, if not entirely identical to fitting the thing with a constant (e.g. Legendre with order = 1). None of the line-by-line structure has been taken out, contrary to what I want/expect. Here's what my parameter file for quadproc looks like BEFORE running the task:
PACKAGE = quadred
TASK = quadproc
images = y*.fits List of CCD images to correct
(ccdtype= ) CCD image type to correct
(max_cac= 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(fixpix = no) Fix bad CCD lines and columns?
(oversca= yes) Apply overscan strip correction?
(trim = yes) Trim the image?
(zerocor= no) Apply zero level correction?
(darkcor= no) Apply dark count correction?
(flatcor= no) Apply flat field correction?
(illumco= no) Apply illumination correction?
(fringec= no) Apply fringe correction?
(readcor= no) Convert zero level image to readout correction?
(scancor= no) Convert flat field image to scan correction?
(readaxi= line) Read out axis (column|line)
(fixfile= ) File describing the bad lines and columns
(xskip1 = INDEF) X pixels to skip at start of overscan
(xskip2 = INDEF) X pixels to skip at end of overscan
(xtrim1 = INDEF) X pixels to trim at start of data
(xtrim2 = INDEF) X pixels to trim at end of data
(ytrim1 = INDEF) Y pixels to trim at start of data
(ytrim2 = INDEF) Y pixels to trim at end of data
(zero = ) Zero level calibration image
(dark = ) Dark count calibration image
(flat = ) Flat field images
(illum = ) Illumination correction images
(fringe = ) Fringe correction images
(minrepl= 1.) Minimum flat field value
(scantyp= shortscan) Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines
(interac= yes) Fit overscan interactively?
(functio= minmax) Fitting function
(order = 1) Number of polynomial terms or spline pieces
(sample = *) Sample points to fit
(naverag= 1) Number of sample points to combine
(niterat= 1) Number of rejection iterations
(low_rej= 3.) Low sigma rejection factor
(high_re= 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = ql)
Then AFTER running quadproc, simply accepting the fits for each amplifier, what it looks like is:
... same until ...
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 1) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
i.e., it simply won't accept "minmax." I have also tried running non-interactively, but the result looks the same.
Frustrated at CTIO ...
Frank/Paco
The line-by-line overscan options are not available for quad format data. There is a simple workaround described below. The final disposition is that in the next release of IRAF and the QUADRED package the capability will be present using the QUADPROC task though not the quad version of CCDPROC.
Frank Valdes
===== Try not to get confused by the two parties both being Frank =====
===== The following discussion is in reverse chronological order =======
Hi Frank,
Well I guess the quad format at CTIO will be around for a while longer! I am adding support for
line-by-line overscan as well as potential other features found in the current CCDPROC but not
found in the quad version of CCDPROC.
I looked at updating the quad version of CCDPROC, which was written with this in mind. However, the changes were too extensive and not simple to merge with the quad modifications.
So I changed things more along the lines of the workaround. So now QUADPROC is no longer as "obsoleted" since it uses the current CCDPROC on each piece to get new features from
CCDPROC. There is no need, of course, to require users to do the "redefine" workaround. I also made a modification so the log output shows the overscan function used. Note that I didn't remember to mention all the possibilities earlier in that there are actually three options for the line-by-line overscan (mean, median, and minmax). So you could have used a median but I still highly recommend minmax.
Regards,
Frank
On Mar 12, 2008, at 4:15 AM, Frank Winkler wrote:
Hi again Frank,
Just wanted to let you know that I processed all the data from our run through last night using minmax, and it worked like a charm. The subtraction looks significantly better than when I was trying to fit it. I suppose that at some point you will make the version of ccdproc that resides in quadred match the updated version (in ccdred?) so the re-definition will not be necessary?
One final point: the logfile, and image header, have lines that look like this:
OVERSC11= 'Mar 11 22:38 Overscan section is [1055:1108,1:1023] with mean=699.33'
i.e., no explicit indication of how the overscan subtraction was done (minmax or otherwise). This does not matter a whit to me, but if you think it does to someone, thought I'd let you know.
Many thanks for your help!
Frank W
On Mar 11, 2008, at 4:13 PM, Frank Valdes wrote:
Hi Frank,
That sounds good. The log file should have an indication that overscan was subtracted with "minmax". I had never thought about interactive=yes for this algorithm which is non-interactive. You probably should have interactive=no unless you really want to fit the overscan.
You should read the help page for ccdproc, of course. But to give you the direct answer, the minmax overscan subtraction does not use any other parameters related to the overscan fitting. So once you set function=minmax the setting of the other parameters doesn't matter.
minmax is simply the following. At each line take all the pixels in the overscan for that line, through out the highest and lowest, average the rest, and subtract the average from every data pixel on that line.
Frank
On Mar 11, 2008, at 12:22 PM, Frank Winkler wrote:
OK, here goes.
Even though I have set interactive = yes, and when it asks if I want to fit the overscan for the first quadrant interactively I hit yes, I do not get the icfit graphics window coming up. Not surprisingly for this algorithm, since there is really nothing to fit.
And, a few seconds later, it finishes normally, and ... apparently success!!
The line-by-line pattern noise has now largely disappeared.
So I think you've done it! Now, just a couple of questions about parameters in quadproc. Do "low_rej" and "high_rej" now refer to how many high and low pixels to reject from each LINE? Etc. for "grow?" And "niterat:" if set to more than 1, does it make a second (or more) pass through rejecting even more pixels? With no interaction, it's hard to judge these; what's your recommendation about how to set them? Whatever it did with the default settings seems pretty good.
Thanks very much!
Frank
On Mar 11, 2008, at 1:30 PM, Frank Valdes wrote:
Hi Frank,
Well it turns out that quadproc actually calls the version of ccdproc in quadred and that version is older than the current ccdproc which has the "minmax" capability. So indeed my suggestion did not really change anything.
I have an idea which has a good chance of working but you will have to be my guinea pig to test it since I don't have an easy way to do it. The idea is quite easy to try!
1. Log into IRAF and load the quadred package as you have been doing.
2. Type the following
quadred> redefine ccdproc = quadsrc$x_ccdred.e
3. Now run quadproc as before.
What is happening is that quadproc is actually a script that breaks up the Arcon format and calls ccdproc on each piece and finally puts things together again. The change should then be to let quadproc run the latest ccdproc but using the parameters in the quadred package. The only thing that could go wrong is that there is a mismatch in parameters. I don't think this should be the case but if it is I think we can make an easy fix.
Let me know how it goes.
Yours,
Frank
On Mar 11, 2008, at 4:01 AM, Frank Winkler wrote:
Hi Frank,
Well, I am sad to report that quadred.ccdproc seems to behave in precisely the same way as quadproc. Does not seem to recognize function = minmax either. I do see from the "help quadred" page that quadred.ccdproc is now preferred over quadproc, so I'll plan to use that in future. But that does not solve my little problem. (If it matters, I'm running IRAF 2.23beta2 on my Mac; have not gotten round to installing 2.14).
It is not a very big deal, since the sort of line-by-line variations I see are less than the read noise, but since I can see it there in the images--at least for bias frames--I'd like to get rid of it.
Thanks very much,
Frank
On Mar 10, 2008, at 11:51 PM, Frank Valdes wrote:
Hi Frank,
QUADPROC was something I did not write. I do try and support but there are things that don't work quite like you expect. There is a better task to use. You load the QUADRED package as before. But in that package is a version of CCDPROC which is designed to work with the funky Arcon quad format. In that task you should be able to use function=minmax. Try typing "help quadred" to see the recommendation.
What Chris was remembering has to do with quick "on-the-fly" calibration. It is not something for reductions nor does it work with the quad format.
Good luck and best wishes,
Francisco/Frank
On Mar 10, 2008, at 12:10 PM, Frank Winkler wrote:
Hi Frank,
Thanks for your quick answer. This answer is very involved, so I'm doing it off-line from the forum; let me know if it's better to do it the other way.
I guess I am a bit confused about what's going on here. First, the "function" param for quadproc does not even mention "minmax" as an option, but it does accept it in eparam. But then running quadproc/ccdproc interactively puts me into the usual icfit graphics window, where some value for each line is plotted, as a function of line number (y) across one quadrant of the chip. And there is some sort of an attempt to fit that shown by a dashed line. And the parameter list at the top shows that it's fitting with a "spline3." I have done an "unlearn quadproc" a couple of times, re-entered minmax as the function type, and the same thing keeps happening. The result in the end appears virtually, if not entirely identical to fitting the thing with a constant (e.g. Legendre with order = 1). None of the line-by-line structure has been taken out, contrary to what I want/expect. Here's what my parameter file for quadproc looks like BEFORE running the task:
PACKAGE = quadred
TASK = quadproc
images = y*.fits List of CCD images to correct
(ccdtype= ) CCD image type to correct
(max_cac= 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(fixpix = no) Fix bad CCD lines and columns?
(oversca= yes) Apply overscan strip correction?
(trim = yes) Trim the image?
(zerocor= no) Apply zero level correction?
(darkcor= no) Apply dark count correction?
(flatcor= no) Apply flat field correction?
(illumco= no) Apply illumination correction?
(fringec= no) Apply fringe correction?
(readcor= no) Convert zero level image to readout correction?
(scancor= no) Convert flat field image to scan correction?
(readaxi= line) Read out axis (column|line)
(fixfile= ) File describing the bad lines and columns
(xskip1 = INDEF) X pixels to skip at start of overscan
(xskip2 = INDEF) X pixels to skip at end of overscan
(xtrim1 = INDEF) X pixels to trim at start of data
(xtrim2 = INDEF) X pixels to trim at end of data
(ytrim1 = INDEF) Y pixels to trim at start of data
(ytrim2 = INDEF) Y pixels to trim at end of data
(zero = ) Zero level calibration image
(dark = ) Dark count calibration image
(flat = ) Flat field images
(illum = ) Illumination correction images
(fringe = ) Fringe correction images
(minrepl= 1.) Minimum flat field value
(scantyp= shortscan) Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines
(interac= yes) Fit overscan interactively?
(functio= minmax) Fitting function
(order = 1) Number of polynomial terms or spline pieces
(sample = *) Sample points to fit
(naverag= 1) Number of sample points to combine
(niterat= 1) Number of rejection iterations
(low_rej= 3.) Low sigma rejection factor
(high_re= 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = ql)
Then AFTER running quadproc, simply accepting the fits for each amplifier, what it looks like is:
... same until ...
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 1) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
i.e., it simply won't accept "minmax." I have also tried running non-interactively, but the result looks the same.
Frustrated at CTIO ...
Frank/Paco
Last post on Mar 10, 2008