photometry prereduction script
H.J. van Heerden wrote on Feb 06, 2007
Hi all
Hope you all had a happy new year? :D
I am trying to put together a photometry pipeline for the Boyden Observatory. The package currently contains scripts to fix the fits headers, to do the batch photometry using daophot, and to dump the necessary data to textfile. (We try to do differential photometry, so "standerdizing" the data is not needed at this time, only want change in mag.)
What is still needed is a reliable reduction script. So Q?
Can I throw ccdproc into a script, define all the necessary param's and then run it, or is a bit of work needed beforehand. :?:
I.E. Do I need to use zerocombine and flatcombine to put together a bias and flat frame for ccdproc to use. (Can I run this in the script as well?)
What I am trying to get to is, I want as little "external" proccessing as possible. That is visiting astronomers shouldn't have to sit through a 50 page users manual to get their data, but just after a 5min preperation run each consegative script and in the end go home with the text file containing only the pertinant data they want.
Hope you can give me some advice on the matter.
Regards
Pat
Hope you all had a happy new year? :D
I am trying to put together a photometry pipeline for the Boyden Observatory. The package currently contains scripts to fix the fits headers, to do the batch photometry using daophot, and to dump the necessary data to textfile. (We try to do differential photometry, so "standerdizing" the data is not needed at this time, only want change in mag.)
What is still needed is a reliable reduction script. So Q?
Can I throw ccdproc into a script, define all the necessary param's and then run it, or is a bit of work needed beforehand. :?:
I.E. Do I need to use zerocombine and flatcombine to put together a bias and flat frame for ccdproc to use. (Can I run this in the script as well?)
What I am trying to get to is, I want as little "external" proccessing as possible. That is visiting astronomers shouldn't have to sit through a 50 page users manual to get their data, but just after a 5min preperation run each consegative script and in the end go home with the text file containing only the pertinant data they want.
Hope you can give me some advice on the matter.
Regards
Pat
Mike Fitzpatrick wrote on Feb 06, 2007
Hi Pat,
CCDPROC can do the zero and flat corrections as part of its normal processing, so long as you properly specify the zero and flat images to be used.
One way to create the type of pipeline you describe is to do the reduction you want by hand first, then use the HISTORY command to recall all the steps that you took. This can be redirected to a file you can edit to modify the filenames and then re-execute as something like "cl < myscript.cl" to run the commands. From there, the next step is to turn this history file into a procedure script where you parameterized the name of the file to reduce, the zero file, etc.
See the script programmer's guide for more about procedure scripts (i.e. http://iraf.net/irafdocs/script), or write back if you still have questions.
Cheers,
-Mike
CCDPROC can do the zero and flat corrections as part of its normal processing, so long as you properly specify the zero and flat images to be used.
One way to create the type of pipeline you describe is to do the reduction you want by hand first, then use the HISTORY command to recall all the steps that you took. This can be redirected to a file you can edit to modify the filenames and then re-execute as something like "cl < myscript.cl" to run the commands. From there, the next step is to turn this history file into a procedure script where you parameterized the name of the file to reduce, the zero file, etc.
See the script programmer's guide for more about procedure scripts (i.e. http://iraf.net/irafdocs/script), or write back if you still have questions.
Cheers,
-Mike
H.J. van Heerden wrote on Feb 06, 2007
Hi Mike
I've read Philip Massey's CCD Reduction guide. What he reccomends is that you use zerocombine and flatcombine to create a single bias and flat frame, from a set of zero and flat frames, for your specific dataset. These single frames are then passed to ccdproc. :?:
Another problem/Q is the setinstrument proc. I suspect that I would have to create a site specific translation file for our pipeline, not realy sure exactly how yet. Have read through the help file on how :?
One of my Q's that I have i.t.o. the above is; Can I save the translation.dat file in the same dir as the pipeline package dir, and call it from there. I.e. will setinstrument/ccdproc pick it up correctly. :)
The reason for all these Q's is that I have to do the pipeline development as a "major" part of my master's. I would like to get things as correct as possible. Don't want someone to tell me they can't give me my masters because the procedures I used was incorrect or ....
Thanx for all the help.
Pat
I've read Philip Massey's CCD Reduction guide. What he reccomends is that you use zerocombine and flatcombine to create a single bias and flat frame, from a set of zero and flat frames, for your specific dataset. These single frames are then passed to ccdproc. :?:
Another problem/Q is the setinstrument proc. I suspect that I would have to create a site specific translation file for our pipeline, not realy sure exactly how yet. Have read through the help file on how :?
One of my Q's that I have i.t.o. the above is; Can I save the translation.dat file in the same dir as the pipeline package dir, and call it from there. I.e. will setinstrument/ccdproc pick it up correctly. :)
The reason for all these Q's is that I have to do the pipeline development as a "major" part of my master's. I would like to get things as correct as possible. Don't want someone to tell me they can't give me my masters because the procedures I used was incorrect or ....
Thanx for all the help.
Pat
Mike Fitzpatrick wrote on Feb 06, 2007
Hi Pat,
Right, you pass them in using the 'zero' and 'flat' parameters to CCDPROC.
See the help page for a description or e.g. "cl> cd ccddb$kno" and have a look at some of the existing files for examples.
Just set the SETINSTRUMENT 'directory' parameter to be the directory of your pipeline code that contains the instrument file.
Cheers,
-Mike
I've read Philip Massey's CCD Reduction guide. What he reccomends is that you use zerocombine and flatcombine to create a single bias and flat frame, from a set of zero and flat frames, for your specific dataset. These single frames are then passed to ccdproc.
Right, you pass them in using the 'zero' and 'flat' parameters to CCDPROC.
Another problem/Q is the setinstrument proc. I suspect that I would have to create a site specific translation file for our pipeline, not realy sure exactly how yet. Have read through the help file on how
See the help page for a description or e.g. "cl> cd ccddb$kno" and have a look at some of the existing files for examples.
One of my Q's that I have i.t.o. the above is; Can I save the translation.dat file in the same dir as the pipeline package dir, and call it from there. I.e. will setinstrument/ccdproc pick it up correctly.
Just set the SETINSTRUMENT 'directory' parameter to be the directory of your pipeline code that contains the instrument file.
Cheers,
-Mike
H.J. van Heerden wrote on Feb 06, 2007
Hi
quick Q. Currently we use only skyflats. Not yet set up for dome flats.
Can I use flatcombine -> ccdproc
or must i use
mkskyflat -> combine -> ccdproc (to include illumination cor.)
Hope u understand what I mean. :wink:
Regards
Pat
quick Q. Currently we use only skyflats. Not yet set up for dome flats.
Can I use flatcombine -> ccdproc
or must i use
mkskyflat -> combine -> ccdproc (to include illumination cor.)
Hope u understand what I mean. :wink:
Regards
Pat
Francisco Valdes wrote on Feb 06, 2007
Hi Pat,
You can use sky flats just like dome flats as in your first case. The only difference is you have to make sure that flatcombine is set to eliminate sky sources as well as possible (you might check out the pclip rejection option). You would use the result in ccdproc as the flats. The skyflat option in CCDPROC is for a secondary correction.
The mkskyflat or other mksky stuff is meant for creating secondary corrections, such as illumination, to an initial dome flat. If you are using sky flats as your primary flat then you should presumably not need an illumination correction.
I hope this answers your question.
Yours,
Frank Valdes
You can use sky flats just like dome flats as in your first case. The only difference is you have to make sure that flatcombine is set to eliminate sky sources as well as possible (you might check out the pclip rejection option). You would use the result in ccdproc as the flats. The skyflat option in CCDPROC is for a secondary correction.
The mkskyflat or other mksky stuff is meant for creating secondary corrections, such as illumination, to an initial dome flat. If you are using sky flats as your primary flat then you should presumably not need an illumination correction.
I hope this answers your question.
Yours,
Frank Valdes
H.J. van Heerden wrote on Feb 06, 2007
Hi all
Got an error message. :?: :?: :?:
and if i call zerocombine ("@"//biaslist, .....)
do not know what is happening.
Thanx
Pat
Got an error message. :?: :?: :?:
boyden> biasred
The biasframes to be combined (@biaslist):
The images to be reduced:
ERROR: No images to combine
"pclip=pclip, grow=0)"
line 47: ccdred$zerocombine.cl
called as: `zerocombine (input=bias_001.fit, output=biasmaster.fits)'
"zerocombine (biaslist, output=biasmaster)"
line 38: boyden$biasred.cl
called as: `biasred ()'
and if i call zerocombine ("@"//biaslist, .....)
boyden> biasred
The biasframes to be combined (@biaslist):
The images to be reduced:
Warning: Cannot open image (SIMPLE=T/filedoesconformtoFITSstandardBITPIX=16/numberofbitsperdatapixelNAXI)
Warning: Cannot open image (EXPTIME=0.001/ExposuretimeinsecondsEXPTYPE='BIAS'/TypeofimageFILTER='C'/Filt)
Warning: Cannot open image (reinCEND\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff)
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff)
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff\uffff
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff\uffff
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff)
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
Warning: Cannot open image (\uffff
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff)
Warning: Cannot open image (\uffff\uffff\uffff\uffff
Warning: Cannot open image (\uffff\uffff\uffff\uffff\uffff\uffff
ERROR: software terminate (interrupt)
"pclip=pclip, grow=0)"
line 47: ccdred$zerocombine.cl
called as: `zerocombine (input=@bias_001.fit, output=biasmaster.fits)'
"zerocombine ("@"//biaslist, output=biasmaster)"
line 38: boyden$biasred.cl
called as: `biasred ()'
do not know what is happening.
Thanx
Pat
Francisco Valdes wrote on Feb 06, 2007
Hi Pat,
The message "No images to combine" usually means that the "ccdtype" is not being recognized. The simple solution to this is to set ccdtype="" and then be explicit in your input lists about the right type of calibration exposure.
The funny errors about "cannot open image" look like an @file is really a FITS file and so it is trying to read each line of the file, which would start with the header keyword SIMPLE, as if it was an image name.
That's all I can tell from the output you sent. I hope this helps.
Yours,
Frank
The message "No images to combine" usually means that the "ccdtype" is not being recognized. The simple solution to this is to set ccdtype="" and then be explicit in your input lists about the right type of calibration exposure.
The funny errors about "cannot open image" look like an @file is really a FITS file and so it is trying to read each line of the file, which would start with the header keyword SIMPLE, as if it was an image name.
That's all I can tell from the output you sent. I hope this helps.
Yours,
Frank
H.J. van Heerden wrote on Feb 06, 2007
Hi Frank
Yes thanx. Setting the ccdtype="" solved the problem. (Also removed the "@"// conitation, now ir reads the list correctly.) Now I get error because of incorrect parameters, so I assume?
Background: I wrote three scripts, each one creates a master-frame, and then applies that to the project images for the reductions. Input is lists containing the reduction frames, and project frames respectively. I suspect my parameters for zero-, dark- and flatcombine isn't correct.
Advice would be helpfull
Thanx
Regards
Pat
Yes thanx. Setting the ccdtype="" solved the problem. (Also removed the "@"// conitation, now ir reads the list correctly.) Now I get error because of incorrect parameters, so I assume?
boyden> biasred
The biasframes to be combined (@biaslist):
The images to be reduced:
Bad minmax rejection parameters
boyden> darkred
The darkframes to be combined (@darklist):
The images to be reduced:
ERROR: No bad pixel mask found
"ccdproc (ims, output="", ccdtype=ccdtype, noproc=no)"
line 37: ccdred$darkcombine.cl
called as: `darkcombine (input=dark_001.fit, output=darkmaster.fits, ccdtype=, scale=exposure)'
"darkcombine (darklist, output=darkmaster, scale='exposure', ccdtype= ..."
line 38: boyden$darkred.cl
called as: `darkred ()'
boyden> flatred
The flatframes to be combined: @flatlist
The images to be reduced:
ERROR: No bad pixel mask found
"ccdproc (ims, output="", ccdtype=ccdtype, noproc=no)"
line 38: ccdred$flatcombine.cl
called as: `flatcombine (input=FLAT_001_C.fit, output=flatmaster.fits, reject=pclip, ccdtype=, scale=mode, rdnoise=rdnoise, gain=gain)'
"reject='pclip', gain='gain', rdnoise='rdnoise', ccdtype="")"
line 39: boyden$flatred.cl
called as: `flatred ()'
Background: I wrote three scripts, each one creates a master-frame, and then applies that to the project images for the reductions. Input is lists containing the reduction frames, and project frames respectively. I suspect my parameters for zero-, dark- and flatcombine isn't correct.
Advice would be helpfull
Thanx
Regards
Pat
Francisco Valdes wrote on Feb 06, 2007
Hi Pat,
You need to post the parameters you have set, which are not explicitly set on the command line, for the combine tasks AND ccdproc. Note that ccdproc may be called as part of flatcombine, etc. To list the parameters do
cl> lpar flatcombine ccdproc
Frank
You need to post the parameters you have set, which are not explicitly set on the command line, for the combine tasks AND ccdproc. Note that ccdproc may be called as part of flatcombine, etc. To list the parameters do
cl> lpar flatcombine ccdproc
Frank
H.J. van Heerden wrote on Feb 06, 2007
Hi Frank
Here are the parameters.
Thanx
Pat
Here are the parameters.
zerocombine:
input = List of zero level images to combine
(output = "Zero") Output zero level name
(combine = "average") Type of combine operation
(reject = "minmax") Type of rejection
(ccdtype = "zero") CCD image type to combine
(process = no) Process images before combining?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(scale = "none") Image scaling
(statsec = "") Image section for computing statistics
(nlow = 0) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject (neg
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 0.) Value if there are no pixels
(mode = "ql")
darkcombine:
input = List of dark images to combine
(output = "Dark") Output dark image root name
(combine = "average") Type of combine operation
(reject = "minmax") Type of rejection
(ccdtype = "dark") CCD image type to combine
(process = yes) Process images before combining?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(scale = "exposure") Image scaling
(statsec = "") Image section for computing statistics
(nlow = 0) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject (neg
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 0.) Value if there are no pixels
(mode = "ql")
flatcombine:
input = List of flat field images to combine
(output = "Flat") Output flat field root name
(combine = "average") Type of combine operation
(reject = "avsigclip") Type of rejection
(ccdtype = "flat") CCD image type to combine
(process = yes) Process images before combining?
(subsets = yes) Combine images by subset parameter?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(scale = "mode") Image scaling
(statsec = "") Image section for computing statistics
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject (neg
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 1.) Value if there are no pixels
(mode = "ql")
ccdproc:
images = "" List of CCD images to correct
(output = "") List of output CCD images
(ccdtype = "object") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?\n
(fixpix = yes) Fix bad CCD lines and columns?
(overscan = yes) Apply overscan strip correction?
(trim = yes) Trim the image?
(zerocor = yes) Apply zero level correction?
(darkcor = yes) Apply dark count correction?
(flatcor = yes) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout correction?
(scancor = no) Convert flat field image to scan correction?\n
(readaxis = "line") Read out axis (column|line)
(fixfile = "") File describing the bad lines and columns
(biassec = "") Overscan strip image section
(trimsec = "") Trim data section
(zero = "") Zero level calibration image
(dark = "") Dark count calibration image
(flat = "") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = no) Fit overscan interactively?
(function = "legendre") Fitting function
(order = 1) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
Thanx
Pat
H.J. van Heerden wrote on Feb 06, 2007
Hi
Another little problem. Think it is the source of some of my other errors.
I created a setup.cl to set the "setinstrument" param file. I also created a instrument translation.dat file in the package dir. Here is the ccdred param and inst.dat file
and the boydeninst.dat file
The keyword for the ccdtype in the headers for the zeroframes is BIAS. The others is correct. The *combine (zero, ....) packages does not seem so pick up the .dat settings. Thus the reason why had to hardcode ccdtype="" as recommended.
Still experimenting with the whole reduction proccess. :)
Thanx for the help
Regards
Pat
Another little problem. Think it is the source of some of my other errors.
I created a setup.cl to set the "setinstrument" param file. I also created a instrument translation.dat file in the package dir. Here is the ccdred param and inst.dat file
boyden> lparam ccdred
(pixeltype = "real real") Output and calculation pixel datatypes
(verbose = no) Print log information to the standard output?
(logfile = "logfile") Text log file
(plotfile = "") Log metacode plot file
(backup = "") Backup directory or prefix
(instrument = "boyden$boydeninst.dat") CCD instrument file
(ssfile = "subsets") Subset translation file
(graphics = "stdgraph") Interactive graphics output device
(cursor = "") Graphics cursor input
(version = "2: October 1987")
(mode = "ql")
($nargs = 0)
and the boydeninst.dat file
exposure exptime
ccdtype exptype
subset filter
BIAS zero
The keyword for the ccdtype in the headers for the zeroframes is BIAS. The others is correct. The *combine (zero, ....) packages does not seem so pick up the .dat settings. Thus the reason why had to hardcode ccdtype="" as recommended.
Still experimenting with the whole reduction proccess. :)
Thanx for the help
Regards
Pat
Francisco Valdes wrote on Feb 06, 2007
Hi,
In the calibration combine tasks you have "process = yes". This tells these scripts to first run CCDPROC before combining. In CCDPROC you have "fixpix=yes" which results in looking for a bad pixel mask specified by the parameter "fixfile". I see in the CCDPROC parameters that while a number of operations are enabled the associated parameters, for example a list of flat exposures to go along with the "flatcor" parameter, are not set. It may be that you should turn the processing off in the combining tasks and combine the raw data. But normally I would recommend that the raw data is first processed for at least overscan and we here generally fully process zeros and flats before combining to make master calibrations. The order then matters:
1. process the zeros for overscan
2. combine the zeros
3. process the flats for overscan and bias using the master zero from the last step
4. combine the flats
5. process the science exposures using the master zeros and flats.
Frank
In the calibration combine tasks you have "process = yes". This tells these scripts to first run CCDPROC before combining. In CCDPROC you have "fixpix=yes" which results in looking for a bad pixel mask specified by the parameter "fixfile". I see in the CCDPROC parameters that while a number of operations are enabled the associated parameters, for example a list of flat exposures to go along with the "flatcor" parameter, are not set. It may be that you should turn the processing off in the combining tasks and combine the raw data. But normally I would recommend that the raw data is first processed for at least overscan and we here generally fully process zeros and flats before combining to make master calibrations. The order then matters:
1. process the zeros for overscan
2. combine the zeros
3. process the flats for overscan and bias using the master zero from the last step
4. combine the flats
5. process the science exposures using the master zeros and flats.
Frank
H.J. van Heerden wrote on Feb 06, 2007
Hi all
The scripts' are now working. But not completely. Two little errors that popped-up.
E1. When I enter a (I assume) null list value for the input, i.e. images?=""
then I get a EOF error.
Q. How can I check for such an error, errorhandler?
E2. When I run the script, it creates the masterbiasframe, but then it doesn't proccess the images. I.E. it doesn't run ccdproc. Currently has to proccess the images outside in cl.
P.S. Is there a way to do the reduction proccess in reverse, i.e. if the darkcor+ does not produce accept. results, can you go back and remove the darkmaster. Instead of running the whole proccess again.
Thanx
Pat
The scripts' are now working. But not completely. Two little errors that popped-up.
E1. When I enter a (I assume) null list value for the input, i.e. images?=""
then I get a EOF error.
The biasframes to be combined:
The images to be reduced:
ERROR: EOF from list param `biaslist' in expression
"zerocombine ("@"//biaslist, output=biasmaster, ccdtype="", process-)"
line 38: boyden$biasred.cl
called as: `biasred ()'
Q. How can I check for such an error, errorhandler?
E2. When I run the script, it creates the masterbiasframe, but then it doesn't proccess the images. I.E. it doesn't run ccdproc. Currently has to proccess the images outside in cl.
# Declare biasmaster
biasmaster = "biasmaster.fits"
# Implement zerocombine
zerocombine ("@"//biaslist, output=biasmaster, ccdtype="", process-)
# Process the images with ccdproc
while (fscan (imglist, img) != EOF) {
# Do the processing
if (access (img // ".fits")) {
ccdproc (img, ccdtype="", fixpix-, overscan+, trim+, zerocor+, darkcor-,
flatcor-, illum-, fringe-, zero=biasmaster)
}
}
P.S. Is there a way to do the reduction proccess in reverse, i.e. if the darkcor+ does not produce accept. results, can you go back and remove the darkmaster. Instead of running the whole proccess again.
Thanx
Pat
Francisco Valdes wrote on Feb 06, 2007
Hi Pat,
I've never seen the error you note before. However I'm not surprised it is an error to use @"" (which is @ by itself). This is a property of IRAF. In a script you can check for a null string with:
You could try and use error handling in the ecl but it is generally better to explicitly handle possible errors without resorting to catching error returns.
I don't understand the second question. But I think this is more about what you want to do. It is certainly possible to write scripts that handle a case such as you mention.
Frank
I've never seen the error you note before. However I'm not surprised it is an error to use @"" (which is @ by itself). This is a property of IRAF. In a script you can check for a null string with:
if (biaslist != "")
[do zerocombine]
else
print ("Skipping zerocombine")
You could try and use error handling in the ecl but it is generally better to explicitly handle possible errors without resorting to catching error returns.
I don't understand the second question. But I think this is more about what you want to do. It is certainly possible to write scripts that handle a case such as you mention.
Frank
H.J. van Heerden wrote on Feb 06, 2007
Hi
I think I found the source of all my problems. It seems that the script does not want to access the file lists that I use to run the tasks.
Here is the script code.
I just can't seem to figure out where the problem can be.
Help
Thanx
Pat
I think I found the source of all my problems. It seems that the script does not want to access the file lists that I use to run the tasks.
Here is the script code.
# Biasred script for creating a compiled bias frame and applying it on
# the project images.
procedure biasred (biasframes, images, proccess)
string biasframes {prompt="The biasframes to be combined"}
string images {prompt="The images to be reduced"}
string proccess = "yes" {prompt="Process images?"}
struct *bslist
struct *imglist
begin
# Define variables
int i
string img, imgfile, bsfile, biasmaster, proc
# Make sure the necessary packages are loaded
if (! defpac ("noao") || ! defpac ("imred"))
bye
# Expand the bias template into a text file list
bsfile = mktemp ("tmp$red")
sections (biasframes, option="fullname", > bsfile)
# Expand the image template into a text file list
imgfile = mktemp ("tmp$red")
sections (images, option="fullname", > imgfile)
# Redirect
bslist = bsfile
imglist = imgfile
# Declare biasmaster
biasmaster = "biasmaster.fits"
# Implement zerocombine
zerocombine (bslist, output=biasmaster, ccdtype="", process-)
# Check the acceptability of the master frame
display (biasmaster, 1)
proc = proccess
if (proc == "yes") {
# Process the images with ccdproc
while (fscan (imglist, img) != EOF) {
# Do the processing
if (access (img)) {
ccdproc (img, ccdtype="", fixpix-, overscan+, trim+, zerocor+, darkcor-,
flatcor-, illum-, fringe-, zero=biasmaster)
}
}
}
# Clean up
delete (bsfile, ver-, >& "dev$null")
delete (imgfile, ver-, >& "dev$null")
end
I just can't seem to figure out where the problem can be.
Help
Thanx
Pat
Francisco Valdes wrote on Feb 06, 2007
Hi Pat,
I believe your confusion is when you can give a list of filenames with or without the @. In your script the input is an image "list" which means it can be a comma separated list if images, wildcard template, or @file:
You would not use biaslist by itself. You then correctly use SECTIONS to convert the input list into a file of names. But then when you call ZEROCOMBINE you have to reference this as an image list since that is what that task expects.
Alternatively you could define the variable bslist as
I like the first style better since it keeps the distinction clearer.
Frank
I believe your confusion is when you can give a list of filenames with or without the @. In your script the input is an image "list" which means it can be a comma separated list if images, wildcard template, or @file:
cl> biasred a.fits,b.fits ...
cl> biasred bias*fits ...
cl> biasred @biaslist ...
You would not use biaslist by itself. You then correctly use SECTIONS to convert the input list into a file of names. But then when you call ZEROCOMBINE you have to reference this as an image list since that is what that task expects.
zerocombine ("@"//bslist, ...)
Alternatively you could define the variable bslist as
bslist = "@" // bsfile
zerocombine (bslist, ...)
I like the first style better since it keeps the distinction clearer.
Frank
H.J. van Heerden wrote on Feb 06, 2007
Hi there.
Sorry for this, but this has been bugging me for the last two weeks. I have finalized my prereduction scripts. But now they do not want to exec...
It seems that when I call them and pass the relevant input, it does not read the relevant files. I ran cl> d_trace and got a whole bunch of "stuff" that I understand to a degree, but I do not know where to look for the error.
Can someone please help!
Here is one of the scripts
and here is the d_trace output
Thanx
Pat
Sorry for this, but this has been bugging me for the last two weeks. I have finalized my prereduction scripts. But now they do not want to exec...
It seems that when I call them and pass the relevant input, it does not read the relevant files. I ran cl> d_trace and got a whole bunch of "stuff" that I understand to a degree, but I do not know where to look for the error.
Can someone please help!
Here is one of the scripts
procedure biasred (biasframes, images, proccess) # Define procedure
# Define procedure variables
string biasframes {prompt="The biasframes to be combined"}
string images {prompt="The images to be reduced"}
string proccess = "yes" {prompt="Process images"}
struct *bslist
struct *imglist
begin
# Define script variables
int i, s
string img, imgfile, bsfile, biasmaster, proc
# Make sure the necessary packages are loaded
if (! defpac ("noao") || ! defpac ("imred")) {
bye()
}
;
# Expand the bias template into a text file list
bsfile = mktemp ("tmp$red")
sections (biasframes, option = "fullname", > bsfile)
# Expand the image template into a text file list
imgfile = mktemp ("tmp$red")
sections (images, option = "fullname", > imgfile)
# Redirect
bslist = bsfile
imglist = imgfile
# Declare biasmaster
biasmaster = "biasmaster.fits"
if (access (biasmaster)){
imdelete (biasmaster)
}
;
# Implement zerocombine
print (bslist) | scan (s)
if (nscan() != 0) {
zerocombine ("@"//bslist, output = biasmaster, ccdtype = "", reject='pclip', process-)
}
;
# Check the acceptability of the master-frame
if (access (biasmaster)) {
display (biasmaster, 1)
proc = proccess
# Process the images with ccdproc
if (proc == "yes") {
# Scan through the imagelist
while (fscan (imglist, img) != EOF) {
# Get rid of the ".fit", ".fits" extention
i = strlen (img)
if (substr (img, i-3, i) == ".fit") {
img = substr (img, 1, i-4)
}
else if (substr (img, i-4, i) == ".fits") {
img = substr (img, 1, i-5)
}
;
# Call ccdproc
if (access (img // ".fit") || access (img // ".fits")) {
ccdproc (img, ccdtype = "", fixpix-, overscan-, trim-, zerocor+,
darkcor-, flatcor-, illum-, fringe-, zero=biasmaster)
}
;
}
;
}
;
}
;
# Clean up
delete (bsfile, ver-, >& "dev$null")
delete (imgfile, ver-, >& "dev$null")
end
and here is the d_trace output
boyden> biasred
----- task cl - line 5 (CALL) -----
5 22+5: call biasred
5 27+3: exec
----- exec script biasred (boyden$biasred.cl) - line 5 -----
----- task biasred - line 24 (PUSHCONST) -----
24 33+8: pushconst `noao'
24 41+6: pushconst 1
24 47+5: intrinsic defpac
24 52+3: not
24 55+8: pushconst `imred'
24 63+6: pushconst 1
24 69+5: intrinsic defpac
24 74+3: not
24 77+3: or
24 80+4: biff +7
30 91+8: pushconst `tmp$red'
30 99+6: pushconst 1
30 105+5: intrinsic mktemp
30 110+5: assign bsfile
31 115+6: call sections
1: CALL sections
31 121+6: pushparam biasframes
The biasframes to be combined (biaslist):
31 127+4: posargset 0
31 131+9: pushconst `fullname'
31 140+5: absargset option
31 145+5: pushparam bsfile
31 150+3: redir
31 153+3: exec
----- exec external task sections - line 34 -----
----- connect to bin$x_images.e -----
sections > $\nimages="biaslist"\noption="fullname"\n
nimages=102\nmode="ql"\n$nargs=1\n
--------------------------------
----- task sections - line 4 (PUSHCONST) -----
4 967+6: pushconst 1
4 973+5: assign nimages
4 978+3: end
----- task sections - line 5 (CALL) -----
5 967+4: call bye
1: CALL bye
5 971+3: exec
----- exec builtin bye : line 5 -----
34 156+8: pushconst `tmp$red'
34 164+6: pushconst 1
34 170+5: intrinsic mktemp
34 175+5: assign imgfile
35 180+6: call sections
2: CALL sections
35 186+5: pushparam images
The images to be reduced (@imagelist):
35 191+4: posargset 0
35 195+9: pushconst `fullname'
35 204+5: absargset option
35 209+5: pushparam imgfile
35 214+3: redir
35 217+3: exec
----- exec external task sections - line 38 -----
sections > $\nimages="@imagelist"\noption="fullname"\n
nimages=1\nmode="ql"\n$nargs=1\n
--------------------------------
----- task sections - line 6 (PUSHCONST) -----
6 967+6: pushconst 102
6 973+5: assign nimages
6 978+3: end
----- task sections - line 7 (CALL) -----
7 967+4: call bye
2: CALL bye
7 971+3: exec
----- exec builtin bye : line 7 -----
38 220+5: pushparam bsfile
38 225+5: assign bslist
39 230+5: pushparam imgfile
39 235+5: assign imglist
42 240+10: pushconst `biasmaster.fits'
42 250+6: assign biasmaster
44 256+6: pushparam biasmaster
44 262+6: pushconst 1
44 268+5: intrinsic access
44 273+4: biff +19
50 296+5: call print
3: CALL print
50 301+5: pushparam bslist
50 306+4: posargset 0
50 310+4: addpipe
50 314+3: redir
50 317+3: exec
----- exec builtin print : line 50 -----
50 320+5: call scan
4: CALL scan
50 325+9: pushconst `biasred.s'
50 334+4: indirposset 0
50 338+5: getpipe
50 343+3: redirin
50 346+3: exec
----- exec builtin scan : line 50 -----
50 349+4: rmpipes 1
51 353+6: pushconst 0
51 359+5: intrinsic nscan
51 364+6: pushconst 0
51 370+3: ne
51 373+4: biff +69
57 446+6: pushparam biasmaster
57 452+6: pushconst 1
57 458+5: intrinsic access
57 463+4: biff +431
91 898+5: call delete
5: CALL delete
91 903+5: pushparam bsfile
91 908+4: posargset 0
91 912+4: swoff ver
91 916+9: pushconst `dev$null'
91 925+3: allredir
91 928+3: exec
----- exec external task delete - line 92 -----
delete > $ 5> $\nfiles="tmp$red4401a"\nverify=no\n
default_action=yes\nallversions=yes\nsubfiles=yes\nmode="ql"\n
$nargs=1\n
--------------------------------
8 967+4: call bye
8 971+3: exec
92 931+5: call delete
7: CALL delete
92 936+5: pushparam imgfile
92 941+4: posargset 0
92 945+4: swoff ver
92 949+9: pushconst `dev$null'
92 958+3: allredir
92 961+3: exec
----- exec external task delete - line 94 -----
delete > $ 5> $\nfiles="tmp$red4401b"\nverify=no\n
default_action=yes\nallversions=yes\nsubfiles=yes\nmode="ql"\n
$nargs=1\n
--------------------------------
9 967+4: call bye
9 971+3: exec
94 964+3: end
----- task cl - line 5 (END) -----
5 30+3: end
Thanx
Pat
Francisco Valdes wrote on Feb 06, 2007
Hi Pat,
The trace output currently just shows that there were no execution errors. It is executing but it is finding nothing to process (I think). A better way to debug in this case might be putting in print statements.
My guess is that you are not specifying the input properly. The value of the biasframes paraemter should either be something like
@mybiases
or
bias1.fits,bias2.fits,nite2*.fits
In other words the input is an image list consisting of some combination of @files (with the @ symbol!!), a file list, or file template. My guess is you are just giving a filename containing the list of biases you want to process.
To check what you should be processing do
cl> sections <biasinput>
where <biasinput> is the value you are giving your script when you execute it.
Yours,
Frank
The trace output currently just shows that there were no execution errors. It is executing but it is finding nothing to process (I think). A better way to debug in this case might be putting in print statements.
My guess is that you are not specifying the input properly. The value of the biasframes paraemter should either be something like
@mybiases
or
bias1.fits,bias2.fits,nite2*.fits
In other words the input is an image list consisting of some combination of @files (with the @ symbol!!), a file list, or file template. My guess is you are just giving a filename containing the list of biases you want to process.
To check what you should be processing do
cl> sections <biasinput>
where <biasinput> is the value you are giving your script when you execute it.
Yours,
Frank
H.J. van Heerden wrote on Feb 06, 2007
Hi Frank
It was the error-handler again. To check if biasframes are null or not.
I finally got a propper error-handler working for it. Just had to sit down and think about it properly, with some help from my programmer buddy.
Think I am finally starting to understand how coding in csh works.
Anyway! Here is the final code. Hope it meets your approvel.
Thanx for all the help.
C.U.Later
Pat
It was the error-handler again. To check if biasframes are null or not.
I finally got a propper error-handler working for it. Just had to sit down and think about it properly, with some help from my programmer buddy.
Think I am finally starting to understand how coding in csh works.
Anyway! Here is the final code. Hope it meets your approvel.
# BIASRED.CL - Biasred script for the BOYDEN package.
# Biasred script for creating a compiled bias frame and applying it on
# the project images.
procedure biasred (biasframes, images, proccess) # Define procedure
# Define procedure variables
string biasframes {prompt="The biasframes to be combined"}
string images {prompt="The images to be reduced"}
string proccess = "yes" {prompt="Process images"}
struct *bslist
struct *imglist
begin
# Define script variables
int i
string img, imgfile, bsfile, biasmaster, proc, bias, project
# Make sure the necessary packages are loaded
if (! defpac ("noao") || ! defpac ("imred")) {
bye()
}
;
# Check for null value for biasframes
bias = biasframes
if (bias != "") {
# Expand the bias template into a text file list
bsfile = mktemp ("tmp$red")
sections (bias, option = "fullname", > bsfile)
}
else {
print ("No biasframes defined")
bye()
}
;
# Check for null value for imageframes
project = images
if (project != "") {
# Expand the image template into a text file list
imgfile = mktemp ("tmp$red")
sections (project, option = "fullname", > imgfile)
}
else {
print ("No images defined")
bye()
}
;
# Redirect
bslist = bsfile
imglist = imgfile
# Declare biasmaster
biasmaster = "biasmaster.fits"
if (access (biasmaster)){
imdelete (biasmaster)
}
;
# Implement zerocombine
zerocombine ("@"//bslist, output = biasmaster, ccdtype = "", reject='pclip', process-)
# Check the acceptability of the master-frame
if (access (biasmaster)) {
display (biasmaster, 1)
proc = proccess
# Process the images with ccdproc
if (proc == "yes") {
# Scan through the imagelist
while (fscan (imglist, img) != EOF) {
# Get rid of the ".fit", ".fits" extention
i = strlen (img)
if (substr (img, i-3, i) == ".fit") {
img = substr (img, 1, i-4)
}
else if (substr (img, i-4, i) == ".fits") {
img = substr (img, 1, i-5)
}
;
# Call ccdproc
if (access (img // ".fit") || access (img // ".fits")) {
ccdproc (img, ccdtype = "", fixpix-, overscan-, trim-, zerocor+,
darkcor-, flatcor-, illum-, fringe-, zero=biasmaster)
}
;
}
;
}
;
}
;
# Clean up
bslist = ""
imglist = ""
delete (bsfile, ver-, >& "dev$null")
delete (imgfile, ver-, >& "dev$null")
end
Thanx for all the help.
C.U.Later
Pat
Last post on Feb 06, 2007