problem with dohydra (apfind) on ctio hydra data
Philip Massey wrote on Nov 12, 2009
Hi, Frank et al
I'm running dohydra on some recently acquired CTIO hydra data. There are a lot of gaps due to unassigned fibers and the like, and so I am checking the IDs carefully.
The peculiar thing is that when I have it automatically find apertures, it always marks one particular aperture as two, (i.e., the aperture at pixel 1035 gets labeled as both 71 and 72).
I've put a copy of the object data, projector flat (used for the ap reference and for ap flat field), and comparison at "http://www.lowell.edu/users/massey/hydraprob.tar.gz
In running dohydra, once it comes up with the automatic identifications I have been doing an "a" and a "d" to delete all of the apertures. I then do a :threshold 5000, followed by a "f" to find all of the apertures. It is at this point that you will find that the aperture at pixel 1041 has two ids.
Any idea why?
thanks,
phil
Here are my parameters:
PACKAGE = imred
TASK = hydra
(observa= observatory) Observatory of data
(interp = poly5) Interpolation type
(dispaxi= 2) Image axis for 2D/3D images
(nsum = 1) Number of lines/columns/bands to sum for 2D/3D i
(databas= database) Database
(verbose= yes) Verbose output?
(logfile= logfile) Log file
(plotfil= ) Plot file
(records= )
(version= HYDRA V1: January 1992)
(mode = ql)
($nargs = 0)
objects = obj334 List of object spectra
(apref = pflat337) Aperture reference spectrum
(flat = pflat337) Flat field spectrum
(through= ) Throughput file or image (optional)
(arcs1 = comp338) List of arc spectra
(arcs2 = ) List of shift arc spectra
(arcrepl= ) Special aperture replacements
(arctabl= ) Arc assignment table (optional)
(readnoi= 5.6) Read out noise sigma (photons)
(gain = 2.4) Photon gain (photons/data number)
(datamax= 65000.) Max data value / cosmic ray threshold
(fibers = 141) Number of fibers
(width = 10.) Width of profiles (pixels)
(minsep = 11.) Minimum separation between fibers (pixels)
(maxsep = 15.) Maximum separation between fibers (pixels)
(apidtab= obj334) Aperture identifications
(crval = INDEF) Approximate central wavelength
(cdelt = INDEF) Approximate dispersion
(objaps = ) Object apertures
(skyaps = ) Sky apertures
(arcaps = ) Arc apertures
(objbeam= 1) Object beam numbers
(skybeam= 0) Sky beam numbers
(arcbeam= ) Arc beam numbers
(scatter= no) Subtract scattered light?
(fitflat= yes) Fit and ratio flat field spectrum?
(clean = yes) Detect and replace bad pixels?
(dispcor= yes) Dispersion correct spectra?
(savearc= no) Save simultaneous arc apertures?
(skyalig= no) Align sky lines?
(skysubt= yes) Subtract sky?
(skyedit= yes) Edit the sky spectra?
(savesky= yes) Save sky spectra?
(splot = no) Plot the final spectrum?
(redo = yes) Redo operations if previously done?
(update = yes) Update spectra if cal data changes?
(batch = no) Extract objects in batch?
(listonl= no) List steps but don't process?
(params = ) Algorithm parameters
(mode = ql)
PACKAGE = hydra
TASK = params
(line = INDEF) Default dispersion line
(nsum = 10) Number of dispersion lines to sum or median
(order = increasing) Order of apertures
(extras = no) Extract sky, sigma, etc.?
-- DEFAULT APERTURE LIMITS --
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center
-- AUTOMATIC APERTURE RESIZING PARAMETERS --
(ylevel = 0.05) Fraction of peak or intensity for resizing
-- TRACE PARAMETERS --
(t_step = 10) Tracing step
(t_funct= spline3) Trace fitting function
(t_order= 3) Trace fitting function order
(t_niter= 5) Trace rejection iterations
(t_low = 3.) Trace lower rejection sigma
(t_high = 3.) Trace upper rejection sigma
-- SCATTERED LIGHT PARAMETERS --
(buffer = 1.) Buffer distance from apertures
(apscat1= ) Fitting parameters across the dispersion
(apscat2= ) Fitting parameters along the dispersion
-- APERTURE EXTRACTION PARAMETERS --
(weights= none) Extraction weights (none|variance)
(pfit = fit1d) Profile fitting algorithm (fit1d|fit2d)
(lsigma = 3.) Lower rejection threshold
(usigma = 3.) Upper rejection threshold
(nsubaps= 1) Number of subapertures
-- FLAT FIELD FUNCTION FITTING PARAMETERS --
(f_inter= yes) Fit flat field interactively?
(f_funct= chebyshev) Fitting function
(f_order= 1) Fitting function order
-- ARC DISPERSION FUNCTION PARAMETERS --
(thresho= 10.) Minimum line contrast threshold
(coordli= linelists$ctiohenear.dat) Line list
(match = -3.) Line list matching limit in Angstroms
(fwidth = 4.) Arc line widths in pixels
(cradius= 10.) Centering radius in pixels
(i_funct= spline3) Coordinate function
(i_order= 3) Order of dispersion function
(i_niter= 2) Rejection iterations
(i_low = 3.) Lower rejection sigma
(i_high = 3.) Upper rejection sigma
(refit = yes) Refit coordinate function when reidentifying?
(addfeat= no) Add features when reidentifying?
-- AUTOMATIC ARC ASSIGNMENT PARAMETERS --
(select = interp) Selection method for reference spectra
(sort = jd) Sort key
(group = ljd) Group key
(time = no) Is sort key a time?
(timewra= 17.) Time wrap point for time sorting
-- DISPERSION CORRECTION PARAMETERS --
(lineari= yes) Linearize (interpolate) spectra?
(log = no) Logarithmic wavelength scale?
(flux = yes) Conserve flux?
-- SKY SUBTRACTION PARAMETERS --
(combine= average) Type of combine operation
(reject = avsigclip) Sky rejection option
(scale = none) Sky scaling option
(mode = ql)
I'm running dohydra on some recently acquired CTIO hydra data. There are a lot of gaps due to unassigned fibers and the like, and so I am checking the IDs carefully.
The peculiar thing is that when I have it automatically find apertures, it always marks one particular aperture as two, (i.e., the aperture at pixel 1035 gets labeled as both 71 and 72).
I've put a copy of the object data, projector flat (used for the ap reference and for ap flat field), and comparison at "http://www.lowell.edu/users/massey/hydraprob.tar.gz
In running dohydra, once it comes up with the automatic identifications I have been doing an "a" and a "d" to delete all of the apertures. I then do a :threshold 5000, followed by a "f" to find all of the apertures. It is at this point that you will find that the aperture at pixel 1041 has two ids.
Any idea why?
thanks,
phil
Here are my parameters:
PACKAGE = imred
TASK = hydra
(observa= observatory) Observatory of data
(interp = poly5) Interpolation type
(dispaxi= 2) Image axis for 2D/3D images
(nsum = 1) Number of lines/columns/bands to sum for 2D/3D i
(databas= database) Database
(verbose= yes) Verbose output?
(logfile= logfile) Log file
(plotfil= ) Plot file
(records= )
(version= HYDRA V1: January 1992)
(mode = ql)
($nargs = 0)
objects = obj334 List of object spectra
(apref = pflat337) Aperture reference spectrum
(flat = pflat337) Flat field spectrum
(through= ) Throughput file or image (optional)
(arcs1 = comp338) List of arc spectra
(arcs2 = ) List of shift arc spectra
(arcrepl= ) Special aperture replacements
(arctabl= ) Arc assignment table (optional)
(readnoi= 5.6) Read out noise sigma (photons)
(gain = 2.4) Photon gain (photons/data number)
(datamax= 65000.) Max data value / cosmic ray threshold
(fibers = 141) Number of fibers
(width = 10.) Width of profiles (pixels)
(minsep = 11.) Minimum separation between fibers (pixels)
(maxsep = 15.) Maximum separation between fibers (pixels)
(apidtab= obj334) Aperture identifications
(crval = INDEF) Approximate central wavelength
(cdelt = INDEF) Approximate dispersion
(objaps = ) Object apertures
(skyaps = ) Sky apertures
(arcaps = ) Arc apertures
(objbeam= 1) Object beam numbers
(skybeam= 0) Sky beam numbers
(arcbeam= ) Arc beam numbers
(scatter= no) Subtract scattered light?
(fitflat= yes) Fit and ratio flat field spectrum?
(clean = yes) Detect and replace bad pixels?
(dispcor= yes) Dispersion correct spectra?
(savearc= no) Save simultaneous arc apertures?
(skyalig= no) Align sky lines?
(skysubt= yes) Subtract sky?
(skyedit= yes) Edit the sky spectra?
(savesky= yes) Save sky spectra?
(splot = no) Plot the final spectrum?
(redo = yes) Redo operations if previously done?
(update = yes) Update spectra if cal data changes?
(batch = no) Extract objects in batch?
(listonl= no) List steps but don't process?
(params = ) Algorithm parameters
(mode = ql)
PACKAGE = hydra
TASK = params
(line = INDEF) Default dispersion line
(nsum = 10) Number of dispersion lines to sum or median
(order = increasing) Order of apertures
(extras = no) Extract sky, sigma, etc.?
-- DEFAULT APERTURE LIMITS --
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center
-- AUTOMATIC APERTURE RESIZING PARAMETERS --
(ylevel = 0.05) Fraction of peak or intensity for resizing
-- TRACE PARAMETERS --
(t_step = 10) Tracing step
(t_funct= spline3) Trace fitting function
(t_order= 3) Trace fitting function order
(t_niter= 5) Trace rejection iterations
(t_low = 3.) Trace lower rejection sigma
(t_high = 3.) Trace upper rejection sigma
-- SCATTERED LIGHT PARAMETERS --
(buffer = 1.) Buffer distance from apertures
(apscat1= ) Fitting parameters across the dispersion
(apscat2= ) Fitting parameters along the dispersion
-- APERTURE EXTRACTION PARAMETERS --
(weights= none) Extraction weights (none|variance)
(pfit = fit1d) Profile fitting algorithm (fit1d|fit2d)
(lsigma = 3.) Lower rejection threshold
(usigma = 3.) Upper rejection threshold
(nsubaps= 1) Number of subapertures
-- FLAT FIELD FUNCTION FITTING PARAMETERS --
(f_inter= yes) Fit flat field interactively?
(f_funct= chebyshev) Fitting function
(f_order= 1) Fitting function order
-- ARC DISPERSION FUNCTION PARAMETERS --
(thresho= 10.) Minimum line contrast threshold
(coordli= linelists$ctiohenear.dat) Line list
(match = -3.) Line list matching limit in Angstroms
(fwidth = 4.) Arc line widths in pixels
(cradius= 10.) Centering radius in pixels
(i_funct= spline3) Coordinate function
(i_order= 3) Order of dispersion function
(i_niter= 2) Rejection iterations
(i_low = 3.) Lower rejection sigma
(i_high = 3.) Upper rejection sigma
(refit = yes) Refit coordinate function when reidentifying?
(addfeat= no) Add features when reidentifying?
-- AUTOMATIC ARC ASSIGNMENT PARAMETERS --
(select = interp) Selection method for reference spectra
(sort = jd) Sort key
(group = ljd) Group key
(time = no) Is sort key a time?
(timewra= 17.) Time wrap point for time sorting
-- DISPERSION CORRECTION PARAMETERS --
(lineari= yes) Linearize (interpolate) spectra?
(log = no) Logarithmic wavelength scale?
(flux = yes) Conserve flux?
-- SKY SUBTRACTION PARAMETERS --
(combine= average) Type of combine operation
(reject = avsigclip) Sky rejection option
(scale = none) Sky scaling option
(mode = ql)
Frank Winkler wrote on Nov 12, 2009
Hi Phil,
I'm not the Frank you were hoping to hear from, but maybe this will help. I've analyzed exactly one set of CTIO Hydra data ever, and spent a huge amount of time figuring out how to make dohydra work. Of course one always has to check these things manually, but here are the params that worked pretty well for me. Main differences from yours that I notice in a quick glance are that for "width" I used my best guess at the actual separation between adjacent profiles, whereas you used a value smaller than your "minsep", and for "fibers" I used only the number of active fibers in my exposure.
I have no clue if any of this will help, but good luck!
Frank Winkler (aka Paco)
==> means PAY ATTENTION TO THESE ESPECIALLY! Be sure they are right for current data.
DOHYDRA params:
objects = "obj1006c" List of object spectra
(apref = "Pflat_g292_f1") Aperture reference spectrum
(flat = "Dflat_n1_circle") Flat field spectrum
(throughput = "") Throughput file or image (optional)
(arcs1 = "comp1009") List of arc spectra
(arcs2 = "") List of shift arc spectra
(arcreplace = "") Special aperture replacements
(arctable = "") Arc assignment table (optional)\n
(readnoise = "rdnoise") Read out noise sigma (photons)
(gain = "gain") Photon gain (photons/data number)
(datamax = 2000.) Max data value / cosmic ray threshold
==> (fibers = 29) Number of fibers
==> (width = 13.6) Width of profiles (pixels)
==> (minsep = 11.) Minimum separation between fibers (pixels)
==> (maxsep = 16.) Maximum separation between fibers (pixels)
(apidtable = "apid_2") Aperture identifications
(crval = "INDEF") Approximate central wavelength
(cdelt = "INDEF") Approximate dispersion
(objaps = "") Object apertures
(skyaps = "") Sky apertures
(arcaps = "") Arc apertures
(objbeams = "0,1") Object beam numbers
(skybeams = "0") Sky beam numbers
(arcbeams = "") Arc beam numbers\n
(scattered = no) Subtract scattered light?
(fitflat = yes) Fit and ratio flat field spectrum?
==> (clean = yes) Detect and replace bad pixels?
(dispcor = yes) Dispersion correct spectra?
(savearcs = yes) Save simultaneous arc apertures?
(skyalign = no) Align sky lines?
(skysubtract = yes) Subtract sky?
(skyedit = yes) Edit the sky spectra?
(saveskys = yes) Save sky spectra?
(splot = yes) Plot the final spectrum?
(redo = no) Redo operations if previously done?
(update = yes) Update spectra if cal data changes?
(batch = no) Extract objects in batch?
(listonly = no) List steps but don't process?\n
(params = "") Algorithm parameters
(mode = "ql")
epar PARAMS (auxilliary parameter set also used by DOHYDRA
(line = INDEF) Default dispersion line
(nsum = 10) Number of dispersion lines to sum or median
==> (order = "increasing") Order of apertures
(extras = no) Extract sky, sigma, etc.?
DEFAULT APERTURE
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center\n\n-- AU
(ylevel = 0.05) Fraction of peak or intensity for resizing\n\n--
(t_step = 10) Tracing step
(t_function = "spline3") Trace fitting function
(t_order = 3) Trace fitting function order
(t_niterate = 1) Trace rejection iterations
(t_low = 3.) Trace lower rejection sigma
(t_high = 3.) Trace upper rejection sigma
SCATTERED LIGH
(buffer = 1.) Buffer distance from apertures
(apscat1 = "") Fitting parameters across the dispersion
(apscat2 = "") Fitting parameters along the dispersion\n\n-- AP
==> (weights = "variance") Extraction weights (none|variance)
(pfit = "fit1d") Profile fitting algorithm (fit1d|fit2d)
(lsigma = 3.) Lower rejection threshold
(usigma = 3.) Upper rejection threshold
(nsubaps = 1) Number of subapertures
FLAT FIELD FUNCTION
(f_interactiv = yes) Fit flat field interactively?
(f_function = "spline3") Fitting function
(f_order = 18) Fitting function order
ARC DISPERSION FUNC
(threshold = 10.) Minimum line contrast threshold
(coordlist = "linelists$ctiohenear.dat") Line list
(match = -3.) Line list matching limit in Angstroms
(fwidth = 4.) Arc line widths in pixels
(cradius = 10.) Centering radius in pixels
==> (i_function = "chebyshev") Coordinate function
==> (i_order = 2) Order of dispersion function
........ used order = 2 (linear) to start; final fit used order = 6 ........
(i_niterate = 2) Rejection iterations
(i_low = 3.) Lower rejection sigma
(i_high = 3.) Upper rejection sigma
(refit = yes) Refit coordinate function when reidentifying?
(addfeatures = no) Add features when reidentifying?
AUTOMATIC
(select = "interp") Selection method for reference spectra
(sort = "jd") Sort key
(group = "ljd") Group key
(time = no) Is sort key a time?
(timewrap = 17.) Time wrap point for time sorting
DISPERSION
(linearize = yes) Linearize (interpolate) spectra?
(log = no) Logarithmic wavelength scale?
(flux = yes) Conserve flux?
SKY SUBTRACTION PARAMETERS
(combine = "average") Type of combine operation
(reject = "avsigclip") Sky rejection option
(scale = "none") Sky scaling option
(mode = "ql")
I'm not the Frank you were hoping to hear from, but maybe this will help. I've analyzed exactly one set of CTIO Hydra data ever, and spent a huge amount of time figuring out how to make dohydra work. Of course one always has to check these things manually, but here are the params that worked pretty well for me. Main differences from yours that I notice in a quick glance are that for "width" I used my best guess at the actual separation between adjacent profiles, whereas you used a value smaller than your "minsep", and for "fibers" I used only the number of active fibers in my exposure.
I have no clue if any of this will help, but good luck!
Frank Winkler (aka Paco)
==> means PAY ATTENTION TO THESE ESPECIALLY! Be sure they are right for current data.
DOHYDRA params:
objects = "obj1006c" List of object spectra
(apref = "Pflat_g292_f1") Aperture reference spectrum
(flat = "Dflat_n1_circle") Flat field spectrum
(throughput = "") Throughput file or image (optional)
(arcs1 = "comp1009") List of arc spectra
(arcs2 = "") List of shift arc spectra
(arcreplace = "") Special aperture replacements
(arctable = "") Arc assignment table (optional)\n
(readnoise = "rdnoise") Read out noise sigma (photons)
(gain = "gain") Photon gain (photons/data number)
(datamax = 2000.) Max data value / cosmic ray threshold
==> (fibers = 29) Number of fibers
==> (width = 13.6) Width of profiles (pixels)
==> (minsep = 11.) Minimum separation between fibers (pixels)
==> (maxsep = 16.) Maximum separation between fibers (pixels)
(apidtable = "apid_2") Aperture identifications
(crval = "INDEF") Approximate central wavelength
(cdelt = "INDEF") Approximate dispersion
(objaps = "") Object apertures
(skyaps = "") Sky apertures
(arcaps = "") Arc apertures
(objbeams = "0,1") Object beam numbers
(skybeams = "0") Sky beam numbers
(arcbeams = "") Arc beam numbers\n
(scattered = no) Subtract scattered light?
(fitflat = yes) Fit and ratio flat field spectrum?
==> (clean = yes) Detect and replace bad pixels?
(dispcor = yes) Dispersion correct spectra?
(savearcs = yes) Save simultaneous arc apertures?
(skyalign = no) Align sky lines?
(skysubtract = yes) Subtract sky?
(skyedit = yes) Edit the sky spectra?
(saveskys = yes) Save sky spectra?
(splot = yes) Plot the final spectrum?
(redo = no) Redo operations if previously done?
(update = yes) Update spectra if cal data changes?
(batch = no) Extract objects in batch?
(listonly = no) List steps but don't process?\n
(params = "") Algorithm parameters
(mode = "ql")
epar PARAMS (auxilliary parameter set also used by DOHYDRA
(line = INDEF) Default dispersion line
(nsum = 10) Number of dispersion lines to sum or median
==> (order = "increasing") Order of apertures
(extras = no) Extract sky, sigma, etc.?
DEFAULT APERTURE
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center\n\n-- AU
(ylevel = 0.05) Fraction of peak or intensity for resizing\n\n--
(t_step = 10) Tracing step
(t_function = "spline3") Trace fitting function
(t_order = 3) Trace fitting function order
(t_niterate = 1) Trace rejection iterations
(t_low = 3.) Trace lower rejection sigma
(t_high = 3.) Trace upper rejection sigma
SCATTERED LIGH
(buffer = 1.) Buffer distance from apertures
(apscat1 = "") Fitting parameters across the dispersion
(apscat2 = "") Fitting parameters along the dispersion\n\n-- AP
==> (weights = "variance") Extraction weights (none|variance)
(pfit = "fit1d") Profile fitting algorithm (fit1d|fit2d)
(lsigma = 3.) Lower rejection threshold
(usigma = 3.) Upper rejection threshold
(nsubaps = 1) Number of subapertures
FLAT FIELD FUNCTION
(f_interactiv = yes) Fit flat field interactively?
(f_function = "spline3") Fitting function
(f_order = 18) Fitting function order
ARC DISPERSION FUNC
(threshold = 10.) Minimum line contrast threshold
(coordlist = "linelists$ctiohenear.dat") Line list
(match = -3.) Line list matching limit in Angstroms
(fwidth = 4.) Arc line widths in pixels
(cradius = 10.) Centering radius in pixels
==> (i_function = "chebyshev") Coordinate function
==> (i_order = 2) Order of dispersion function
........ used order = 2 (linear) to start; final fit used order = 6 ........
(i_niterate = 2) Rejection iterations
(i_low = 3.) Lower rejection sigma
(i_high = 3.) Upper rejection sigma
(refit = yes) Refit coordinate function when reidentifying?
(addfeatures = no) Add features when reidentifying?
AUTOMATIC
(select = "interp") Selection method for reference spectra
(sort = "jd") Sort key
(group = "ljd") Group key
(time = no) Is sort key a time?
(timewrap = 17.) Time wrap point for time sorting
DISPERSION
(linearize = yes) Linearize (interpolate) spectra?
(log = no) Logarithmic wavelength scale?
(flux = yes) Conserve flux?
SKY SUBTRACTION PARAMETERS
(combine = "average") Type of combine operation
(reject = "avsigclip") Sky rejection option
(scale = "none") Sky scaling option
(mode = "ql")
Philip Massey wrote on Nov 12, 2009
Hi, Frank---
A few comments on your parameters...
The width parameter is used only by the centering algorithm, and should be
"base to base". I made mine a little smaller than yours because there isn't
clean separation between different fibers. The nice thing about using
the treashold parameter is that it stops it from finding a bunch of fibers that
wasn't assigned. I see you get around this by specifying nfibs=29 but then your numbering must not agree with the id's in the header. Using 141 results in most of them being numbered correctly.
Also, it's harmless but redundant to specify weighting=variance if you have clean set to yes.
My experience is that a 1st order cubic spline is needed for a good comparison solution.
Someone really should write a manual for all of this....
But the behavior described above (two apertures being identified for a single
one) is clearly some sort of bug, and it may be affecting other things as
well.
Thanks!
---phil
A few comments on your parameters...
The width parameter is used only by the centering algorithm, and should be
"base to base". I made mine a little smaller than yours because there isn't
clean separation between different fibers. The nice thing about using
the treashold parameter is that it stops it from finding a bunch of fibers that
wasn't assigned. I see you get around this by specifying nfibs=29 but then your numbering must not agree with the id's in the header. Using 141 results in most of them being numbered correctly.
Also, it's harmless but redundant to specify weighting=variance if you have clean set to yes.
My experience is that a 1st order cubic spline is needed for a good comparison solution.
Someone really should write a manual for all of this....
But the behavior described above (two apertures being identified for a single
one) is clearly some sort of bug, and it may be affecting other things as
well.
Thanks!
---phil
Last post on Nov 12, 2009