Dohydra Problems with flat-fielding
Skyler Grammer wrote on Dec 14, 2009
I'm trying to reduce CTIO-HYDRA data using dohydra and the results have been rather perplexing. dohydra does not appear to be flat-fielding the spectra. The ccd response is clearly dominating the shape of the spectrum and I can't seem to figure out why. I'll list my steps, parameters and examples.
Bias correction completed using ccdproc. Currently I'm creating an average domeflat and scaling it by the mode. Then, using the parameters listed below, I'm running dohydra.
Within dohydra, I identify the apertures using the domeflat then the apertures are fit interactively from the quartz flat and a 1D flat-field spectrum is created.
One note regarding the aperture identification: using Knut's Hydra Notes (www.ctio.noao.edu/spectrographs/hydra/hydra.html) he says to set the dispersion axis to 1. This is contrary to my understanding of the data. From my knowledge, setting the dispersion axis to 1 means that the dispersion axis is in the horizontal direction. However when displaying my images in ds9, it's clear that the dispersion axis in the vertical. Thus I have my dispaxis=2. Doyhdra claims that it is performing the flat-fielding but it is apparent from the standard star spectrum, the flat-fielding is not being performed.
As is apparent from my parameters, I'm not performing the wavelength calibration nor the sky-subtraction at this time.
If anyone has a clue as to why I'm getting such odd results, I'd really love to know! Even if you don't know what the problem is, I'd still like some feedback. Even ideas or things I could try would be helpful.
The images of relevant spectra and parameters are linked below.
Skyler
Fitted domeflat spectrum:
[img:26bd5144e3]http://img526.imageshack.us/img526/1856/dflat.th.png[/img:26bd5144e3]
Standard star after flat-fielding:
[img:26bd5144e3]http://img46.imageshack.us/img46/5093/hd86986.th.gif[/img:26bd5144e3]
My Parameters:
hydra:
(observatory = "observatory") Observatory of data
(interp = "poly5") Interpolation type
(dispaxis = 2) Image axis for 2D/3D images
(nsum = "1") Number of lines/columns/bands to sum for 2D/3D
(database = "database") Database
(verbose = yes) Verbose output?
(logfile = "logfile") Log file
(plotfile = "") Plot file\n
(records = "")
(version = "HYDRA V1: January 1992")
(mode = "ql")
($nargs = 0)
dohydra:
objects = "object" List of object spectra
(apref = "average_domeflat") Aperture reference spectrum
(flat = "quartzflat") Flat field spectrum
(throughput = "") Throughput file or image (optional)
(arcs1 = "") List of arc spectra
(arcs2 = "") List of shift arc spectra
(arcreplace = "") Special aperture replacements
(arctable = "") Arc assignment table (optional)\n
(readnoise = "5.2") Read out noise sigma (photons)
(gain = "2.36") Photon gain (photons/data number)
(datamax = 65000) Max data value / cosmic ray threshold
(fibers = 141) Number of fibers
(width = 12.) Width of profiles (pixels)
(minsep = 6.) Minimum separation between fibers (pixels)
(maxsep = 14.) Maximum separation between fibers (pixels)
(apidtable = "object") Aperture identifications
(crval = "INDEF") Approximate central wavelength
(cdelt = "INDEF") Approximate dispersion
(objaps = "") Object apertures
(skyaps = "") Sky apertures
(arcaps = "") Arc apertures
(objbeams = "0,1") Object beam numbers
(skybeams = "0") Sky beam numbers
(arcbeams = "") Arc beam numbers\n
(scattered = no) Subtract scattered light?
(fitflat = yes) Fit and ratio flat field spectrum?
(clean = yes) Detect and replace bad pixels?
(dispcor = no) Dispersion correct spectra?
(savearcs = no) Save simultaneous arc apertures?
(skyalign = no) Align sky lines?
(skysubtract = no) Subtract sky?
(skyedit = no) Edit the sky spectra?
(saveskys = no) Save sky spectra?
(splot = yes) Plot the final spectrum?
(redo = no) Redo operations if previously done?
(update = yes) Update spectra if cal data changes?
(batch = no) Extract objects in batch?
(listonly = no) List steps but don't process?\n
(params = "") Algorithm parameters
(mode = "ql")
params:
(line = INDEF) Default dispersion line
(nsum = 10) Number of dispersion lines to sum or median
(order = "increasing") Order of apertures
(extras = no) Extract sky, sigma, etc.?\n\n-- DEFAULT APERTUR
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center\n\n-- A
(ylevel = 0.05) Fraction of peak or intensity for resizing\n\n-
(t_step = 10) Tracing step
(t_function = "spline3") Trace fitting function
(t_order = 3) Trace fitting function order
(t_niterate = 1) Trace rejection iterations
(t_low = 3.) Trace lower rejection sigma
(t_high = 3.) Trace upper rejection sigma\n\n-- SCATTERED LIG
(buffer = 1.) Buffer distance from apertures
(apscat1 = "") Fitting parameters across the dispersion
(apscat2 = "") Fitting parameters along the dispersion\n\n-- A
(weights = "none") Extraction weights (none|variance)
(pfit = "fit1d") Profile fitting algorithm (fit1d|fit2d)
(lsigma = 3.) Lower rejection threshold
(usigma = 3.) Upper rejection threshold
(nsubaps = 1) Number of subapertures\n\n-- FLAT FIELD FUNCTIO
(f_interactiv = yes) Fit flat field interactively?
(f_function = "spline3") Fitting function
(f_order = 10) Fitting function order\n\n-- ARC DISPERSION FUN
(threshold = 10.) Minimum line contrast threshold
(coordlist = "linelists$ctiohenear.dat") Line list
(match = -3.) Line list matching limit in Angstroms
(fwidth = 4.) Arc line widths in pixels
(cradius = 10.) Centering radius in pixels
(i_function = "legendre") Coordinate function
(i_order = 5) Order of dispersion function
(i_niterate = 2) Rejection iterations
(i_low = 3.) Lower rejection sigma
(i_high = 3.) Upper rejection sigma
(refit = yes) Refit coordinate function when reidentifying?
(addfeatures = no) Add features when reidentifying?\n\n-- AUTOMATI
(select = "interp") Selection method for reference spectra
(sort = "jd") Sort key
(group = "ljd") Group key
(time = no) Is sort key a time?
(timewrap = 17.) Time wrap point for time sorting\n\n-- DISPERSI
(linearize = yes) Linearize (interpolate) spectra?
(log = no) Logarithmic wavelength scale?
(flux = yes) Conserve flux?\n\n-- SKY SUBTRACTION PARAMETERS
(combine = "average") Type of combine operation
(reject = "avsigclip") Sky rejection option
(scale = "none") Sky scaling option
(mode = "ql")
Bias correction completed using ccdproc. Currently I'm creating an average domeflat and scaling it by the mode. Then, using the parameters listed below, I'm running dohydra.
Within dohydra, I identify the apertures using the domeflat then the apertures are fit interactively from the quartz flat and a 1D flat-field spectrum is created.
One note regarding the aperture identification: using Knut's Hydra Notes (www.ctio.noao.edu/spectrographs/hydra/hydra.html) he says to set the dispersion axis to 1. This is contrary to my understanding of the data. From my knowledge, setting the dispersion axis to 1 means that the dispersion axis is in the horizontal direction. However when displaying my images in ds9, it's clear that the dispersion axis in the vertical. Thus I have my dispaxis=2. Doyhdra claims that it is performing the flat-fielding but it is apparent from the standard star spectrum, the flat-fielding is not being performed.
As is apparent from my parameters, I'm not performing the wavelength calibration nor the sky-subtraction at this time.
If anyone has a clue as to why I'm getting such odd results, I'd really love to know! Even if you don't know what the problem is, I'd still like some feedback. Even ideas or things I could try would be helpful.
The images of relevant spectra and parameters are linked below.
Skyler
Fitted domeflat spectrum:
[img:26bd5144e3]http://img526.imageshack.us/img526/1856/dflat.th.png[/img:26bd5144e3]
![[img:26bd5144e3]http://img526.imageshack.us/img526/1856/dflat.th.png[/img:26bd5144e3]](http://img526.imageshack.us/i/dflat.png/){kind=link}
Standard star after flat-fielding:
[img:26bd5144e3]http://img46.imageshack.us/img46/5093/hd86986.th.gif[/img:26bd5144e3]
![[img:26bd5144e3]http://img46.imageshack.us/img46/5093/hd86986.th.gif[/img:26bd5144e3]](http://img46.imageshack.us/i/hd86986.gif/){kind=link}
My Parameters:
hydra:
(observatory = "observatory") Observatory of data
(interp = "poly5") Interpolation type
(dispaxis = 2) Image axis for 2D/3D images
(nsum = "1") Number of lines/columns/bands to sum for 2D/3D
(database = "database") Database
(verbose = yes) Verbose output?
(logfile = "logfile") Log file
(plotfile = "") Plot file\n
(records = "")
(version = "HYDRA V1: January 1992")
(mode = "ql")
($nargs = 0)
dohydra:
objects = "object" List of object spectra
(apref = "average_domeflat") Aperture reference spectrum
(flat = "quartzflat") Flat field spectrum
(throughput = "") Throughput file or image (optional)
(arcs1 = "") List of arc spectra
(arcs2 = "") List of shift arc spectra
(arcreplace = "") Special aperture replacements
(arctable = "") Arc assignment table (optional)\n
(readnoise = "5.2") Read out noise sigma (photons)
(gain = "2.36") Photon gain (photons/data number)
(datamax = 65000) Max data value / cosmic ray threshold
(fibers = 141) Number of fibers
(width = 12.) Width of profiles (pixels)
(minsep = 6.) Minimum separation between fibers (pixels)
(maxsep = 14.) Maximum separation between fibers (pixels)
(apidtable = "object") Aperture identifications
(crval = "INDEF") Approximate central wavelength
(cdelt = "INDEF") Approximate dispersion
(objaps = "") Object apertures
(skyaps = "") Sky apertures
(arcaps = "") Arc apertures
(objbeams = "0,1") Object beam numbers
(skybeams = "0") Sky beam numbers
(arcbeams = "") Arc beam numbers\n
(scattered = no) Subtract scattered light?
(fitflat = yes) Fit and ratio flat field spectrum?
(clean = yes) Detect and replace bad pixels?
(dispcor = no) Dispersion correct spectra?
(savearcs = no) Save simultaneous arc apertures?
(skyalign = no) Align sky lines?
(skysubtract = no) Subtract sky?
(skyedit = no) Edit the sky spectra?
(saveskys = no) Save sky spectra?
(splot = yes) Plot the final spectrum?
(redo = no) Redo operations if previously done?
(update = yes) Update spectra if cal data changes?
(batch = no) Extract objects in batch?
(listonly = no) List steps but don't process?\n
(params = "") Algorithm parameters
(mode = "ql")
params:
(line = INDEF) Default dispersion line
(nsum = 10) Number of dispersion lines to sum or median
(order = "increasing") Order of apertures
(extras = no) Extract sky, sigma, etc.?\n\n-- DEFAULT APERTUR
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center\n\n-- A
(ylevel = 0.05) Fraction of peak or intensity for resizing\n\n-
(t_step = 10) Tracing step
(t_function = "spline3") Trace fitting function
(t_order = 3) Trace fitting function order
(t_niterate = 1) Trace rejection iterations
(t_low = 3.) Trace lower rejection sigma
(t_high = 3.) Trace upper rejection sigma\n\n-- SCATTERED LIG
(buffer = 1.) Buffer distance from apertures
(apscat1 = "") Fitting parameters across the dispersion
(apscat2 = "") Fitting parameters along the dispersion\n\n-- A
(weights = "none") Extraction weights (none|variance)
(pfit = "fit1d") Profile fitting algorithm (fit1d|fit2d)
(lsigma = 3.) Lower rejection threshold
(usigma = 3.) Upper rejection threshold
(nsubaps = 1) Number of subapertures\n\n-- FLAT FIELD FUNCTIO
(f_interactiv = yes) Fit flat field interactively?
(f_function = "spline3") Fitting function
(f_order = 10) Fitting function order\n\n-- ARC DISPERSION FUN
(threshold = 10.) Minimum line contrast threshold
(coordlist = "linelists$ctiohenear.dat") Line list
(match = -3.) Line list matching limit in Angstroms
(fwidth = 4.) Arc line widths in pixels
(cradius = 10.) Centering radius in pixels
(i_function = "legendre") Coordinate function
(i_order = 5) Order of dispersion function
(i_niterate = 2) Rejection iterations
(i_low = 3.) Lower rejection sigma
(i_high = 3.) Upper rejection sigma
(refit = yes) Refit coordinate function when reidentifying?
(addfeatures = no) Add features when reidentifying?\n\n-- AUTOMATI
(select = "interp") Selection method for reference spectra
(sort = "jd") Sort key
(group = "ljd") Group key
(time = no) Is sort key a time?
(timewrap = 17.) Time wrap point for time sorting\n\n-- DISPERSI
(linearize = yes) Linearize (interpolate) spectra?
(log = no) Logarithmic wavelength scale?
(flux = yes) Conserve flux?\n\n-- SKY SUBTRACTION PARAMETERS
(combine = "average") Type of combine operation
(reject = "avsigclip") Sky rejection option
(scale = "none") Sky scaling option
(mode = "ql")
Skyler Grammer wrote on Dec 14, 2009
I've run a few tests and I can say definitively that the flat-fielding it the problem. Simply extracting the spectra without any modification whatsoever yields a spectrum that has the same shape as the 'flat-fielded" spectrum but with higher counts. This suggests that the flat-fielding being performed is erroneous and perhaps is simply dividing by a number and not a curve.
Has anyone heard of issues like this or know someone who might be of some help?
Skyler
Has anyone heard of issues like this or know someone who might be of some help?
Skyler
Francisco Valdes wrote on Dec 14, 2009
Hello Skyler,
My apologies for the long delay. I have been on sabbatical and after returning officially to work have a major project that has taken most of my time. Our hope was that others would jump in and help in an open forum. But it is the case that I am the primary expert. Direct email is not encouraged but in this case it was justified. Sorry again.
I believe this is a misconception problem. The step that converts a spectrum to its true shape in flux (either per unit wavelength or flux) is the standard star calibrations. A flat field can change the shape in various ways. The true purpose of a flat field is to change the relative pixel response. So what the default parameters are doing is fitting the shape of the flat field spectrum (due to the color of the lamp and the large scale response of the detector to different wavelengths) and normalizing that away prior to dividing into the data. This means any shape in the fit will not introduce the reciprocal in the flat fielded data. What you are expecting is that the response of the detector is to be removed by the flat field which can be done but then you end up introducing the shape of the lamp continuum. As I said the standard star step is needed to get the true spectral shape.
After this mini-lecture, the thing that controls the fitting are the parameters f_function and f_order in the params parameter set. If you set the order to 1 then the function fit to the flat field will be a constant. Then when the flat field is normalized it is basically just adjusted to unit mean. Then when it is applied to the science data the reciprocal of the flat field shape is introduced. This will remove the common detector response but introduce the flat field lamp continuum. So to do what you want, as just noted, set f_order=1.
Yours,
Frank Valdes
My apologies for the long delay. I have been on sabbatical and after returning officially to work have a major project that has taken most of my time. Our hope was that others would jump in and help in an open forum. But it is the case that I am the primary expert. Direct email is not encouraged but in this case it was justified. Sorry again.
I believe this is a misconception problem. The step that converts a spectrum to its true shape in flux (either per unit wavelength or flux) is the standard star calibrations. A flat field can change the shape in various ways. The true purpose of a flat field is to change the relative pixel response. So what the default parameters are doing is fitting the shape of the flat field spectrum (due to the color of the lamp and the large scale response of the detector to different wavelengths) and normalizing that away prior to dividing into the data. This means any shape in the fit will not introduce the reciprocal in the flat fielded data. What you are expecting is that the response of the detector is to be removed by the flat field which can be done but then you end up introducing the shape of the lamp continuum. As I said the standard star step is needed to get the true spectral shape.
After this mini-lecture, the thing that controls the fitting are the parameters f_function and f_order in the params parameter set. If you set the order to 1 then the function fit to the flat field will be a constant. Then when the flat field is normalized it is basically just adjusted to unit mean. Then when it is applied to the science data the reciprocal of the flat field shape is introduced. This will remove the common detector response but introduce the flat field lamp continuum. So to do what you want, as just noted, set f_order=1.
Yours,
Frank Valdes
Last post on Dec 14, 2009