Extracting specific rows with apall
rachelm wrote on Jul 23, 2008
Hi,
Is there a way of non-interactively telling apall to extract specific rows from a data set? I.e., specifying the aperture centre on the command line? I've been able to set the aperture width on the cl, and set the centre using the colon keys in interactive mode, but I can't find a way of simply telling apall where to put the aperture and then letting it get on with the extraction.
Thanks!
Rachel
Is there a way of non-interactively telling apall to extract specific rows from a data set? I.e., specifying the aperture centre on the command line? I've been able to set the aperture width on the cl, and set the centre using the colon keys in interactive mode, but I can't find a way of simply telling apall where to put the aperture and then letting it get on with the extraction.
Thanks!
Rachel
rachelm wrote on Jul 23, 2008
By the way, I know that you can use scopy to extract a particular rows but I was hoping to use the background subtraction that's available in apall.
Francisco Valdes wrote on Jul 23, 2008
Hello Rachel,
There is no command line or parameter to define an aperture non-interactively. Yes this would be a good idea! However, you can achieve the behavior you want by substituting a file for the cursor input. The file would contain something like
where you replace the 50 with the line you want. You would also use the width parameters. See "help cursor" for some more information.
Now we get a bit tricky. To disable the queries you need to use a hidden parameter set.
Note setting initialize to "no" and setting the ans.... to YES or NO.
Finally you need to tell apall to use your cursor input file:
You need to be sure to "unlearn apall1" and cl.gcur="" to avoid later confusion.
Finally you would just do
and everything would happen non-interactively. To eliminate the graph you would redirect the graphics output with
Yours,
Frank Valdes
P.S. To get even trickier, you can change the cursor input to the terminal:
There is no command line or parameter to define an aperture non-interactively. Yes this would be a good idea! However, you can achieve the behavior you want by substituting a file for the cursor input. The file would contain something like
cl> type mycursor.dat
50 0 1 m
q
where you replace the 50 with the line you want. You would also use the width parameters. See "help cursor" for some more information.
Now we get a bit tricky. To disable the queries you need to use a hidden parameter set.
apextract> lpar apall1
e_output = Output spectra rootname
e_profiles = Profile reference image
(format = )apall.format) >apall.format
(extras = )apall.extras) >apall.extras
(dbwrite = "NO") Write to database?
(initialize = no) Initialize answers?
(verbose = )_.verbose) Verbose output?\n\n# DEFAULT APERTURE PARAMETER
(lower = )apall.lower) >apall.lower
(upper = )apall.upper) >apall.upper
(apidtable = )apall.apidtable) >apall.apidtable\n\n# DEFAULT BACKGROUND PARA
(b_function = )apall.b_function) >apall.b_function
(b_order = )apall.b_order) >apall.b_order
(b_sample = )apall.b_sample) >apall.b_sample
(b_naverage = )apall.b_naverage) >apall.b_naverage
(b_niterate = )apall.b_niterate) >apall.b_niterate
(b_low_reject = )apall.b_low_reject) >apall.b_low_reject
(b_high_rejec = )apall.b_high_reject) >apall.b_high_reject
(b_grow = )apall.b_grow) >apall.b_grow\n\n# APERTURE CENTERING PARAMETER
(width = )apall.width) >apall.width
(radius = )apall.radius) >apall.radius
(threshold = )apall.threshold) >apall.threshold\n\n# AUTOMATIC FINDING AND O
(nfind = )apall.nfind) >apall.nfind
(minsep = )apall.minsep) >apall.minsep
(maxsep = )apall.maxsep) >apall.maxsep
(order = )apall.order) >apall.order\n\n# RECENTERING PARAMETERS\n
(aprecenter = )apall.aprecenter) >apall.aprecenter
(npeaks = )apall.npeaks) >apall.npeaks
(shift = )apall.shift) >apall.shift\n\n# RESIZING PARAMETERS\n
(llimit = )apall.llimit) >apall.llimit
(ulimit = )apall.ulimit) >apall.ulimit
(ylevel = )apall.ylevel) >apall.ylevel
(peak = )apall.peak) >apall.peak
(bkg = )apall.bkg) >apall.bkg
(r_grow = )apall.r_grow) >apall.r_grow
(avglimits = )apall.avglimits) >apall.avglimits\n\n# EDITING PARAMETERS\n
(t_nsum = )apall.t_nsum) >apall.t_nsum
(t_step = )apall.t_step) >apall.t_step
(t_nlost = )apall.t_nlost) >apall.t_nlost
(t_width = )apall.width) >apall.width
(t_function = )apall.t_function) >apall.t_function
(t_order = )apall.t_order) >apall.t_order
(t_sample = )apall.t_sample) >apall.t_sample
(t_naverage = )apall.t_naverage) >apall.t_naverage
(t_niterate = )apall.t_niterate) >apall.t_niterate
(t_low_reject = )apall.t_low_reject) >apall.t_low_reject
(t_high_rejec = )apall.t_high_reject) >apall.t_high_reject
(t_grow = )apall.t_grow) >apall.t_grow\n\n# EXTRACTION PARAMETERS\n
(background = )apall.background) >apall.background
(skybox = )apall.skybox) >apall.skybox
(weights = )apall.weights) >apall.weights
(pfit = )apall.pfit) >apall.pfit
(clean = )apall.clean) >apall.clean
(nclean = 0.5) Maximum number of pixels to clean
(niterate = 5) Number of profile fitting iterations
(saturation = )apall.saturation) >apall.saturation
(readnoise = )apall.readnoise) >apall.readnoise
(gain = )apall.gain) >apall.gain
(lsigma = )apall.lsigma) >apall.lsigma
(usigma = )apall.usigma) >apall.usigma
(polysep = 0.9) Marsh algorithm polynomial spacing
(polyorder = 10) Marsh algorithm polynomial order
(nsubaps = )apall.nsubaps) >apall.nsubaps\n\n# ANSWER PARAMETERS\n
(ansclobber = "no")
(ansclobber1 = "no")
(ansdbwrite = "NO")
(ansdbwrite1 = "yes")
(ansedit = "YES")
(ansextract = "YES")
(ansfind = "NO")
(ansfit = "NO")
(ansfitscatte = "yes")
(ansfitsmooth = "yes")
(ansfitspec = "yes")
(ansfitspec1 = "yes")
(ansfittrace = "no")
(ansfittrace1 = "NO")
(ansflat = "yes")
(ansnorm = "yes")
(ansrecenter = "yes")
(ansresize = "yes")
(ansreview = "NO")
(ansreview1 = "NO")
(ansscat = "yes")
(anssmooth = "yes")
(anstrace = "NO")
(mode = "ql")
Note setting initialize to "no" and setting the ans.... to YES or NO.
Finally you need to tell apall to use your cursor input file:
cl.gcur="mycursor.dat"
You need to be sure to "unlearn apall1" and cl.gcur="" to avoid later confusion.
Finally you would just do
apall inimage
and everything would happen non-interactively. To eliminate the graph you would redirect the graphics output with
apall inimage >G dev$null Yours,
Frank Valdes
P.S. To get even trickier, you can change the cursor input to the terminal:
ap> cl.gcur="STDIN"
ap> apall test >G dev$null
50 0 1 m
aperture = 1 beam = 1 center = 49.91 low = -5.00 upper = 5.00
q
Extracting apertures ...
Jul 24 15:03: EXTRACT - Aperture 1 from test --> test.ms
rachelm wrote on Jul 23, 2008
Hi Frank,
Thanks for helping! I just tried to naively follow your instructions, but it appears that I'm doing something wrong. Here's what I'm doing:
cl> twodspec
cl> apextract
cl> imdel test.ms
cl> type mycursor.dat
95.5 0 1 m
q
cl> epar apall1
set initialize=no and
(ansclobber = "no")
(ansclobber1 = "no")
(ansdbwrite = "NO")
(ansdbwrite1 = "yes")
(ansedit = "YES")
(ansextract = "YES")
(ansfind = "NO")
(ansfit = "NO")
(ansfitscatte = "yes")
(ansfitsmooth = "yes")
(ansfitspec = "yes")
(ansfitspec1 = "yes")
(ansfittrace = "no")
(ansfittrace1 = "NO")
(ansflat = "yes")
(ansnorm = "yes")
(ansrecenter = "yes")
(ansresize = "yes")
(ansreview = "NO")
(ansreview1 = "NO")
(ansscat = "yes")
(anssmooth = "yes")
(anstrace = "NO")
cl>cl.gcur="mycursor.dat"
cl>apall test
At this point apall quits and in the graphics window it complains "No apertures defined for test". I've experimented with a few parameters and had a look at the (scary) cursor help but I can't see what the problem is. Any ideas?
Cheers,
Rachel
Thanks for helping! I just tried to naively follow your instructions, but it appears that I'm doing something wrong. Here's what I'm doing:
cl> twodspec
cl> apextract
cl> imdel test.ms
cl> type mycursor.dat
95.5 0 1 m
q
cl> epar apall1
set initialize=no and
(ansclobber = "no")
(ansclobber1 = "no")
(ansdbwrite = "NO")
(ansdbwrite1 = "yes")
(ansedit = "YES")
(ansextract = "YES")
(ansfind = "NO")
(ansfit = "NO")
(ansfitscatte = "yes")
(ansfitsmooth = "yes")
(ansfitspec = "yes")
(ansfitspec1 = "yes")
(ansfittrace = "no")
(ansfittrace1 = "NO")
(ansflat = "yes")
(ansnorm = "yes")
(ansrecenter = "yes")
(ansresize = "yes")
(ansreview = "NO")
(ansreview1 = "NO")
(ansscat = "yes")
(anssmooth = "yes")
(anstrace = "NO")
cl>cl.gcur="mycursor.dat"
cl>apall test
At this point apall quits and in the graphics window it complains "No apertures defined for test". I've experimented with a few parameters and had a look at the (scary) cursor help but I can't see what the problem is. Any ideas?
Cheers,
Rachel
Francisco Valdes wrote on Jul 23, 2008
Hi,
All that looks correct. Could you send the apall parameters too. Do you have an existing aperture definition database file? The behavior appears like it is not getting into the aperture editor where the aperture marking takes place.
The cursor stuff may be long but it is not so complicated. Let me summarize. Whenever a task reads the graphics cursor it gets back
x y id key [colon command]
x and y are the coordinates in the current graph coordinate system. id is a graph id which is 1 for all but very sophisticated graphics programs. key is what is entered which is either a single character or :. In the latter case of a colon this is followed by the colon command string. If the cursor input does not depend on position in the graph the "x y id" can be left off.
A lot of the info in the cursor help is about "cursor mode" which is a layer of keys that apply to all tasks. These can be useful at times but don't apply to the cursor files I'm talking about.
Cheers,
Frank
All that looks correct. Could you send the apall parameters too. Do you have an existing aperture definition database file? The behavior appears like it is not getting into the aperture editor where the aperture marking takes place.
The cursor stuff may be long but it is not so complicated. Let me summarize. Whenever a task reads the graphics cursor it gets back
x y id key [colon command]
x and y are the coordinates in the current graph coordinate system. id is a graph id which is 1 for all but very sophisticated graphics programs. key is what is entered which is either a single character or :. In the latter case of a colon this is followed by the colon command string. If the cursor input does not depend on position in the graph the "x y id" can be left off.
A lot of the info in the cursor help is about "cursor mode" which is a layer of keys that apply to all tasks. These can be useful at times but don't apply to the cursor files I'm talking about.
Cheers,
Frank
rachelm wrote on Jul 23, 2008
Hi Frank,
That stuff about the cursor makes sense. My apall parameter file is below; normally I'd set it up a bit differently but I've basically been sticking to the defaults for this test.
I assume that the hidden parameters overrule the usual apall parameters because it seems to make no difference whether I set find=yes or no in the apall parameter file. As for the database file, I have been deleting those (database/aplast and database/aptest) because if I don't, apall uses the aperture centre that I'd previously used for extracting the spectrum interactively, rather than the one in my cursor file. Is that not what I should be doing?
Thanks,
Rachel
input = "test" List of input images
nfind = Number of apertures to be found automatically
(output = "") List of output spectra
(apertures = "") Apertures
(format = "multispec") Extracted spectra format
(references = "") List of aperture reference images
(profiles = "") List of aperture profile images\n
(interactive = yes) Run task interactively?
(find = yes) Find apertures?
(recenter = yes) Recenter apertures?
(resize = yes) Resize apertures?
(edit = yes) Edit apertures?
(trace = yes) Trace apertures?
(fittrace = yes) Fit the traced points interactively?
(extract = yes) Extract spectra?
(extras = yes) Extract sky, sigma, etc.?
(review = yes) Review extractions?\n
(line = INDEF) Dispersion line
(nsum = 10) Number of dispersion lines to sum or median\n
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center
(apidtable = "") Aperture ID table (optional)\n\n# DEFAULT BAC
(b_function = "chebyshev") Background function
(b_order = 1) Background function order
(b_sample = "-10:-6,6:10") Background sample regions
(b_naverage = -3) Background average or median
(b_niterate = 0) Background rejection iterations
(b_low_reject = 3.) Background lower rejection sigma
(b_high_rejec = 3.) Background upper rejection sigma
(b_grow = 0.) Background rejection growing radius\n\n# APER
(width = 5.) Profile centering width
(radius = 10.) Profile centering radius
(threshold = 0.) Detection threshold for profile centering\n\n
(minsep = 5.) Minimum separation between spectra
(maxsep = 1000.) Maximum separation between spectra
(order = "increasing") Order of apertures\n\n# RECENTERING PARAMETER
(aprecenter = "") Apertures for recentering calculation
(npeaks = INDEF) Select brightest peaks
(shift = yes) Use average shift instead of recentering?\n\n
(llimit = INDEF) Lower aperture limit relative to center
(ulimit = INDEF) Upper aperture limit relative to center
(ylevel = 0.1) Fraction of peak or intensity for automatic w
(peak = yes) Is ylevel a fraction of the peak?
(bkg = yes) Subtract background in automatic width?
(r_grow = 0.) Grow limits by this factor
(avglimits = no) Average limits over all apertures?\n\n# TRACI
(t_nsum = 10) Number of dispersion lines to sum
(t_step = 10) Tracing step
(t_nlost = 3) Number of consecutive times profile is lost b
(t_function = "legendre") Trace fitting function
(t_order = 2) Trace fitting function order
(t_sample = "*") Trace sample regions
(t_naverage = 1) Trace average or median
(t_niterate = 0) Trace rejection iterations
(t_low_reject = 3.) Trace lower rejection sigma
(t_high_rejec = 3.) Trace upper rejection sigma
(t_grow = 0.) Trace rejection growing radius\n\n# EXTRACTIO
(background = "none") Background to subtract
(skybox = 1) Box car smoothing length for sky
(weights = "none") Extraction weights (none|variance)
(pfit = "fit1d") Profile fitting type (fit1d|fit2d)
(clean = no) Detect and replace bad pixels?
(saturation = INDEF) Saturation level
(readnoise = "0.") Read out noise sigma (photons)
(gain = "1.") Photon gain (photons/data number)
(lsigma = 4.) Lower rejection threshold
(usigma = 4.) Upper rejection threshold
(nsubaps = 1) Number of subapertures per aperture
(mode = "ql")
That stuff about the cursor makes sense. My apall parameter file is below; normally I'd set it up a bit differently but I've basically been sticking to the defaults for this test.
I assume that the hidden parameters overrule the usual apall parameters because it seems to make no difference whether I set find=yes or no in the apall parameter file. As for the database file, I have been deleting those (database/aplast and database/aptest) because if I don't, apall uses the aperture centre that I'd previously used for extracting the spectrum interactively, rather than the one in my cursor file. Is that not what I should be doing?
Thanks,
Rachel
input = "test" List of input images
nfind = Number of apertures to be found automatically
(output = "") List of output spectra
(apertures = "") Apertures
(format = "multispec") Extracted spectra format
(references = "") List of aperture reference images
(profiles = "") List of aperture profile images\n
(interactive = yes) Run task interactively?
(find = yes) Find apertures?
(recenter = yes) Recenter apertures?
(resize = yes) Resize apertures?
(edit = yes) Edit apertures?
(trace = yes) Trace apertures?
(fittrace = yes) Fit the traced points interactively?
(extract = yes) Extract spectra?
(extras = yes) Extract sky, sigma, etc.?
(review = yes) Review extractions?\n
(line = INDEF) Dispersion line
(nsum = 10) Number of dispersion lines to sum or median\n
(lower = -5.) Lower aperture limit relative to center
(upper = 5.) Upper aperture limit relative to center
(apidtable = "") Aperture ID table (optional)\n\n# DEFAULT BAC
(b_function = "chebyshev") Background function
(b_order = 1) Background function order
(b_sample = "-10:-6,6:10") Background sample regions
(b_naverage = -3) Background average or median
(b_niterate = 0) Background rejection iterations
(b_low_reject = 3.) Background lower rejection sigma
(b_high_rejec = 3.) Background upper rejection sigma
(b_grow = 0.) Background rejection growing radius\n\n# APER
(width = 5.) Profile centering width
(radius = 10.) Profile centering radius
(threshold = 0.) Detection threshold for profile centering\n\n
(minsep = 5.) Minimum separation between spectra
(maxsep = 1000.) Maximum separation between spectra
(order = "increasing") Order of apertures\n\n# RECENTERING PARAMETER
(aprecenter = "") Apertures for recentering calculation
(npeaks = INDEF) Select brightest peaks
(shift = yes) Use average shift instead of recentering?\n\n
(llimit = INDEF) Lower aperture limit relative to center
(ulimit = INDEF) Upper aperture limit relative to center
(ylevel = 0.1) Fraction of peak or intensity for automatic w
(peak = yes) Is ylevel a fraction of the peak?
(bkg = yes) Subtract background in automatic width?
(r_grow = 0.) Grow limits by this factor
(avglimits = no) Average limits over all apertures?\n\n# TRACI
(t_nsum = 10) Number of dispersion lines to sum
(t_step = 10) Tracing step
(t_nlost = 3) Number of consecutive times profile is lost b
(t_function = "legendre") Trace fitting function
(t_order = 2) Trace fitting function order
(t_sample = "*") Trace sample regions
(t_naverage = 1) Trace average or median
(t_niterate = 0) Trace rejection iterations
(t_low_reject = 3.) Trace lower rejection sigma
(t_high_rejec = 3.) Trace upper rejection sigma
(t_grow = 0.) Trace rejection growing radius\n\n# EXTRACTIO
(background = "none") Background to subtract
(skybox = 1) Box car smoothing length for sky
(weights = "none") Extraction weights (none|variance)
(pfit = "fit1d") Profile fitting type (fit1d|fit2d)
(clean = no) Detect and replace bad pixels?
(saturation = INDEF) Saturation level
(readnoise = "0.") Read out noise sigma (photons)
(gain = "1.") Photon gain (photons/data number)
(lsigma = 4.) Lower rejection threshold
(usigma = 4.) Upper rejection threshold
(nsubaps = 1) Number of subapertures per aperture
(mode = "ql")
Francisco Valdes wrote on Jul 23, 2008
All that looks correct. Yes, the hidden parameters with initialize=no means it does not matter with the switches are set to in apall.
The only advice I can give now is to do everything interactively once. In particularly, see that the 'm' marking does result in an aperture. A thought just occurred to me. The 'm' key is not guaranteed to succeed since it depends on centering on a profile. Instead use 'n' which defines an aperture at the cursor position (such as specified in your cursor input file) without centering.
Frank
The only advice I can give now is to do everything interactively once. In particularly, see that the 'm' marking does result in an aperture. A thought just occurred to me. The 'm' key is not guaranteed to succeed since it depends on centering on a profile. Instead use 'n' which defines an aperture at the cursor position (such as specified in your cursor input file) without centering.
Frank
rachelm wrote on Jul 23, 2008
That works! If I use n instead of m, I mean, and make sure to type cl.gcur="mycursor.dat" every time I repeat the extraction.
Thanks very much for your help, this is going to be useful.
Rachel
Thanks very much for your help, this is going to be useful.
Rachel
Last post on Jul 23, 2008