dohydra on linux RHEL3 does not work
Dario Fadda wrote on May 14, 2006
I'm trying to run the Hydra pipeline (noao/imred/hydra) on my linux laptop
(Thinkpad T3, 1GB RAM, Redhat Linux Entreprise 3).
The versions of IRAF that I tried are: 2.12.2 with patch2a and the last
beta version 2.13 beta2. Everything (like displaying, combining, etc..)
works well.
With the two versions, the pipeline stops at the identification of the lines in the arcs.
It does not give any error message. Simply hangs up.
The problem appears when the pipeline uses the code "autoidentify"
in the procedure: noao/imred/src/fibers/arcrefs.cl
I also noticed the same type of problems during another stage of the pipeline
when it is calling the code "reidentify".
The pipeline runs perfectly on my old Sun/Solaris machine.
I would like to know if this is a known problem or just a problem with my installation. I didn't try to recompile the codes on my machine, also because
I don't know exactly how to do it.
Thanks, Dario
(Thinkpad T3, 1GB RAM, Redhat Linux Entreprise 3).
The versions of IRAF that I tried are: 2.12.2 with patch2a and the last
beta version 2.13 beta2. Everything (like displaying, combining, etc..)
works well.
With the two versions, the pipeline stops at the identification of the lines in the arcs.
It does not give any error message. Simply hangs up.
The problem appears when the pipeline uses the code "autoidentify"
in the procedure: noao/imred/src/fibers/arcrefs.cl
I also noticed the same type of problems during another stage of the pipeline
when it is calling the code "reidentify".
The pipeline runs perfectly on my old Sun/Solaris machine.
I would like to know if this is a known problem or just a problem with my installation. I didn't try to recompile the codes on my machine, also because
I don't know exactly how to do it.
Thanks, Dario
Francisco Valdes wrote on May 14, 2006
Hi Dario,
This is not a previous reported problem as far as I'm aware. You are probably right but just a comment that maybe AUTOIDENTIFY is not "hanging" but simply taking a very long time. This routine can be very slow. However, REIDENTIFY would not be a case of slowness. I don't know what to tell you further.
Since it sounds like you understand enough to find the source code and look at it I could suggest you put the command "d_trace" immediately before the problem call. This turns on some debugging information that is written to the terminal. To turn this off you use "d_trace" again (it is a toggle command). If it hangs shortly after printing out the information you could forward the output to me to look at. As far as I know there should be nothing unusual about your OS and version of IRAF and the behavior should be the same as you are used to on other systems.
Yours,
Frank Valdes
This is not a previous reported problem as far as I'm aware. You are probably right but just a comment that maybe AUTOIDENTIFY is not "hanging" but simply taking a very long time. This routine can be very slow. However, REIDENTIFY would not be a case of slowness. I don't know what to tell you further.
Since it sounds like you understand enough to find the source code and look at it I could suggest you put the command "d_trace" immediately before the problem call. This turns on some debugging information that is written to the terminal. To turn this off you use "d_trace" again (it is a toggle command). If it hangs shortly after printing out the information you could forward the output to me to look at. As far as I know there should be nothing unusual about your OS and version of IRAF and the behavior should be the same as you are used to on other systems.
Yours,
Frank Valdes
Dario Fadda wrote on May 14, 2006
Dear Frank,
thanks for your reply.
I did what you suggested (inserting d_trace before the call of autoidentify
in the routine noao/imred/src/fibers/arcrefs.cl).
In the following I wrote the parameters I chose to run dohydra and the
output of the code:
hydra> dohydra.apref="Flat_A1731_1"
hydra> dohydra.flat="Flat_A1731_1"
hydra> dohydra.arcs1="Arc_A1731_1"
hydra> dohydra.readnoi="RDNOISE"
hydra> dohydra.gain="GAIN"
hydra> dohydra.apidtab="Flat_A1731_1"
hydra> dohydra.datamax=65000.
hydra> dohydra.crval=6510.
hydra> dohydra.skyedit=yes
hydra> dohydra.savesky=yes
hydra> dohydra.skysubt=yes
hydra> dohydra.fibers=98
hydra> params.coordli="linelists$cuar.dat"
hydra> params.match=10.
hydra> params.order="decreasing"
hydra> params.t_niter = 5
hydra> params.f_order = 50
hydra> dohydra A1731_1
Set reference apertures for Flat_A1731_1
Resize apertures for Flat_A1731_1? (yes):
Edit apertures for Flat_A1731_1? (yes):
Create response function Flat_A1731_1Flat_1.ms
Extract flat field Flat_A1731_1
Fit and ratio flat field Flat_A1731_1
Create the normalized response Flat_A1731_1Flat_1.ms
Extract arc reference image Arc_A1731_1
Determine dispersion solution for Arc_A1731_1
154 17270+7: call autoidentify
154 17277+6: pushparam arcrefms
154 17283+4: posargset 0
154 17287+5: pushparam crval
154 17292+4: posargset 1
154 17296+5: pushparam cdelt
154 17301+4: posargset 2
155 17305+8: pushparam params.coordlist
155 17313+6: absargset coordlist
156 17319+7: pushconst `YES'
156 17326+6: absargset interactive
156 17332+9: pushconst `middle line'
156 17341+5: absargset section
156 17346+7: pushconst `1'
156 17353+5: absargset nsum
157 17358+9: pushconst `emission'
157 17367+5: absargset ftype
157 17372+7: pushparam params.fwidth
157 17379+5: absargset fwidth
158 17384+7: pushparam params.cradius
158 17391+5: absargset cradius
158 17396+8: pushparam params.threshold
158 17404+6: absargset threshold
159 17410+6: pushconst 2.
159 17416+5: absargset minsep
159 17421+7: pushparam params.match
159 17428+5: absargset match
159 17433+8: pushparam params.i_function
159 17441+6: absargset function
160 17447+7: pushparam params.i_order
160 17454+5: absargset order
160 17459+7: pushconst `*'
160 17466+5: absargset sample
161 17471+8: pushparam params.i_niterate
161 17479+6: absargset niterate
161 17485+7: pushparam params.i_low
161 17492+6: absargset low_reject
162 17498+7: pushparam params.i_high
162 17505+6: absargset high_reject
162 17511+6: pushconst 0.
162 17517+5: absargset grow
162 17522+7: pushconst `YES'
162 17529+5: absargset dbwrite
163 17534+6: pushconst yes
163 17540+6: absargset overwrite
163 17546+9: pushconst `database'
163 17555+6: absargset database
163 17561+6: pushconst yes
163 17567+5: absargset verbose
164 17572+5: pushparam logfile
164 17577+5: absargset logfile
164 17582+6: pushparam plotfile
164 17588+6: absargset plotfile
165 17594+7: pushconst `'
165 17601+5: absargset reflist
165 17606+7: pushconst `'
165 17613+5: absargset refspec
165 17618+8: pushconst `INDEF'
165 17626+5: absargset crpix
165 17631+8: pushconst `unknown'
165 17639+5: absargset cddir
166 17644+8: pushconst `-0.5'
166 17652+6: absargset crsearch
166 17658+8: pushconst `INDEF'
166 17666+6: absargset cdsearch
166 17672+7: pushconst `'
166 17679+5: absargset aidpars
166 17684+3: exec
----- exec external task autoidentify - line 169 -----
autoidentify\nimages="Arc_A1731_1Flat_1.ms"\ncrval="6510."\n
cdelt="INDEF"\ncoordlist="linelists$cuar.dat"\nunits=""\n
interactive="YES"\naidpars=""\nsection="middle line"\n
nsum="1"\nftype="emission"\nfwidth=4.\ncradius=10.\n
threshold=10.\nminsep=2.\nmatch=10.\nfunction="spline3"\n
order=2\nsample="*"\nniterate=2\nlow_reject=3.\n
high_reject=3.\ngrow=0.\ndbwrite="YES"\noverwrite=yes\n
database="database"\nverbose=yes\nlogfile="logfile"\n
plotfile=""\ngraphics="stdgraph"\nmode="ql"\n$nargs=3\n
aidpars.reflist=""\naidpars.refspec=""\n
aidpars.crpix="INDEF"\naidpars.crquad="INDEF"\n
aidpars.cddir="sign"\naidpars.crsearch="INDEF"\n
aidpars.cdsearch="INDEF"\naidpars.ntarget=100\n
aidpars.npattern=5\naidpars.nneighbors=10\naidpars.nbins=6\n
aidpars.ndmax=500\naidpars.aidord=3\naidpars.maxnl=0.02\n
aidpars.nfound=6\naidpars.sigma=0.05\naidpars.minratio=0.1\n
aidpars.rms=0.1\naidpars.fmatch=0.2\naidpars.debug=""\n
aidpars.mode="ql"\n
--------------------------------
----- task autoidentify - line 3029 (INSPECT) -----
3029 22642+7: inspect identify.zwidth
3029 22649+3: end
At this point the code runs without giving any signal of life ...
The same code on my old Solaris/Sun takes just a few second to assign
the lines.
Thanks for your help,
Dario
thanks for your reply.
I did what you suggested (inserting d_trace before the call of autoidentify
in the routine noao/imred/src/fibers/arcrefs.cl).
In the following I wrote the parameters I chose to run dohydra and the
output of the code:
hydra> dohydra.apref="Flat_A1731_1"
hydra> dohydra.flat="Flat_A1731_1"
hydra> dohydra.arcs1="Arc_A1731_1"
hydra> dohydra.readnoi="RDNOISE"
hydra> dohydra.gain="GAIN"
hydra> dohydra.apidtab="Flat_A1731_1"
hydra> dohydra.datamax=65000.
hydra> dohydra.crval=6510.
hydra> dohydra.skyedit=yes
hydra> dohydra.savesky=yes
hydra> dohydra.skysubt=yes
hydra> dohydra.fibers=98
hydra> params.coordli="linelists$cuar.dat"
hydra> params.match=10.
hydra> params.order="decreasing"
hydra> params.t_niter = 5
hydra> params.f_order = 50
hydra> dohydra A1731_1
Set reference apertures for Flat_A1731_1
Resize apertures for Flat_A1731_1? (yes):
Edit apertures for Flat_A1731_1? (yes):
Create response function Flat_A1731_1Flat_1.ms
Extract flat field Flat_A1731_1
Fit and ratio flat field Flat_A1731_1
Create the normalized response Flat_A1731_1Flat_1.ms
Extract arc reference image Arc_A1731_1
Determine dispersion solution for Arc_A1731_1
154 17270+7: call autoidentify
154 17277+6: pushparam arcrefms
154 17283+4: posargset 0
154 17287+5: pushparam crval
154 17292+4: posargset 1
154 17296+5: pushparam cdelt
154 17301+4: posargset 2
155 17305+8: pushparam params.coordlist
155 17313+6: absargset coordlist
156 17319+7: pushconst `YES'
156 17326+6: absargset interactive
156 17332+9: pushconst `middle line'
156 17341+5: absargset section
156 17346+7: pushconst `1'
156 17353+5: absargset nsum
157 17358+9: pushconst `emission'
157 17367+5: absargset ftype
157 17372+7: pushparam params.fwidth
157 17379+5: absargset fwidth
158 17384+7: pushparam params.cradius
158 17391+5: absargset cradius
158 17396+8: pushparam params.threshold
158 17404+6: absargset threshold
159 17410+6: pushconst 2.
159 17416+5: absargset minsep
159 17421+7: pushparam params.match
159 17428+5: absargset match
159 17433+8: pushparam params.i_function
159 17441+6: absargset function
160 17447+7: pushparam params.i_order
160 17454+5: absargset order
160 17459+7: pushconst `*'
160 17466+5: absargset sample
161 17471+8: pushparam params.i_niterate
161 17479+6: absargset niterate
161 17485+7: pushparam params.i_low
161 17492+6: absargset low_reject
162 17498+7: pushparam params.i_high
162 17505+6: absargset high_reject
162 17511+6: pushconst 0.
162 17517+5: absargset grow
162 17522+7: pushconst `YES'
162 17529+5: absargset dbwrite
163 17534+6: pushconst yes
163 17540+6: absargset overwrite
163 17546+9: pushconst `database'
163 17555+6: absargset database
163 17561+6: pushconst yes
163 17567+5: absargset verbose
164 17572+5: pushparam logfile
164 17577+5: absargset logfile
164 17582+6: pushparam plotfile
164 17588+6: absargset plotfile
165 17594+7: pushconst `'
165 17601+5: absargset reflist
165 17606+7: pushconst `'
165 17613+5: absargset refspec
165 17618+8: pushconst `INDEF'
165 17626+5: absargset crpix
165 17631+8: pushconst `unknown'
165 17639+5: absargset cddir
166 17644+8: pushconst `-0.5'
166 17652+6: absargset crsearch
166 17658+8: pushconst `INDEF'
166 17666+6: absargset cdsearch
166 17672+7: pushconst `'
166 17679+5: absargset aidpars
166 17684+3: exec
----- exec external task autoidentify - line 169 -----
autoidentify\nimages="Arc_A1731_1Flat_1.ms"\ncrval="6510."\n
cdelt="INDEF"\ncoordlist="linelists$cuar.dat"\nunits=""\n
interactive="YES"\naidpars=""\nsection="middle line"\n
nsum="1"\nftype="emission"\nfwidth=4.\ncradius=10.\n
threshold=10.\nminsep=2.\nmatch=10.\nfunction="spline3"\n
order=2\nsample="*"\nniterate=2\nlow_reject=3.\n
high_reject=3.\ngrow=0.\ndbwrite="YES"\noverwrite=yes\n
database="database"\nverbose=yes\nlogfile="logfile"\n
plotfile=""\ngraphics="stdgraph"\nmode="ql"\n$nargs=3\n
aidpars.reflist=""\naidpars.refspec=""\n
aidpars.crpix="INDEF"\naidpars.crquad="INDEF"\n
aidpars.cddir="sign"\naidpars.crsearch="INDEF"\n
aidpars.cdsearch="INDEF"\naidpars.ntarget=100\n
aidpars.npattern=5\naidpars.nneighbors=10\naidpars.nbins=6\n
aidpars.ndmax=500\naidpars.aidord=3\naidpars.maxnl=0.02\n
aidpars.nfound=6\naidpars.sigma=0.05\naidpars.minratio=0.1\n
aidpars.rms=0.1\naidpars.fmatch=0.2\naidpars.debug=""\n
aidpars.mode="ql"\n
--------------------------------
----- task autoidentify - line 3029 (INSPECT) -----
3029 22642+7: inspect identify.zwidth
3029 22649+3: end
At this point the code runs without giving any signal of life ...
The same code on my old Solaris/Sun takes just a few second to assign
the lines.
Thanks for your help,
Dario
Francisco Valdes wrote on May 14, 2006
Hi Dario,
Thank you for the information. This tells me the exact command and parameters. Of hand I don't see any thing obviously wrong. The thing I can offer is that if you want to put the file Arc_A1731_1Flat_1.ms on the anonymous ftp directory ftp://iraf.noao.edu/pub and let me know you have done that I'll run the command on systems here at NOAO and see what happens.
Yours,
Frank
Thank you for the information. This tells me the exact command and parameters. Of hand I don't see any thing obviously wrong. The thing I can offer is that if you want to put the file Arc_A1731_1Flat_1.ms on the anonymous ftp directory ftp://iraf.noao.edu/pub and let me know you have done that I'll run the command on systems here at NOAO and see what happens.
Yours,
Frank
Dario Fadda wrote on May 14, 2006
Dear Frank,
I put the data for the test on:
ftp://iraf.noao.edu/pub/DarioFadda.tgz
I included also a README file with the parameters used by autoidentify
when running the hydra pipeline (I copied them from the d_trace output).
I tested them on my linux laptop obtaining the same result as in the case
of the hydra pipeline.
Thanks for your help,
Dario
I put the data for the test on:
ftp://iraf.noao.edu/pub/DarioFadda.tgz
I included also a README file with the parameters used by autoidentify
when running the hydra pipeline (I copied them from the d_trace output).
I tested them on my linux laptop obtaining the same result as in the case
of the hydra pipeline.
Thanks for your help,
Dario
Francisco Valdes wrote on May 14, 2006
Hi Dario,
I'm sorry for not replying sooner. I did receive your data and have reproduced the problem. The automatic identification algorithm does not always work and can take a very long time if an initial wavelength and dispersion are not given. I suggest you set the values to INDEF in DOHYDRA and just rely on the old-fashioned identification from a line list.
You might find the following reference useful
http://www.noao.edu/kpno/specatlas/cuar/cuar.html
Yours,
Frank
I'm sorry for not replying sooner. I did receive your data and have reproduced the problem. The automatic identification algorithm does not always work and can take a very long time if an initial wavelength and dispersion are not given. I suggest you set the values to INDEF in DOHYDRA and just rely on the old-fashioned identification from a line list.
You might find the following reference useful
http://www.noao.edu/kpno/specatlas/cuar/cuar.html
Yours,
Frank
Dario Fadda wrote on May 14, 2006
Dear Frank,
thanks for your help. I finally found why the autoidentify was finding a solution
on my SUN and not on my linux box.
The difference was in how the aidpars were set.
Once I used the same settings as for the SUN:
aidpars.reflist=""
aidpars.refspec=""
aidpars.crpix="INDEF"
aidpars.crquad="INDEF"
aidpars.cddir="increasing"
aidpars.crsearch="INDEF"
aidpars.cdsearch="INDEF"
aidpars.ntarget=30
aidpars.npattern=5
aidpars.nneighbors=10
aidpars.nbins=6
aidpars.ndmax=20
aidpars.aidord=3
aidpars.maxnl=0.02
aidpars.nfound=6
aidpars.sigma=0.05
aidpars.minratio=0.1
aidpars.rms=0.3
aidpars.fmatch=0.3
aidpars.debug=""
aidpars.mode="ql"
the pipeline worked fine also on my linux laptop.
Thanks again for your advices,
cheers, Dario
thanks for your help. I finally found why the autoidentify was finding a solution
on my SUN and not on my linux box.
The difference was in how the aidpars were set.
Once I used the same settings as for the SUN:
aidpars.reflist=""
aidpars.refspec=""
aidpars.crpix="INDEF"
aidpars.crquad="INDEF"
aidpars.cddir="increasing"
aidpars.crsearch="INDEF"
aidpars.cdsearch="INDEF"
aidpars.ntarget=30
aidpars.npattern=5
aidpars.nneighbors=10
aidpars.nbins=6
aidpars.ndmax=20
aidpars.aidord=3
aidpars.maxnl=0.02
aidpars.nfound=6
aidpars.sigma=0.05
aidpars.minratio=0.1
aidpars.rms=0.3
aidpars.fmatch=0.3
aidpars.debug=""
aidpars.mode="ql"
the pipeline worked fine also on my linux laptop.
Thanks again for your advices,
cheers, Dario
Dario Fadda wrote on May 14, 2006
Just a final note about how I modified the pipeline.
I changed the line in:
hydra$hydra.cl
task aidpars = "hydra$aidpars.par"
and I added the file aidpars.par in the hydra directory:
# Parameters for autoidentify task.
reflist,s,h,"",,,Reference coordinate list
refspec,s,h,"",,,Reference spectrum
#crval,s,h,"INDEF",,,Coordinate reference value
#cdelt,s,h,"INDEF",,,Coordinate interval per pixel
crpix,s,h,"INDEF",,,Coordinate reference pixel
crquad,s,h,"INDEF",,,Quadratic pixel distortion at reference pixel
cddir,s,h,"increasing","unknown|sign|increasing|decreasing",,Dispersion direction
crsearch,s,h,"INDEF",,,Coordinate value search radius
cdsearch,s,h,"INDEF",,,Coordinate interval search radius
ntarget,i,h,30,,,Number of target features
#nreference,i,h,40,,,Number of reference features
npattern,i,h,5,3,10,Number of lines in patterns
nneighbors,i,h,10,2,,Number of nearest neighbors in patterns
nbins,i,h,6,1,,Maximum number of search bins
ndmax,i,h,20,1,,Maximum number of dispersions to evaluate
aidord,i,h,3,2,,Dispersion fitting order
maxnl,r,h,0.02,0.,,Maximum non-linearity
nfound,i,h,6,3,,Minimum number of lines in final solution
sigma,r,h,.05,0.,,Sigma of line centering (pixels)
minratio,r,h,0.1,0., 1.,Minimum spacing ratio to use
rms,r,h,0.3,0.,,RMS goal (fwidths)
fmatch,r,h,0.3,0.,1.,Matching goal (fraction unmatched)
debug,s,h,"",,,Print debugging information
Cheers, Dario
I changed the line in:
hydra$hydra.cl
task aidpars = "hydra$aidpars.par"
and I added the file aidpars.par in the hydra directory:
# Parameters for autoidentify task.
reflist,s,h,"",,,Reference coordinate list
refspec,s,h,"",,,Reference spectrum
#crval,s,h,"INDEF",,,Coordinate reference value
#cdelt,s,h,"INDEF",,,Coordinate interval per pixel
crpix,s,h,"INDEF",,,Coordinate reference pixel
crquad,s,h,"INDEF",,,Quadratic pixel distortion at reference pixel
cddir,s,h,"increasing","unknown|sign|increasing|decreasing",,Dispersion direction
crsearch,s,h,"INDEF",,,Coordinate value search radius
cdsearch,s,h,"INDEF",,,Coordinate interval search radius
ntarget,i,h,30,,,Number of target features
#nreference,i,h,40,,,Number of reference features
npattern,i,h,5,3,10,Number of lines in patterns
nneighbors,i,h,10,2,,Number of nearest neighbors in patterns
nbins,i,h,6,1,,Maximum number of search bins
ndmax,i,h,20,1,,Maximum number of dispersions to evaluate
aidord,i,h,3,2,,Dispersion fitting order
maxnl,r,h,0.02,0.,,Maximum non-linearity
nfound,i,h,6,3,,Minimum number of lines in final solution
sigma,r,h,.05,0.,,Sigma of line centering (pixels)
minratio,r,h,0.1,0., 1.,Minimum spacing ratio to use
rms,r,h,0.3,0.,,RMS goal (fwidths)
fmatch,r,h,0.3,0.,1.,Matching goal (fraction unmatched)
debug,s,h,"",,,Print debugging information
Cheers, Dario
Last post on May 14, 2006