imcombine has stopped working
grayro wrote on Jan 29, 2019
Hi,
I have encountered a mysterious problem in a cl hosting script which I have been running as part of a pipeline for about 2 years. Suddenly, a few weeks ago, imcombine stopped working and is producing an output file of 0 length. The offending line is:
imcombine(input="@tlist.lis",output=s1,combine="average",reject="none")
tlist.lis is non-empty, and contains the names of 5 valid fits images which, for instance, can be displayed in ds9 without any problems.
I have combined those images at the ecl > prompt manually with:
imcombine @tlist.lis try.fits combine="average" reject="none"
and they were combined with no problems and the result output to try.fits. try.fits is non-zero and looks exactly as I expected.
My IRAF version is 2.16.1. I am running under Linux Mint, and as I mentioned above, the pipeline has been running flawlessly for about 2 years. I can post the entire script if that would help.
Thanks, Richard
grayro wrote on Jan 30, 2019
The mystery deepens. When I use the line
imcombine(input="@tlist.lis",output="try.fits",combine="average",reject="none")
in the script instead of
imcombine(input="@tlist.lis",output=s1,combine="average",reject="none")
The file try.fits is created and contains the expected combined image. However, when I try to copy try.fits to s1 (which is a variable name containing the name of the desired output image),
imcopy(input="try.fits",output=s1)
the file is created with the correct name, but it is empty. Even if I try a subterfuge with something like
printf("cp try.fits %s.fits\n",s1, > "copyit.sh")
!chmod a+x copyit.sh
!./copyit.sh
The intended file is empty. However, if I execute the script copyit.sh manually, it copies without any problems!
The file system has plenty of room, and like I said in my previous message, this script has been working without fault for about 2 years and then suddenly failed.
Hope someone can give me some clues about how to proceed here. I am knocking my head against the wall!
Richard
Mike Fitzpatrick wrote on Jan 31, 2019
If the 's1' variable contains a path then I'd suspect a permissions problem, but since the problem seems to be the use of s1 itself.... What exactly is the filename you're trying to create? How is the 's1' variable derived? Is this a local script declaration or the CL parameter? Does doing "unlearn cl" to reset the parameters help? Does working in a different directory make a difference? Do you mean by "cl hosting script" a standalone host CL script, and if so do you set a uparm directory that may have recently changed?
Lastly, any chance the /tmp directory on your machine is full?
grayro wrote on Feb 12, 2019
Hi,
Thank you for answering my post. I was out of town for a few days so am only getting back now to this problem.
To answer your questions:
s1 is the cl parameter, so it is not declared in the script (the script is pasted in below).
The filename that I am trying to create will be something like hd131445_B.fits, so no path is involved, and the file is created in the local working directory. The curious thing is that the file is created, but it is empty.
I have tried working in a different directory but it makes no difference. I have full rw permissions on the local directory.
The script is a standalone host CL script, so it is run from the command line, not within the iraf environment. Yes, I do set an uparm directory, but I have not physically changed anything in that directory, although various scripts which set parameters do cause changes in that directory.
I did check the \tmp directory. It had lots of timeshift images in it, but still plenty of space. But to make certain, I deleted a number of those images and re-ran the script with the same sorry result.
I am not certain how to run "unlearn cl". Can I include that as a line in the script after the definition of the uparm directory?
The script is pasted below with the offending line indicated with a ===>>> (toward the end)
best wishes, Richard
#!/home/grayro/iraf/bin.linux64/ecl.e -f
set home = "/home/grayro/.iraf/"
set imdir = "/home/grayro/.iraf/imdir/grayro/"
set cache = "/home/grayro/.iraf/cache/grayro/"
set uparm = "home$uparm/"
set userid = "grayro"
set arch = ".linux64"
set clobber = yes
set imclobber = yes
images
plot
dataio
lists
tv
utilities
noao
imred
digiphot
apphot
ccdred
int expos=0
int crop=0
int naxis=0
string list2
string filter
string *plist
real fwhm=3.0
files orig/Bias*.fit > Bias.lis
files orig/Dark2s*.fit > Dark2s.lis
files orig/Dark5s*.fit > Dark5s.lis
files orig/Dark10s*.fit > Dark10s.lis
files orig/Dark15s*.fit > Dark15s.lis
files orig/Dark20s*.fit > Dark20s.lis
files orig/Dark30s*.fit > Dark30s.lis
files orig/Dark45s*.fit > Dark45s.lis
files orig/Dark60s*.fit > Dark60s.lis
files orig/Dark90s*.fit > Dark90s.lis
printf("radgauss\n", > "ukey.dat")
printf("3.0\n", >> "ukey.dat")
printf("14\n", >> "ukey.dat")
printf("INDEF\n", >> "ukey.dat")
printf("INDEF\n", >> "ukey.dat")
rfits @Bias.lis * bias
imcombine bias* BIAS combine=average reject=avsigclip
imdelete bias*
rfits @Dark2s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK2s
imdelete TMP
imdelete dark*
rfits @Dark5s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK5s
imdelete TMP
imdelete dark*
rfits @Dark10s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK10s
imdelete TMP
imdelete dark*
rfits @Dark15s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK15s
imdelete TMP
imdelete dark*
rfits @Dark20s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK20s
imdelete TMP
imdelete dark*
rfits @Dark30s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK30s
imdelete TMP
imdelete dark*
rfits @Dark45s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK45s
imdelete TMP
imdelete dark*
rfits @Dark60s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK60s
imdelete TMP
imdelete dark*
rfits @Dark90s.lis * dark
imcombine dark* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS DARK90s
imdelete TMP
imdelete dark*
files orig/FlatSv*.fit > FlatSv.lis
rfits @FlatSv.lis * flatSv
imcombine flatSv* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS TMP1
imarith TMP1 "-" DARK45s FTMP
imstat("FTMP",fields="midpt",format-) | scan(y)
imarith("FTMP","/",y,"FLATSv")
imdelete TMP
imdelete FTMP
imdelete TMP1
imdelete flatSv*
files orig/FlatHa*.fit > FlatHa.lis
rfits @FlatHa.lis * flatHa
imcombine flatHa* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS TMP1
imarith TMP1 "-" DARK45s FTMP
imstat("FTMP",fields="midpt",format-) | scan(y)
imarith("FTMP","/",y,"FLATHa")
imdelete TMP
imdelete FTMP
imdelete TMP1
imdelete flatHa*
files orig/FlatB*.fit > FlatB.lis
rfits @FlatB.lis * flatB
imcombine flatB* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS TMP1
imarith TMP1 "-" DARK5s FTMP
imstat("FTMP",fields="midpt",format-) | scan(y)
imarith("FTMP","/",y,"FLATB")
imdelete TMP
imdelete FTMP
imdelete TMP1
imdelete flatB*
files orig/FlatR*.fit > FlatR.lis
rfits @FlatR.lis * flatR
imcombine flatR* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS TMP1
imarith TMP1 "-" DARK2s FTMP
imstat("FTMP",fields="midpt",format-) | scan(y)
imarith("FTMP","/",y,"FLATR")
imdelete TMP
imdelete FTMP
imdelete TMP1
imdelete flatR*
files orig/FlatV*.fit > FlatV.lis
rfits @FlatV.lis * flatV
imcombine flatV* TMP combine=average reject=avsigclip
imarith TMP "-" BIAS TMP1
imarith TMP1 "-" DARK2s FTMP
imstat("FTMP",fields="midpt",format-) | scan(y)
imarith("FTMP","/",y,"FLATV")
imdelete TMP
imdelete FTMP
imdelete TMP1
imdelete flatV*
!rm *.lis
imcopy BIAS[1024:3071,1024:3071] BIAScrop
imcopy DARK2s[1024:3071,1024:3071] DARK2scrop
imcopy DARK5s[1024:3071,1024:3071] DARK5scrop
imcopy DARK10s[1024:3071,1024:3071] DARK10scrop
imcopy DARK15s[1024:3071,1024:3071] DARK15scrop
imcopy DARK20s[1024:3071,1024:3071] DARK20scrop
imcopy DARK30s[1024:3071,1024:3071] DARK30scrop
imcopy DARK45s[1024:3071,1024:3071] DARK45scrop
imcopy DARK60s[1024:3071,1024:3071] DARK60scrop
imcopy DARK90s[1024:3071,1024:3071] DARK90scrop
imcopy FLATB[1024:3071,1024:3071] FLATBcrop
imcopy FLATHa[1024:3071,1024:3071] FLATHacrop
imcopy FLATR[1024:3071,1024:3071] FLATRcrop
imcopy FLATSv[1024:3071,1024:3071] FLATSvcrop
imcopy FLATV[1024:3071,1024:3071] FLATVcrop
!getstarobs
list = "observations.txt"
while(fscan(list,s1,s2) != EOF) {
imcopy(s1,s2)
imgets(image=s2,param="i_naxis1")
naxis = int(imgets.value)
if(naxis == 2048) crop=1
if(naxis == 4096) crop=0
if(crop == 0) {
imarith(s2, "-", "BIAS", s2)
} else {
imarith(s2, "-", "BIAScrop", s2)
}
imgets(image=s2, param="EXPTIME")
expos = int(imgets.value)
if(crop == 0) {
if(expos == 2) imarith(s2, "-", "DARK2s", s2)
else if(expos == 5) imarith(s2, "-", "DARK5s", s2)
else if(expos == 10) imarith(s2, "-", "DARK10s", s2)
else if(expos == 15) imarith(s2, "-", "DARK15s", s2)
else if(expos == 20) imarith(s2, "-", "DARK20s", s2)
else if(expos == 30) imarith(s2, "-", "DARK30s", s2)
else if(expos == 45) imarith(s2, "-", "DARK45s", s2)
else if(expos == 90) imarith(s2, "-", "DARK60s", s2)
else imarith(s2, "-", "DARK90s", s2)
imgets(image=s2,param="FILTER")
if(imgets.value == "Ha") imarith(s2, "/", "FLATHa", s2)
else if(imgets.value == "Sv") imarith(s2, "/", "FLATSv", s2)
else if(imgets.value == "B") imarith(s2, "/", "FLATB", s2)
else if(imgets.value == "R") imarith(s2, "/", "FLATR", s2)
else if(imgets.value == "V") imarith(s2, "/", "FLATV", s2)
else printf("We should not be here!\n")
} else {
if(expos == 2) imarith(s2, "-", "DARK2scrop", s2)
else if(expos == 5) imarith(s2, "-", "DARK5scrop", s2)
else if(expos == 10) imarith(s2, "-", "DARK10scrop", s2)
else if(expos == 15) imarith(s2, "-", "DARK15scrop", s2)
else if(expos == 20) imarith(s2, "-", "DARK20scrop", s2)
else if(expos == 30) imarith(s2, "-", "DARK30scrop", s2)
else if(expos == 45) imarith(s2, "-", "DARK45scrop", s2)
else if(expos == 90) imarith(s2, "-", "DARK60scrop", s2)
else imarith(s2, "-", "DARK90scrop", s2)
imgets(image=s2,param="FILTER")
if(imgets.value == "Ha") imarith(s2, "/", "FLATHacrop", s2)
else if(imgets.value == "Sv") imarith(s2, "/", "FLATSvcrop", s2)
else if(imgets.value == "B") imarith(s2, "/", "FLATBcrop", s2)
else if(imgets.value == "R") imarith(s2, "/", "FLATRcrop", s2)
else if(imgets.value == "V") imarith(s2, "/", "FLATVcrop", s2)
else printf("We should not be here!\n")
}
}
!combineimage
list = ""
list = "combined.txt"
while(fscan(list,s1,k) != EOF) {
j = fscan(list,s2)
imgets(image=s2,param="i_naxis1")
naxis = int(imgets.value)
imgets(image=s2,param="FILTER")
filter = imgets.value
delete combine.lis
delete tlist.lis
delete shifts.db
printf("%s\n",s2, >> "combine.lis")
printf("t%s\n",s2, >> "tlist.lis")
for(i=2;i<=k;i+=1) {
j = fscan(list,s3)
printf("%s\n",s3, >> "combine.lis")
printf("t%s\n",s3, >> "tlist.lis")
}
if(naxis == 4096) {
xregister(input="@combine.lis",ref=s2,region="[1536:2560,1536:2560]",shifts="shifts.db",out="@tlist.lis",xcbox=8,ycbox=8,xwindow=20,ywindow=20)
} else {
xregister(input="@combine.lis",ref=s2,region="[512:1536,512:1536]",shifts="shifts.db",out="@tlist.lis",xcbox=8,ycbox=8,xwindow=20,ywindow=20)
}
===>>> imcombine(input="@tlist.lis",output=s1,combine="average",reject="none")
imdelete @combine.lis
imdelete @tlist.lis
printf("rwfsolve.sh %s.fits\n",s1, > "try.sh")
!chmod a+x try.sh
!./try.sh
if(access(s1//".new")) {
print("Plate solved!")
if(filter == "B") {
printf("cp %s.wcs B.wcs\n",s1, > "cp.sh")
!chmod a+x cp.sh
!./cp.sh
}
printf("rm %s.fits\n",s1, > "try1.sh")
printf("mv %s.new %s.fits\n",s1,s1, >> "try1.sh")
!chmod a+x try1.sh
!./try1.sh
} else {
printf("new-wcs -d -i %s.fits -w B.wcs -o tmp.fits\n",s1, > "Bwcs.sh")
printf("cp tmp.fits %s.fits\n",s1, >> "Bwcs.sh")
!chmod a+x Bwcs.sh
!./Bwcs.sh
}
!rm *.solved
delete(files=s1//".wcs")
!rm *.axy
!rm *.corr
!rm *.xyls
!rm *.match
!rm *.rdls
!rm try.sh
!rm temp.lst
!rm temp.psf
daofind(image=s1,output="temp.lst",threshold=100,sigma=1.2,readnoise=14,interactive=no,verify=no,fwhmpsf=3.0)
fitpsf(image=s1,box=11,coords="temp.lst",output="temp.psf",interact=no,<"ukey.dat"
!psfrd
plist = "psf.out"
j = fscan(plist,fwhm)
datapars.epadu = 1.5
datapars.readnoise = 14
photpars.apertures = 3.5*fwhm
fitskypars.annulus = 3.5*fwhm + 6
daofind(image=s1,output=s1//".coo",threshold=100,sigma=1.2,readnoise=14,interactive=no,verify=no,fwhmpsf=fwhm)
wcsctran(input=s1//".coo",output=s1//".wcs",image=s1,inwcs="logical",outwcs="world"
phot(image=s1,coords=s1//".coo",output=s1//".mag",interac=no,verify=no)
}
logout
Mike Fitzpatrick wrote on Feb 19, 2019
My first suspicion is that the contents of the 'combined.txt' file might not be correct for the script, have you checked this? Otherwise, I would add the command "d_on" before the problematic IMCOMBINE call to get more information (you would use 'd_off' afterwards to disable it). Does the 's1' value contain a ".fits" extension explicitly? I'm also a bit confused where earlier in the script you call imcombine with 'TMP' as the output name, but I don't see where that gets set in the script. Does calling imcombine using the command-mode (i.e. no parens) versus the script-mode you used earlier make any difference?
grayro wrote on Feb 26, 2019
Sorry for the long delay; I can get back to this problem only occasionally:
To answer your questions:
1) Yes, I have inspected the combine.txt file, and the contents are correct.
2) I added the command d_on before the imcombine statement, and this is what I got:
received `bye' from `d_on'
paramfind() looking down pfile `cl'/7DE3D0 for `$errno'
paramfind() looking down pfile `cl'/7DE3D0 for `$errmsg'
paramfind() looking down pfile `cl'/7DE3D0 for `$errtask'
flushing io for task `d_on'
restoring task `cl', tp: 18491248
topd 28094/28094
run: pc=879, op=
callnewtask: name=imcombine, currentask=11A2770
pfileload, task imcombine
pfileread, task imcombine, pfile uparm$imhimcome.par
copying pfile for `imcombine'
run: pc=884, op=
run: pc=891, op=
paramfind() looking down pfile `imcombine'/81DA80 for `input'
run: pc=895, op=
paramfind() looking down pfile `ccdred'/80FEB0 for `s1'
paramfind() looking down pfile `cl'/7DE3D0 for `s1'
run: pc=899, op=
paramfind() looking down pfile `imcombine'/81DA80 for `output'
run: pc=903, op=
run: pc=909, op=
paramfind() looking down pfile `imcombine'/81DA80 for `combine'
run: pc=913, op=
run: pc=919, op=
paramfind() looking down pfile `imcombine'/81DA80 for `reject'
run: pc=923, op=
execnewtask: pc = 926 (line 247)
paramfind() looking down pfile `imcombine'/81DA80 for `$nargs'
paramfind() looking down pfile `imcombine'/81DA80 for `mode'
CALL imcombine\
input="@tlist.lis"\
output="hd76218_B"\
headers=""\
bpmasks=""\
rejmasks=""\
nrejmasks=""\
expmasks=""\
sigmas=""\
imcmb="$I"\
logfile="STDOUT"\
combine="average"\
reject="none"\
project=no\
outtype="real"\
outlimits=""\
offsets="none"\
masktype="none"\
maskvalue="0"\
blank=0.\
scale="none"\
zero="none"\
weight="none"\
statsec=""\
expname=""\
lthreshold=INDEF\
hthreshold=INDEF\
nlow=1\
nhigh=1\
nkeep=1\
mclip=yes\
lsigma=3.\
hsigma=3.\
rdnoise="0."\
gain="1."\
snoise="0."\
sigscale=0.1\
pclip=-0.5\
grow=0.\
mode="ql"\
$nargs=0
connect: *in, *out, t_in, t_out: 18676968 18676976 18676872 18676824
Returning from execnewtask.yyin, ct_in, nt_in:18676968 18676968 18676968
startblock (1)
startblock: ifseen=0
Feb 20 16:25: IMCOMBINE
combine = average, scale = none, zero = none, weight = none
blank = 0.
Images
thd76218_B-1
thd76218_B-2
thd76218_B-3
thd76218_B-4
thd76218_B-5
Output image = hd76218_B, ncombine = 5
bye
parse init (block)...
run: pc=1857, op=
callnewtask: name=bye, currentask=11A2698
run: pc=1861, op=
execnewtask: pc = 1864 (line 537)
execnewtask: calling new task@@11A25C0
received `bye' from `imcombine'
paramfind() looking down pfile `cl'/7DE3D0 for `$errno'
paramfind() looking down pfile `cl'/7DE3D0 for `$errmsg'
paramfind() looking down pfile `cl'/7DE3D0 for `$errtask'
flushing io for task `imcombine'
pfcopyback uparm$imhimcome.par
restoring task `cl', tp: 18491248
topd 36572/28094
run: pc=926, op=
callnewtask: name=d_off, currentask=11A2770
run: pc=930, op=
execnewtask: pc = 933 (line 248)
execnewtask: calling new task@@11A2698
Reading input file 1 of 1: "hd76218_B.fits"...
ERROR: Image type not recognized: Could not determine file type (does the file exist?): hd76218_B.fits
So, at the end, a problem is encountered with the image hd76218_B.fits. It does exist, but it is empty. This is not due to defective data; this problem is encountered with all data sets, and the input images are fine.
3) the s1 value does not explicitly contain a ".fits" extension.
4) When I try to execute imcombine in the script using 'TMP' as the output file (that is not shown in the script that I sent you) in the command mode it works; TMP not only exists but it contains the correct combined image. But when I use imcombine in the parentheses mode, it fails to work; the output file is produced but it is empty. As a matter of fact, you will see that this is the only time in the script where I use imcombine in the parentheses mode. But I do not know how to introduce a variable name in the command mode.
Richard
Last post on Feb 26, 2019