existing file error in mscsrc$ccdproc.cl
smart wrote on Jan 26, 2009
I am reducing a number of mosaic images using the esowfi package which
seems to be an overlay of the mcs package. I reduce multiple nights one after the other and every so often during the ccdproc phase I get the following error:
WFI.2007-09-02T01_14_27.146.fits[im5]: Jan 26 13:47 Fix pixels in vip/im8.pl
WFI.2007-09-02T01_14_27.146.fits[im5]: Jan 26 13:47 Flat is Flat.fits[im5], scale 9528.781
ERROR: Operation would overwrite existing file (tmp$iraf5353a)
"> extlist)"
line 446: mscsrc$ccdproc.cl
called as: `ccdproc (images=W*.fits)'
esowfi>
this seems to be random in as much as if I then rerun the command file the problem does not always represent itself.
I include below the command file I have, but it is not very well commented (surprise surprise) the error occurs at line 93.
-----------------------------------------------------------------------------------
print "Reduction started"
time
#
# define some varibles
string junk
real m1,m2,sd,mx,mn
char fil
int *f,fb,fi,fs,fr,fv,ib,ii,is,ir,iv,b,iall,fall,ff,tf
#
# find night name
list=("wfitotopp.in")
junk=fscan(list,s1)
print (s1)
#
# We wish to clean up the area to start from just the .fts files
# at the moment verify is off for debugging, in normal operation
# we may want to turn that on (verify=yes)
delete.verify=no
#
print "Warning: This next com mand deletes all these files:"
dir ("*.ftp,logfile,*.lis,tmp*,bup/*"//s1//"*.*,*.err,*.war")
dir (s1//"/*"//s1//"*,*"//s1//"*.*,todo/*"//s1//"*,bup/*"//s1//"*")
dir ("t*.sta,"//s1//"t*.sta")
dir (s1//"/*"//substr(s1,3,8)//"*")
dir ("*"//substr(s1,3,8)//"*.*")
dir ("todo/*"//substr(s1,3,8)//"*")
dir ("bup/*"//substr(s1,3,8)//"*")
dir (s1//"/Flat.fits")
dir (s1//"/logfile")
###
del ("f.scale*,*.ftp,logfile,*.lis,tmp*,bup/*"//s1//"*.*,*.err,*.war")
del (s1//"/*"//s1//"*,*"//s1//"*.*,todo/*"//s1//"*,bup/*"//s1//"*")
del ("t*.sta,"//s1//"t*.sta")
del (s1//"/*"//substr(s1,3,8)//"*")
del ("*"//substr(s1,3,8)//"*.*")
del ("todo/*"//substr(s1,3,8)//"*")
del ("bup/*"//substr(s1,3,8)//"*,lctsta")
del (s1//"/Flat.fits")
del (s1//"/Frin*.fits")
del (s1//"/logfile")
###
movef *.fits todelete
movef *.imh todelete
#
# copy files=(s1//"/*.*") newdir=.
copy input=(s1//"/*.fits*") output=.
copy input=(s1//"/*.lis") output=.
!gunzip *.gz
#
esowfi
#
# initialize all params
unlearn flatcombine
unlearn ccdproc
unlearn objmasks
unlearn mscskysub
unlearn rmfringe
#
# restart task, not sure if some defaults are set here
bye
esowfi
esosetinst
esohdr.querytype=no
esohdr *.fits
# set defaults:
flatcombine.ccdtype=""
ccdproc.ccdtype = ""
ccdproc.flatcor=no
ccdproc.fixfile="vip/im*.pl"
ccdproc.fixpix = yes
#ccdproc.fixpix = no
#ccdproc.fixfile=""
ccdproc.zerocor=no
objmasks.omtype="boolean"
objmasks.masks ="vip"
#objmasks.masks =""
objmasks1.fitstep = 10
mscskysub.mask = "!objmask"
mscskysub.type_output = "fit"
mscskysub.lower = 3
mscskysub.upper = 3
mscskysub.niter = 2
combine.combine="median"
combine.masktype="goodvalue"
flatcombine @flat.lis
imdel @flat.lis
##### PROBLEM OCCURS HERE
ccdproc.flatcor=yes
ccdproc W*.fits
# first fringe iteration
objmasks.hsigma = 9
objmask @fringein.lis "m//@fringein.lis"
msccmd "mscskysub $input $output 2 2" @fringein.lis sk@fringein.lis
mscarith @fringein.lis - sk@fringein.lis s//@fringein.lis
msccmd "hedit $input BPM '(objmask)' add+" s@fringein.lis
combine.scale="exposure"
combine s@fringein.lis outp=Fringe1
msccmd "hedit $input BPM vip add+" Fringe1
rmfringe.back="sk@fringein.lis"
rmfringe @fringein.lis f//@fringein.lis Fringe1.fits m//@fringein.lis
# make some space
imdel s@fringein.lis
imdel m//@fringein.lis
imdel sk@fringein.lis
# second iteration
objmasks.hsigma = 5
objmask f//@fringein.lis m2//@fringein.lis
msccmd "mscskysub $input $output 4 4" f@fringein.lis sk2@fringein.lis
# mscarith @fringein.lis - sk2@fringein.lis s2//@fringein.lis
# msccmd "hedit $input BPM '(objmask)' add+" s2@fringein.lis
# msccmd "hselect $input rmfringe yes" f//@fringein1.lis > f.scale1
# !awk '{print 1./$3}' f.scale1 > f.scale2
# combine.scale="@f.scale2"
# combine s2@fringein.lis outp=Fringe2
# msccmd "hedit $input BPM vip add+" Fringe2
# # mscdisplay Fringe2.fits 3
# rmfringe.back="sk2@fringein.lis"
# rmfringe @fringein.lis I@fringeot.lis Fringe2.fits m2//@fringein.lis
# control the second iteration make some space
imdel s2@fringein.lis
mscarith f@fringein.lis - sk2@fringein.lis s3//@fringein.lis
msccmd "hedit $input BPM '(objmask)' add+" s3@fringein.lis
msccmd "hselect $input rmfringe yes" f//@fringein1.lis > f.scale3
!awk '{print 1./$3}' f.scale3 > f.scale4
combine.scale="@f.scale4"
combine s3@fringein.lis outp=Fringe3
msccmd "hedit $input BPM vip add+" Fringe3
# mscdisplay Fringe3.fits 3
rmfringe.back="sk2@fringein.lis"
rmfringe f@fringein.lis @fringeot.lis Fringe3.fits m2//@fringein.lis
imdel *WFI*.fits
mscsplit @fringeot.lis
imdel @ot.lis
wfits @in.lis @ot.lis bs=1.0 bz=32768.0 ma+ lo- sh+ bi=16 bl=0. sc+ au- fextn=""
movef files = "F*.fits,logfile" newdir=(s1)
imdel *.fits
seems to be an overlay of the mcs package. I reduce multiple nights one after the other and every so often during the ccdproc phase I get the following error:
WFI.2007-09-02T01_14_27.146.fits[im5]: Jan 26 13:47 Fix pixels in vip/im8.pl
WFI.2007-09-02T01_14_27.146.fits[im5]: Jan 26 13:47 Flat is Flat.fits[im5], scale 9528.781
ERROR: Operation would overwrite existing file (tmp$iraf5353a)
"> extlist)"
line 446: mscsrc$ccdproc.cl
called as: `ccdproc (images=W*.fits)'
esowfi>
this seems to be random in as much as if I then rerun the command file the problem does not always represent itself.
I include below the command file I have, but it is not very well commented (surprise surprise) the error occurs at line 93.
-----------------------------------------------------------------------------------
print "Reduction started"
time
#
# define some varibles
string junk
real m1,m2,sd,mx,mn
char fil
int *f,fb,fi,fs,fr,fv,ib,ii,is,ir,iv,b,iall,fall,ff,tf
#
# find night name
list=("wfitotopp.in")
junk=fscan(list,s1)
print (s1)
#
# We wish to clean up the area to start from just the .fts files
# at the moment verify is off for debugging, in normal operation
# we may want to turn that on (verify=yes)
delete.verify=no
#
print "Warning: This next com mand deletes all these files:"
dir ("*.ftp,logfile,*.lis,tmp*,bup/*"//s1//"*.*,*.err,*.war")
dir (s1//"/*"//s1//"*,*"//s1//"*.*,todo/*"//s1//"*,bup/*"//s1//"*")
dir ("t*.sta,"//s1//"t*.sta")
dir (s1//"/*"//substr(s1,3,8)//"*")
dir ("*"//substr(s1,3,8)//"*.*")
dir ("todo/*"//substr(s1,3,8)//"*")
dir ("bup/*"//substr(s1,3,8)//"*")
dir (s1//"/Flat.fits")
dir (s1//"/logfile")
###
del ("f.scale*,*.ftp,logfile,*.lis,tmp*,bup/*"//s1//"*.*,*.err,*.war")
del (s1//"/*"//s1//"*,*"//s1//"*.*,todo/*"//s1//"*,bup/*"//s1//"*")
del ("t*.sta,"//s1//"t*.sta")
del (s1//"/*"//substr(s1,3,8)//"*")
del ("*"//substr(s1,3,8)//"*.*")
del ("todo/*"//substr(s1,3,8)//"*")
del ("bup/*"//substr(s1,3,8)//"*,lctsta")
del (s1//"/Flat.fits")
del (s1//"/Frin*.fits")
del (s1//"/logfile")
###
movef *.fits todelete
movef *.imh todelete
#
# copy files=(s1//"/*.*") newdir=.
copy input=(s1//"/*.fits*") output=.
copy input=(s1//"/*.lis") output=.
!gunzip *.gz
#
esowfi
#
# initialize all params
unlearn flatcombine
unlearn ccdproc
unlearn objmasks
unlearn mscskysub
unlearn rmfringe
#
# restart task, not sure if some defaults are set here
bye
esowfi
esosetinst
esohdr.querytype=no
esohdr *.fits
# set defaults:
flatcombine.ccdtype=""
ccdproc.ccdtype = ""
ccdproc.flatcor=no
ccdproc.fixfile="vip/im*.pl"
ccdproc.fixpix = yes
#ccdproc.fixpix = no
#ccdproc.fixfile=""
ccdproc.zerocor=no
objmasks.omtype="boolean"
objmasks.masks ="vip"
#objmasks.masks =""
objmasks1.fitstep = 10
mscskysub.mask = "!objmask"
mscskysub.type_output = "fit"
mscskysub.lower = 3
mscskysub.upper = 3
mscskysub.niter = 2
combine.combine="median"
combine.masktype="goodvalue"
flatcombine @flat.lis
imdel @flat.lis
##### PROBLEM OCCURS HERE
ccdproc.flatcor=yes
ccdproc W*.fits
# first fringe iteration
objmasks.hsigma = 9
objmask @fringein.lis "m//@fringein.lis"
msccmd "mscskysub $input $output 2 2" @fringein.lis sk@fringein.lis
mscarith @fringein.lis - sk@fringein.lis s//@fringein.lis
msccmd "hedit $input BPM '(objmask)' add+" s@fringein.lis
combine.scale="exposure"
combine s@fringein.lis outp=Fringe1
msccmd "hedit $input BPM vip add+" Fringe1
rmfringe.back="sk@fringein.lis"
rmfringe @fringein.lis f//@fringein.lis Fringe1.fits m//@fringein.lis
# make some space
imdel s@fringein.lis
imdel m//@fringein.lis
imdel sk@fringein.lis
# second iteration
objmasks.hsigma = 5
objmask f//@fringein.lis m2//@fringein.lis
msccmd "mscskysub $input $output 4 4" f@fringein.lis sk2@fringein.lis
# mscarith @fringein.lis - sk2@fringein.lis s2//@fringein.lis
# msccmd "hedit $input BPM '(objmask)' add+" s2@fringein.lis
# msccmd "hselect $input rmfringe yes" f//@fringein1.lis > f.scale1
# !awk '{print 1./$3}' f.scale1 > f.scale2
# combine.scale="@f.scale2"
# combine s2@fringein.lis outp=Fringe2
# msccmd "hedit $input BPM vip add+" Fringe2
# # mscdisplay Fringe2.fits 3
# rmfringe.back="sk2@fringein.lis"
# rmfringe @fringein.lis I@fringeot.lis Fringe2.fits m2//@fringein.lis
# control the second iteration make some space
imdel s2@fringein.lis
mscarith f@fringein.lis - sk2@fringein.lis s3//@fringein.lis
msccmd "hedit $input BPM '(objmask)' add+" s3@fringein.lis
msccmd "hselect $input rmfringe yes" f//@fringein1.lis > f.scale3
!awk '{print 1./$3}' f.scale3 > f.scale4
combine.scale="@f.scale4"
combine s3@fringein.lis outp=Fringe3
msccmd "hedit $input BPM vip add+" Fringe3
# mscdisplay Fringe3.fits 3
rmfringe.back="sk2@fringein.lis"
rmfringe f@fringein.lis @fringeot.lis Fringe3.fits m2//@fringein.lis
imdel *WFI*.fits
mscsplit @fringeot.lis
imdel @ot.lis
wfits @in.lis @ot.lis bs=1.0 bz=32768.0 ma+ lo- sh+ bi=16 bl=0. sc+ au- fextn=""
movef files = "F*.fits,logfile" newdir=(s1)
imdel *.fits
smart wrote on Jan 26, 2009
An update on the problem I am having. If I carry out the .cl listed cutting and pasting each line line by line things work, it is almost like IRAF cannot keep up with the tasks if there commands come too fast. Don't suppose it could be because I am working on a raid and the delete time is sometimes long., e.g. a second or so.... Anyway since it happens only sporadically I can do this cut and paste trick when it does so if no other ideas come up thats what I'll do.
Francisco Valdes wrote on Jan 26, 2009
From the error it seems that it has to do with temporary files used by the task. The algorithm that generates these files, which is based on a PID and checking if a file already exists and incrementing a suffix, could, in principle, fail. This must be very rare since we don't get questions about this. But speed and file system issues could make this fail. It also depends on how the application (usually scripts as with mscred.ccdproc) use the temp file; i.e. do they delete and reuse it so that the delete might not actually complete when the next step tries to write. So my guess is that your system does fail in this way occasionally. There is really no solution without modifying the system mktemp command. I hope you can work around this and tolerate an occasional failure.
We're sorry for this difficulty.
Yours,
Frank Valdes
We're sorry for this difficulty.
Yours,
Frank Valdes
smart wrote on Jan 26, 2009
Frank,
This maybe a problem with my original installation as I found that in the /tmp directory there are many irafXXXX files (es. iraf4270ya) and they are very old. Prehaps there should be some clean up happening eveyr so often that does not happen. Anyway within my script I put the lines
ccdproc W*.fits
imdel tmp*.fits
!rm /tmp/iraf*
ccdproc W*.fits
e.g. when things fail I rm all tmp images and iraf files from the /tmp directory
and it seems to work. The images already processed are skipped as they should be an the ccdproc picks where things left off.
Is there any danger to delteing the /tmp/iraf* files?
-Ricky
This maybe a problem with my original installation as I found that in the /tmp directory there are many irafXXXX files (es. iraf4270ya) and they are very old. Prehaps there should be some clean up happening eveyr so often that does not happen. Anyway within my script I put the lines
ccdproc W*.fits
imdel tmp*.fits
!rm /tmp/iraf*
ccdproc W*.fits
e.g. when things fail I rm all tmp images and iraf files from the /tmp directory
and it seems to work. The images already processed are skipped as they should be an the ccdproc picks where things left off.
Is there any danger to delteing the /tmp/iraf* files?
-Ricky
Francisco Valdes wrote on Jan 26, 2009
Ricky,
Anything in tmp can be deleted without problem. tmp is used because typically this area is cleaned up when the machine is rebooted (which usually happens on times scales of weeks or months rather than years). Programs that create these temporary files are supposed to clean up after they successfully complete but if they crash or abort unexpectedly the files may be left behind. Running the task again should use a different tmp file name (and in fact they usually check that there is no pre-existing file of the same name). I still can't fully explain your error, though some OS timing thing is still the likely culprit, but what you did is perfectly fine and if things are working then case closed.
Frank
Anything in tmp can be deleted without problem. tmp is used because typically this area is cleaned up when the machine is rebooted (which usually happens on times scales of weeks or months rather than years). Programs that create these temporary files are supposed to clean up after they successfully complete but if they crash or abort unexpectedly the files may be left behind. Running the task again should use a different tmp file name (and in fact they usually check that there is no pre-existing file of the same name). I still can't fully explain your error, though some OS timing thing is still the likely culprit, but what you did is perfectly fine and if things are working then case closed.
Frank
Last post on Jan 26, 2009