More DECam image questions and issues
andrej wrote on Apr 25, 2015
Greetings.
I am having issues with imcombine on some DECam data. I am discovering that there is some slight blur/misalignment when using imcombine for multiple dither points of a single frame. I am using combine=median and offset=wcs.
*EXAMPLE WITH DECAM FRAME N3*
I have linked you to an example of one of the CCDs (frame N3). Each file was generated from an original stack of five subs, already median-combined and trimmed. Each file is one of the 10-point dithers ("D01" to "D10" appear in the file names correspondingly).
I have also included one of the imcombined FITS files in the folder:
https://drive.google.com/folderview?id=0B_eLF8j7l3I-fkFZdXFPMWdzSHJoZEJGQ1o2Y2ZtVU9RdDNJSWNac3JsMDJibm5jTTdhcVE&usp=sharing
I noticed that when we combine the 10 dither points using imcombine, the alignment is slightly off. See the combined fits file "MicroscopiumMedianFN3Combined.fits" in the drive link. I don’t understand why the images produced by imcombine are slightly offset when the 10-points of the image are merged together.
*EXAMPLE WITH DECAM FRAME N1*
Even more bizarre, is when we choose a different frame (say, frame N1) and combine the 10 dither points using this frame, imcombine doesn't work right, and the resulting image has echoing. See the combined fits file "MicroscopiumCombinedN1.fits" in the drive link.
I don't understand why imcombine is producing these kinds of blurry images even though we have set offset=wcs. The offsets for the stack of 10 theoretically ought to be the same since each point of the dither is of the same target for all CCDs simultaneously.
I mean, we could make a manual offset adjustment to the DECam data, but for each CCD and dither point that just seems like a lot of manual work.
What's going on?
Francisco Valdes wrote on Apr 27, 2015
Hello,
My first question is what data product type are you using? This would be the PROCTYPE keyword or, for more recent data, the name of the downloaded file would have something lik ooi or opi. The point here is that one type, called InstCal, has a calibrated world coordinate system but has not been resampled to a distortion free grid. The other, called Resampled, should allow good registration and combining with IMCOMBINE provided you use offsets=world.
Yours,
Frank Valdes
andrej wrote on Apr 29, 2015
They are all InstCal and the name of the file has "ooi"
So, should I just throw out all the InstCal files and start again with Resampled? Is there anything different about the image quality (or any sort of manual calibration that I need to do) that is in InstCal that isn't in Resampled?
Francisco Valdes wrote on Apr 29, 2015
Yes that is right, the simplest way to combine is to use the rsampled versions with imcombine using the parameter offsets="world". The resampled versions exactly align except for an integer offset. Note this only applies when the exposures have the same tangent point: the keywords CRVAL1 and CRVAL2. This will be the case for pipeline reduced data if the exposures largely overlap.
The only difference between InstCal and Resampled is the remapping of the pixels to a standard orientation and grid with distortions removed. In some cases the resampled versions will also have a local (within a CCD) sky structure subtracted but this onlly happens in certain types of datasets. The remapping is done with a high quality sinc interpolation. Some people do science on the InstCal to avoid any concerns about interpolation, but combining pixel data can only be done with a remapping step. So scientifically the two types are essentially the same.
Frank
andrej wrote on Apr 30, 2015
I appreciate the clarification and with that I plan to retrieve the Resampled data as appropriate.
andrej wrote on May 11, 2015
So I am trying to download the Resampled versions of the DECam image files, instead of the InstCal files. I have made absolutely sure that I am retrieving files that correspond to my observations and with the correct time stamps.
But I am encountering two weird issues:
[1] While the "InstCal" fits.fz files ("ooi"), which contain the full DECam field, are about 315 MB each, the corresponding "Resampled" fits.fz files ("opi") are only 4.7 MB each and contain only the frame [S7]. Why do the resampled data files not contain the full DECam field?
[2] Also, I am having trouble with using the Download Manager on a workstation. The NOAO Science Archive Download Manager, the one that uses Java, will download these 4.7 MB resampled image files fine, but it won't download the 315 MB files. The manager just hangs there forever when it gets to these files and will not download them. I don't understand why it is having these problems since I used to be able to use this same program to download the 315 MB files last year.
What's going on? How do I get the full "Resampled" DECam field images so that I can do imcombine?
Francisco Valdes wrote on May 12, 2015
Hi,
There was a bug in the pipeline for a period of time where the resampled files contained only one extension as you found. The immediate solution is to reprocess. Please tell me the calendar nights you are interested in. I'm sorry for the problems.
As for the second problem you should report it to sdmhelp@noao.edu or Mark Dickinson at med@noao.edu.
Yours,
Frank
andrej wrote on May 14, 2015
Hi, the nights are:
July 2, 2014 and July 3, 2014
Francisco Valdes wrote on May 18, 2015
Hi,
The data have been reprocessed. When you go to the NOAO portal you can put the string "v2" in the filename section of the query form to select only this latest processing.
Because the fields in your program contain very large sources the global or local sky subtraction version may not be to your liking, though photometry should be unaffected since nearby sky estimation is usually part of the process.
Yours,
Frank Valdes
andrej wrote on May 19, 2015
Alright. When I put
v2
into the filename section of the query form, it keeps telling me
"Result Set Empty"
I tried both with and without wildcards and it gave the same result. I tried both the Original Filename and the Archive Filename parameters, but that didn't make it work. I made sure to put in my program number and unchecked everything else. So, I'm not sure what's going on. Though I do see the new files with the "v2" in the file name when I query all of the files for my program, so I can weed out the necessary files manually if need be.
WeidaHu wrote on May 29, 2015
Hi,
I am combining some DECam data these days. I found a software that called SWARP may do these things easily. I also want to use iraf to combine them. I have some trouble in getting the shift in different images.Because I have a lot of images, I don't want to use imexamine to get the shift of each image like this website:http://www.astro.washington.edu/courses/astro480/co-add_unm.html. Could you please help me, if possible.
Mike Fitzpatrick wrote on May 29, 2015
If you're using the pipeline reduced data you can probably just use the offset="wcs" option to IMCOMBINE to use the WCS to derive the image offsets automatically, e.g.
cl> imcombine *.fits new.fits offset="wcs"
WeidaHu wrote on May 31, 2015
Thank you.
andrej wrote on Jun 01, 2015
That is what I am doing as well. However, I am trying to create a master mosaic of 600 FITS files (I took each of the original 60 CCD images, cleaned up the data one frame at a time, and combined it with 10 dither points)... the data are all in a 3 sq. degree region of the sky.
So I just want to use imcombine to my 600 FITS files, each 30 MB in size, so that I have a master mosaic, so that I can use Source Extractor to do a weak lensing analysis of the whole wide field.
But when I try to run my routine on my files, using
imcombine input=M* output=MCombined.fits offset=wcs combine=median
I receive the following error message:
"ERROR: Out of space in image header (IMCMB498)"
How do I get around this?
Mike Fitzpatrick wrote on Jun 01, 2015
The size of an output image header is controlled by the 'min_lenuserarea' environment variable, this is set as a system default in the hlib$zzsetenv.def file or optionally in your login.cl. The default value is 64000 chars but can be increased as needed before running the task. For example,
cl> reset min_lenuserarea = 256000
cl> flpr 0 # zero, no oh, to flush all processes
cl> imcombine *.fits .......
You can uncomment the setting and change the value in your login.cl as well.
andrej wrote on Jun 03, 2015
This worked. I again appreciate the help.
andrej wrote on Jul 16, 2015
So, I have an update.
I am able to create a master mosaic, consisting of 10 dither points in a line, and 5 subs within each dither.
With the Resampled data, there is still a bright section caused by frame S7. In each sub, half of the image (sections 1024x4096) is about 70 points brighter in value than the rest of the image.
I realize that I could do imarith and manually adjust the brightness but could you point me to a routine that can correct this?
Also, the master mosaic has sections of lighter and darker due to the dithers:
https://drive.google.com/file/d/0B_eLF8j7l3I-aV9JVXpnSU9YNEU/view?usp=sharing
I first tried imcombine with scale=none. Then when I tried imcombine with scale=mode, that made the image a lot more uniform... but it still had these narrow straight segments of lines that look like trim lines:
https://drive.google.com/file/d/0B_eLF8j7l3I-bGdvdkNManJFcWM/view?usp=sharing
[1] How can I get a more uniform mosaic?
[2] Has anyone published a general pipeline for how to go about generally stacking the DECam Resampled data and then preparing the analysis for Source Extractor?
WeidaHu wrote on Jul 18, 2015
Hi,
I found some sentence about combining DECam images in NOAO Data Handbook.
http://archive.noao.edu/doc/NOAO_DHB/NOAO_Data_Handbookv1.1.pdf
Image Stacking
The CP(Community Pipeline) combines exposures that overlap significantly (exposures are considered overlapping when they have the same tangent-point after remapping). Exposure patterns that map fields larger in radius than a DECam FoV will generally produce separate co-adds rather than one very large one. The overlapping exposures are separated into multiple stacks by exposure time: very short (t 300s). If a group has more than 50 exposures it is divided in the smallest number of subgroups which all have less than 50 exposures (not for scientific reasons, but rather to make the data handling manageable). Of the exposures that are identified for a stack, some are exclud- ed if they have the potential to degrade the image quality of the stack. The following metrics are exam- ined to detect and remove such outliers:
Relative flux scaling (e.g., low magnitude zero-points due to clouds)
Seeing
Sky brightness
Note that fewer than half of the full-FPA exposures are allowed to be rejected; otherwise all exposures are used.
When stacking, the pipeline scales the input images to a common magnitude zero-point and sky transpar- ency. The co-adds are the weighted sum of the contributing exposures, excluding pixels marked in the data quality mask. There are two stacks produced for each set of exposures that satisfy the criteria above. One has no background adjustments beyond those applied to the non-remapped data as described previ- ously. The other subtracts an additional background from each CCD. This is a higher spatial frequency filtering, which can produce a better matching of overlapping CCDs but can also subtract light from sources subtending 10% or more of a CCD. Because it is difficult for the pipeline to decide which is ap- propriate for a data set, the two versions are produced from which the investigator can choose.
But I can not find the Pipeline. Maybe it can help you. If you find the Pipeline, please tell me.
I am also combining DECam images this days.I have some question. The PROCTYPE is resampled. Do you have some data which PRODTYPE is wtmap or dqmask? How do you deal with those data?
WeidaHu wrote on Jul 18, 2015
And how do you deal with background? I used to subtract the average of the image.
Francisco Valdes wrote on Jul 24, 2015
This subject about restacking starting with the projected images (the opi ones) is something that is coming up with multiple people. Since we do say that this is what people should do we really need to help people do it. So I need to write a guide for how to stack the projected images from DECam. I now understand the term pipeline means "how to do the steps". I will find time to write a guide. Note that this will involve using IRAF imcombine. So those who want to use other software would need to interpret the steps appropriately.
Frank
Last post on Jul 24, 2015