Max number of images to combine
Stefano Berta wrote on Feb 26, 2009
Hi,
I have a question about imcombine (or mscstack).
I was testing whether it will be possible to use IRAF to produce mosaics out of huge numbers of images, but I could not figure out what is the actual limit.
I'm running the NOAO PC-IRAF Revision 2.12.2a-EXPORT Wed Jul 14 20:45:34 MST 2004, installed in the SCISOFT 6.1 package.
The general (images.immatch) imcombine HELP says that the limit is approx 4000 images (but somewhere else I read 4020, just can't remind where).
I'm using very small, 16x16 pix images, but I might have around 5e4 images (or more in the future) to pile up. I used the MSCSTACK incarnation on the combine. Apparently it loads all images, but then crashes when computing the stack at the ~7700th image. I'm not using any BPM mask or additional imputs/outputs rather than the science frames and the stacked image.
It don't think it's a memory issue, because it's using roughly 30% only of my RAM.
So my questions are:
- is the max number of images related also to their size? e.g. 4000 is a standard number for a 512x512 image (?)
- what's the main cause of this limit? coding issues, memory/records managment, or something else?
- If I use MEFs, will I be limited to 4000 extensions or to 4000 MEFs? In the latter case I could use some MSCJOIN tricks.
- is there a way to increase this number by playing with login.cl parameters or during IRAF compilation/installation?
Thanks a lot,
Have a nice day,
Stefano
P.S.: why couldn't I register to IRAF.net with my yahoo email address?
I have a question about imcombine (or mscstack).
I was testing whether it will be possible to use IRAF to produce mosaics out of huge numbers of images, but I could not figure out what is the actual limit.
I'm running the NOAO PC-IRAF Revision 2.12.2a-EXPORT Wed Jul 14 20:45:34 MST 2004, installed in the SCISOFT 6.1 package.
The general (images.immatch) imcombine HELP says that the limit is approx 4000 images (but somewhere else I read 4020, just can't remind where).
I'm using very small, 16x16 pix images, but I might have around 5e4 images (or more in the future) to pile up. I used the MSCSTACK incarnation on the combine. Apparently it loads all images, but then crashes when computing the stack at the ~7700th image. I'm not using any BPM mask or additional imputs/outputs rather than the science frames and the stacked image.
It don't think it's a memory issue, because it's using roughly 30% only of my RAM.
So my questions are:
- is the max number of images related also to their size? e.g. 4000 is a standard number for a 512x512 image (?)
- what's the main cause of this limit? coding issues, memory/records managment, or something else?
- If I use MEFs, will I be limited to 4000 extensions or to 4000 MEFs? In the latter case I could use some MSCJOIN tricks.
- is there a way to increase this number by playing with login.cl parameters or during IRAF compilation/installation?
Thanks a lot,
Have a nice day,
Stefano
P.S.: why couldn't I register to IRAF.net with my yahoo email address?
Mike Fitzpatrick wrote on Feb 26, 2009
Frank will respond if my answer is too far off the mark. IRAF has a limit of 4096 file descriptors that it can keep open at any one time (changing this requires rebuilding the system), but by default IMCOMBINE will know when the limit is about to be exceeded and close unused files to open new ones so that it can process the entire list. The other major limitation is the current 32-bit constraint on the system, i.e. if the combined image or the amount of memory used exceeds 2GB the program will crash and you'll need to process large lists in smaller chunks or use another non-IRAF program to make large field mosaics. I believe the FITS kernel opens multiple descriptors on a file but off-hand I can't remember whether these are open for the life of the image or just when initially mapping it (i.e. to locate the offsets for the MEF extensions).
For your large list of small images I don't think you're exceeding the 2GB limit. You didn't say what kind of crash happened at the 7700th image, if it was something like a segmentation violation or a bus error then this may indicate a bug and some pointer just got trashed during a large list. Frank would have to be able to reproduce this to fix it, but you might try upgrading to a newer IRAF to see if recent changes to the task have already taken care of it. It would also help to know whether this is a linux or Mac system and what version.
About the only think you could change in your login.cl that might affect things is the 'min_lenuserarea' variable that determines the max size of a header, try increasing this if you get an error indicating you might be out of header room but if it's a generic segfault crash more debugging will be required.
Hope this helps,
-Mike
Public email like yahoo and gmail are currently blocked due to excessive spam registrations. I can create a login manually if needed.
For your large list of small images I don't think you're exceeding the 2GB limit. You didn't say what kind of crash happened at the 7700th image, if it was something like a segmentation violation or a bus error then this may indicate a bug and some pointer just got trashed during a large list. Frank would have to be able to reproduce this to fix it, but you might try upgrading to a newer IRAF to see if recent changes to the task have already taken care of it. It would also help to know whether this is a linux or Mac system and what version.
About the only think you could change in your login.cl that might affect things is the 'min_lenuserarea' variable that determines the max size of a header, try increasing this if you get an error indicating you might be out of header room but if it's a generic segfault crash more debugging will be required.
Hope this helps,
-Mike
P.S.: why couldn't I register to IRAF.net with my yahoo email address?
Public email like yahoo and gmail are currently blocked due to excessive spam registrations. I can create a login manually if needed.
Francisco Valdes wrote on Feb 26, 2009
What you describe will stretch the system. A lot of effort has gone into IMCOMBINE to handle the problem of many images. This includes opening and closing files to get past limits in the number of open files (file descriptors). There are also steps to only open files that are overlapping in the case where a large number of partially overlapping tiles are being stacked (such as a strip of the sky). There are many aspects to this and I would not be surprised that it cannot handle the full case you describe. I've never gone beyond 5000 images.
This would be hard to diagnose. I think I would have to say that you should try and figure out a way to do things in multiple steps; i.e. do subset combinations and then combine those.
Yours,
Frank Valdes
This would be hard to diagnose. I think I would have to say that you should try and figure out a way to do things in multiple steps; i.e. do subset combinations and then combine those.
Yours,
Frank Valdes
Last post on Feb 26, 2009