Problem with large FITS binary catalog

SongHuang wrote on Jul 23, 2009

Hi, Everyone

I met some problem when I tried to use the tasks from STScI Tables package to read the FITS binary catalog from NYU-VAGC catalog:

http://sdss.physics.nyu.edu/vagc/

The tasks failed to open the object_sdss_spectro.fits file whose size is about 2.5GB and this size is a little larger than my physical memory.

So, I do not know whether this problem could be solved when I do this on a computer has memory larger than 2.5GB? Or maybe there is a limit from the task it self...

And, of course, is there any better way to deal with a FITS catalog as large as this. I already tried this on Topcat software by using the -disk flag, but it did not work out.

Thanks a lot

Quietskies

---

Mike Fitzpatrick wrote on Jul 23, 2009

IRAF is a 32-bit application (at present, I'm working on it though) and 2.5GB exceeds the max size file that can be used. For now you'll need to use some other tools to either extract the data you need or else split the file into smaller pieces you can manage.

Problems with TOPCAT should be directed to Mark Taylor who is usually very responsive to bugs. You didn't specify what exactly didn't "work out", but note the STILTS tools are commandline versions of some TOPCAT functionality and might work better if what you saw was an interface issue of some kind. Hope this helps.

-Mike

---

SongHuang wrote on Jul 23, 2009

Hi

Thank you for your answer

I got the same answer from people at NYU, it seems impossible to handle such a large file on a 32-bit computer....

And, about TOPCAT, the error is something like the size is beyond the limit of Integer.MAX....

I think I'd better find a 64-bit computer to finish this job..

Best wishes

Yours Quietskies

---

Last post on Jul 23, 2009