hsel non-graceful handles invalid string as keyword

Jason Quinn wrote on May 24, 2008

I ran across some images that have in their header

FILTER  = 'r''                                     / FILTER NAME

which violates the FITS 2.0 standard with its quoting and causes hsel to freak out. I think it would be valuable if hedit did a basic check on the formatting of strings in keywords. It could then either throw an error or try to work around problems. For instance, in this case, it might be a good idea to default to the maximum size string defined by the first and last single quote.

Jason

FITS 2.0
5.2.1 Character String
If the value is a fixed format character string, column 11 shall contain a single quote
(hexadecimal code 27, “’”); the string shall follow, starting in column 12, followed by
a closing single quote (also hexadecimal code 27) that should not occur before column
20 and must occur in or before column 80. The character string shall be composed only
of ASCII text. A single quote is represented within a string as two successive single
quotes, e.g., O’HARA = ’O’’HARA’. Leading blanks are significant; trailing blanks are
not.
Free format character strings follow the same rules as fixed format character strings
except that the starting and closing single quote characters may occur anywhere within
columns 11–80. Any columns preceding the starting quote character and after column
10 must contain the space character.

---

Mike Fitzpatrick wrote on May 24, 2008

Hi Jason,

This particular keyword violates the standard on two separate counts (i.e. the embedded quote and the string not terminating before column 20). The problem however is farther down in the image i/o routines getting the header keyword before HSELECT even sees it so the fix is a little more subtle than you suggest.

I can add this to the list of future fixes, for now you might just want to use HFIX to fix the keyword value and send a nasty email to the author of the code that write the file. For example,

 hfix foo.fits command="!sed \"s/''/''        '/g\" $fname > t;mv t $fname"

should fix both issues, but be careful you get the quotes and escaped-quotes correct.

Cheers,
-Mike

---

Jason Quinn wrote on May 24, 2008

fitz

 hfix foo.fits command="!sed \"s/''/''        '/g\" $fname > t;mv t $fname"

Very cool solution. I never even knew about the hfix task.

I wanted it put into a UNIX-level script though. Here is the solution that I used the other day:

perl -pi -e "s/FILTER  = \'(.)\'\'/FILTER  = \'\1\' /" badfile.fits

It seems like extended regex replacement from the command line on a binary file is currently a chink in Linux's armor. Despite several hours of searching, I only found two ways to tackle the problem at the UNIX level with command programs (vi and perl) and only one can be scripted (perl). There appears to be a number of good 3rd party binary regex replacement programs although I only would use them as a last resort. Unfortunately, GNU sed can do simple regex replaces for binary files with text but not extended regex replaces in that circumstance. Vi has a binary edit mode and you can do a regex search-n-replace; however this seems to only work properly from within the visual editor and fails if you try to pass it an extended regex from the command line like from within a script. Perl's command line version seems to work after getting the all-important quoting correct but I'm not convinced it does extended regexs properly from the command line either.

Jason

---

Last post on May 24, 2008