memory corruption in xyxymatch

gelato wrote on Jul 05, 2010

A colleague has stumbled upon a bug in xyxymatch. When nrefstars==3, nrmaxtri (which is used in computing the allocation size for reftri) ends up being equal to 1. Later, rg_tvote() (called via rg_match()) assigns values to reftri(2,RG_MATCH) and reftri(3,RG_MATCH) which are out of bounds.

I suggest the following changes to the code:

[list=1:473cb7fdde]

in pkg/images/immatch/src/listmatch/t_xyxymatch.x add a line

      nrmaxtri = max (nrefstars, nrmaxtri)

before calling calloc(reftri, ...). A similar change is almost certainly required in pkg/images/imcoords/src/t_ccxymatch.x . Also, the computed value for ninmaxtri may need to be adjusted by

      ninmaxtri = max (nliststars, nrefstars, ninmaxtri)

in pkg/images/lib/rgmerge.x, document the requirement that nrmtri>=nrefstars in rg_tvote().

also in pkg/images/lib/rgmerge.x, correct the definition of rg_factorial() to conform with the documentation: there is a hardcoded 3 which should really be ngroup. It doesn't matter to t_xyxymatch() which calls rg_factorial with ngroup==3, but it isn't right.
</ul> </div> --- **Mike Fitzpatrick** wrote on Jul 05, 2010

Thanks for the detailed tracking of the problem. We'll have a closer look and fix this for the next release of the system. I'll post again if we have questions and will post the buglog once it's fixed.

Thanks again,
--Mike

--- *Last post on Jul 05, 2010*