iraf-v216 · Code · Issues (50) · Pull requests (81)
iraf.net Issue #3
Please contribute to the official IRAF repository
open olebole opened this issue on 2017-04-27 · 29 comments
olebole commented on 2017-04-27
Hi all,
This is basically a call for all people which I know who tried to adopt IRAF, to contribute their patches to the official github repository. This is @joequant, @pkgw, @robsteele49, @steelewool, @sergiuspro (Fedora), @fedya (Mandriva), @jehturner (Astroconda). If you know more, please let them know.
While it seems still unclear what happens with this repository long-term, it could be a central place to collect and coordinate all patches that are available so far. It would be a pity if all these attempts got lost!
Maybe @iraf can write here a bit about the rules and goals of this repository.
Best regards
Ole
joequant commented on 2017-04-27
COOL!!!!!
It will take me about two weeks, but I’ll start moving over my patchs.
Thanks!!!
On Thu, Apr 27, 2017 at 4:11 PM, Ole Streicher wrote:
[…]
RobSteele49 commented on 2017-04-27
I’m looking forward to this collaboration.
On Thu, Apr 27, 2017 at 5:25 AM Joseph Wang wrote:
[…]
pkgw commented on 2017-04-27
I am happy to pitch in! You are certainly encouraged to steal any patches you want from my hacked-up repository.
By the way, the work that I did was in service of the goal of creating IRAF and Pyraf packages for conda. My Conda recipes are here. However, my recipe no longer works because new Conda requires software that can handle paths longer than 256 characters, and stock IRAF does not. I did some preliminary work to try to patch IRAF to fix this, but didn’t get it working. There are some data structures that have fixed-size path buffers and I didn’t figure out how to resize them appropriately.
jehturner commented on 2017-04-27
Thanks Ole & Mike! It’s good that there’s now an official repo. To be honest, however, there are a handful of practical obstacles to us adopting this in the foreseeable future:
-
The source we are using is still based on 2.16, not 2.16.1. We just haven’t found the effort to re-base on 2.16.1 when our patched 2.16 from Ureka was working for us and we’ve had to prioritize more specific problems such as new OS support. I did, however, report a number of our original 2.16 bug fixes to Mike at the time, so I’d hope those are already in this repo. Also, we only build 32-bit binaries, since STScI has never ported most of their substantial code base to 64 bits (which I think will limit the useful lifetime of our distribution, as filesystems with >32-bit pointers become more common).
-
I’m just moving my focus back to unrelated work that needs to take priority, after an extended period of focusing on repackaging IRAF software in Astroconda (with some updates soon to be released). When I do next work on Astroconda IRAF, I’ll have to spend some weeks troubleshooting test differences on newer-OS builds, which will probably account for most of my available time this year.
-
We (Gemini & STScI) don’t distribute the 100% complete IRAF source, due to some third-party copyright licensing restrictions. The Ureka/Astroconda binary distributions do include most of the source files, but a handful have been omitted to meet the conditions of the licences we are using (which differ from NOAO’s older agreement with the authors, following discussion with our staff lawyer at the time). This is something I might have to revisit with management if we were to collaborate on a common source distribution that includes these files.
I might nevertheless be able to dig out a few patches that you might find helpful (eg. if you’ have problems building on recent Apple OSs).
Cheers,
James.
olebole commented on 2017-04-27
@pkgw your IRAF repo has quite a number of commits, and it is hard for others to steal decide what is useful and what was a test. Maybe you could combine them into a few pull requests where you find it is useful for the common use?
@jehturner could you give me some details on the license restrictions? (maybe by personal mail, olebole@debian.org if not of general interest). In Debian, we are hit by the Numerical Recipes code in IRAF, and if you have the same problem, we should use a similar way to solve this (so, remove the same functions and such).
jehturner commented on 2017-04-27
By the way, is there any plan to pull X11IRAF into a github repo.? I think our source code for that is closer to the original tree, but we do have a few fixes to build on recent MacOS versions (with one tcl problem still in need of further troubleshooting, if anyone is a tcl expert…).
pkgw commented on 2017-04-28
@olebole What is the status of the build system in this repository? It will be much more helpful if I can test that my PRs actually compile, but I had an enormous amount of difficulty with getting the build system to work at all on a modern system. The script that I came up with is quite a gross hack so it should be improved on … although it is still probably not as bad as the classical IRAF build system!
olebole commented on 2017-04-28
I guess, this is basically a (cleaned) dump of the original source tarball in the current development version from Mike Fitzpatrick (@iraf). He can probably say more about that.
Since the general feeling is that the IRAF build system needs some improvement, it may worth to bring yours as a proposal. Ideally, @iraf could comment then, so that we are able to converge to something better than the current.
iraf commented on 2017-04-28
A “make sysgen” is still the target build-from-source command but there are
known bootstrapping issues (mostly with the vendor dir and libVO.a) still
to be addressed. If you don’t want to wade through these right away,
installing the binaries will allow you to recompile any other source
changes for testing.
On Fri, Apr 28, 2017 at 5:26 AM, Ole Streicher wrote:
[…]
RobSteele49 commented on 2017-04-28
The short story is that as delivered to the repository the ‘make sysgen’
doesn’t work. Zach Steele (@zcsteele) & I are trying to fix that. We forked from
iraf-v216 to steelewool if we want to see our status/work. Our goal is that when we have something fixed we will issue a pull request.
On Fri, Apr 28, 2017 at 12:17 PM, Peter Williams
wrote:
[…]
joequant commented on 2017-04-29
Please take a look at the branch linux-build on joequant/iraf.
I have a build.sh script that bootstraps IRAF on linux. Also there is an
rpm generated on Mageia which you can find via google.
On Sat, Apr 29, 2017 at 7:07 AM, RobSteele49 wrote:
[…]
pkgw commented on 2017-04-29
My repo expands @joequant’s build.sh script to provide a more full build on OSX and Linux, but it is not general purpose — it was only designed to work in the context of the conda-build system. I am not sure how much work it would take to make it more general; it would definitely take a lot of work to make it all less gross.
RobSteele49 commented on 2017-04-29
Zach @zcsteele and I will both look over the build.sh scripts. But it makes more sense to me that we stick with the (./install; make linux64; make sysgen) method since that is how it appears IRAF was build and fix the problem with not being able to build linux64. But I’ll look to merge the changes I find in @joequant and other repositories into my forked version of iraf-v216. It looks to me like the current build system is close to working for linux64 and my goal is to get that working and then move to get things working on a Raspberry Pi.
joequant commented on 2017-04-30
Just a note. I’m trying to get my stuff working with the current tree, and once that’s done (probably in one day or so), I’ll put in a pull request.
I agree that we should be do stuff through the Makefile since you can do stuff like parallel compiles, and the build.sh is just a placeholder to get something that basically works.
Also, it turns out that there was a lot of hairy pointer arithmetic that I needed to go through to get something working on linux64. It’s in the patch, but I forgot the details.
olebole commented on 2017-04-30
I think that these are basically two more or less independent problems:
- Do the bootstrapping steps in the correct order
- Patch the source files so that they work without the need of having an installed
/iraf/tree
As far as I understand, the build.sh solves the first problem, right? For this, I also have a (kind-of) Makefile, since in Debian we use make to create our packages. I could re-base that to the root of the current tree and submit a pull request. But this does not solve the second issue.
pkgw commented on 2017-05-01
I am not sure if those problems are that independent. @joequant and I ended up with trees with build.sh scripts that were able to build everything from scratch, but I certainly needed to patch the source all over the place to get there.
The IRAF build system desperately needs replacement (edit: eventually, that is; I agree with @olebole below that first one should get something going based on what exists now) — I have a lot of experience in this field and IRAF is probably the worst I have ever seen; by modern standards it would get you sued for malpractice, if software engineering were regulated as tightly as it should be. If anyone is going to work on a fundamental rework, I would highly suggest basing it on ninja rather than make. I have begin using ninja virtually everywhere I used to use make and it is a pleasure to work with — clean, modern, fast, simple.
olebole commented on 2017-05-01
IMO we need to first get something that just works. I agree with @iraf that it should be proven to build/work on all supported platforms (Linux and MacOSX, 32 and 64 bit). And I also think it is good to start with some incremental steps.
For example, one of the problems is that the code still expects to have an installation in /iraf/iraf/: Solving this is either renaming files and code content, or adding some CFLAGS options to the Makefiles. This can ofcourse be done in a completely new build system, but it can also first done in the current one, and I would prefer the latter.
So, unless you come with a “ninja” build system including .travis.yml as well as a circle.yml that proves the code builds on all platforms, I would be quite conservative here and go step-by-step.
BTW, CMake would be another candidate here. Its out-of-source build functionality would be great for the bootstrapping process.
pkgw commented on 2017-05-01
Yes, definitely, step one is to get things going with minimal changes. Unfortunately, it is hard for me to see how to gradually transition from the current system to a new one — I just wanted to provide a recommendation if anyone has been thinking about taking a burn-it-down-and-start-again approach.
I have never enjoyed using CMake …. but, that being said, in some cases CMake can actually emit Ninja files as its output, the way it can emit Makefiles.
steelewool commented on 2017-05-01
I think the references in the code to ‘/iraf/iraf’ is something that needs to be fixed independent of the build system. Just a half hearted try by me find the following references:
steele@steele-VirtualBox:~/git/iraf-v216$ grep “iraf/iraf” //*.[xch]
sys/fmtio/evexpr.x:# line 1 “/iraf/iraf/lib/yaccpar.x”
sys/fmtio/evvexpr.x:# line 1 “/iraf/iraf/lib/yaccpar.x”
steele@steele-VirtualBox:~/git/iraf-v216$ grep “iraf/iraf” ///.[xch]
unix/boot/mkpkg/host.c: * ./libsys.a -> /iraf/iraf/lib/libsys.a
unix/boot/mkpkg/host.c: * /iraf/iraf/lib/libsys.a -> ../bin/libsys.a
unix/boot/mkpkg/host.c: * -> /iraf/iraf/bin/libsys.a
unix/boot/mkpkg/host.c: * /
olebole commented on 2017-05-01
IMO this should be better discussed in #7. And maybe one could make a PR that just adresses that? It must be contained in the currently existing build changes, and this would already be a big advantage!
RobSteele49 commented on 2017-05-01
Maybe we are on different pages. The problem I was seeing with ‘/iraf/iraf’
was in file that were referencing this in include files with statements
like “#include “/iraf/iraf…”. It doesn’t seem to me that changes to the
install script will fix this. Alos, CFLAGS is the makefile convention for
passing collections of parameters to the compiler - so I would not consider
it a ‘workaround’. It seems to me that the source files should be
referencing the path ‘/iraf/iraf’ at all. Thoughts?
On Mon, May 1, 2017 at 8:46 AM, NOAO Data Lab Project <
notifications@github.com> wrote:
[…]
iraf commented on 2017-05-01
‘make’ is used as a driver, most of the actual build is done with the iraf
‘mkpkg’ command (or ‘mkpkg.sh’ scripts when building the kernel). Passing
in a CFLAGS environment from the initial “make sysgen” is a lot more work
than you think.
On Mon, May 1, 2017 at 9:56 AM, RobSteele49 wrote:
[…]
RobSteele49 commented on 2017-05-01
The point I was trying to raise is that the ‘#include /iraf/iraf/…”
statements in the code is wrong. I did not suggest modifying CFLAGS. I was
suggesting changing the path names in the code to be correct. I’m sure
using the /iraf/iraf made perfectly good sense in the past - but for the
present I think it should be changed.
On Mon, May 1, 2017 at 10:00 AM, NOAO IRAF Project <notifications@github.com
wrote:
[…]
noao-datalab commented on 2017-05-01
The point I was trying to make is that the install script edits that path,
most notoriously in the
On Mon, May 1, 2017 at 10:24 AM, RobSteele49 wrote:
[…]
steelewool commented on 2017-05-01
I’m not sure why you are referencing iraf.h file. I was attempting to
discuss the problem I saw with libc.h and other files that were referencing
/iraf/iraf. Having /iraf/iraf somehow link back to the unix/bin directory
doesn’t really make sense. I’m pretty sure the relative link would fix the
problem I’m seeing. Would you suggest something else giving the current
build system?
steele@steele-VirtualBox:~/git/iraf-v216/unix/hlib/libc$ grep “iraf/iraf” *
libc.h:#include “/iraf/iraf/unix/bin/f2c.h”
libc.h:#include “/iraf/iraf/unix/hlib/libc/vosproto.h”
On Mon, May 1, 2017 at 10:34 AM, NOAO Data Lab Project <
notifications@github.com> wrote:
[…]
iraf commented on 2017-05-01
Add ‘libc/libc.h” to the PATHFILES in the install script.
On Mon, May 1, 2017 at 10:57 AM, steelewool wrote:
[…]
steelewool commented on 2017-05-01
Thanks. I’ll give this a whack.
olebole commented on 2017-09-09
@jehturner
I did, however, report a number of our original 2.16 bug fixes to Mike at the time, so I’d hope those are already in this repo.
Sorry that I come back to this only now: The commit 906ae18e (in my own fork) contains all real source changes that were done between 2.16.1 and the first commit of this repository.
It would be good if we can figure out whether your patches are already included, so that we can create pull requests if they are still needed. Could you check that, and maybe create the pull requests for the remaining ones? And/or send the patches to me so that I can compare them and/or create the pull requests?
It would be a pity if we miss here what was already done elsewhere.
@joequant I think that the bugfix patches you had are already converted to separate PRs (mainly #57) right?
Best regards
Ole
sergiuspro commented on 2017-10-15
Hello, Ole! Glad to hear from you since my days at the AIP’s eScience. :)
Sorry for my so much delayed reaction to your call, I am not a “heavy user” of github and just now I’ve noticed to be mentioned by you here.
The IRAF repo in my github account (like some others, too) is only a kind of an IRAF user manual in Russian language, which I’ve been slowly writing since summer 2015 (when I’ve attended an IRAF photometry workshop in Russia) and much later decided to “misuse” github as a static website to let others access the manual on the web. So, I am not sure if my user manual (which is more or less permanently “under construction”, because I work on it alone and it will probably never end in some sense) should be contributed to @iraf. What do you think?
Best regards
Sergei
Last updated on 2017-10-15