HSC Pipeline Binary Distribution

日本語

Here is the HSC analysis pipeline distributed in a binary format.

NOTE: The pipeline distributed here is not yet in a phase where users can simply trust its outputs for science. The help desk (Open-use support site) offers support for HSC open-use users if they use those versions of the pipeline which were used in producing the (internally or publicly) released SSP data. The other versions are for engineering use only, and we cannot provide support for them.

Suggestions or bug reports are welcome to helpdesk. (Do not send the address bug reports related to the pipeline itself, except those related to the compiled binary.)

This page is a translation from the Japanese original.

  1. Environment
  2. Download
  3. Install
  4. Usage (bash)
  5. Tutorial (in different pages)
  6. Build from source
  7. Links

Environment

Assume you want to process 300 shots that you have taken in 3 nights. Then, a typical environment required is as follows:

Remarks
CPU64bitOnly x86_64 is tested.
Cores12 (24 hyper-threads)Not a requirement but a recommendation.
Memoy64GBStacking too wide a region might require more.
Storage10TBIn case of 300 shots × object frames.

Download

Download the program and data.

Program

Download one of those in the following list, one that is appropriate to your environment. Recent versions of the binary packages, when expanded, are about 10GB.

The central part (hscPipe, LSST stack) of the program is distributed under GPLv3, and the rest is under other open source licenses like BSD. Scripts added by the binary distributor are under GPLv3.

versionRemarksOSarch
6.7 S18A + afterburner CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
build script script
6.5.3 S18A CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
build script script
5.4 S17A CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
build script script
4.0.5 r2 filter support CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
Ubuntu 16 x86_64 (sha256sum)
build script script
4.0.2 S15b incremental (i2 filter support from this on) CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
Ubuntu 16 x86_64 (sha256sum)
build script script
4.0.1 S15b CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
Ubuntu 15 x86_64 (sha256sum)
build script script
3.8.6c_hsc Critical bug fixes for 3.8.5 CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
Ubuntu 15 x86_64 (sha256sum)
build script script
3.8.5 S15a CentOS 6 x86_64 (sha256sum)
CentOS 7 x86_64 (sha256sum)
Ubuntu 15 x86_64 (sha256sum)
build script script
3.4.1 S14A_0b Wide (still alpha) CentOS 6 x86_64 (sha256sum)
2.12.4d_hsc S14A_0 CentOS 6 x86_64 (sha256sum)
build script script

Data

Reference catalog

The analysis pipeline requires a reference catalog for photometric and astrometric calibration. The catalog can be downloaded from community.lsst.org ("LSST-style format"). The linked page recommends that you take a subset instead of grabbing all the files. In case you want to get all, we have made a tarball containing all of them: reference-ps1_pv3_3pi_20170110.tar.xz.

Y-band straylight

The analysis pipeline >= v6.x requires straylight pattern files to process Y-band images. (https://community.lsst.org/t/y-band-stray-light-correction-for-hsc/2517) Here is the tarball of the straylight files: STRAY_LIGHT.tar.xz The contents of this tarball are the same as those in the linked page.

Raw image files for tutorial

If you want to try the tutorial, download image files: rawdata.tar.xz

Obsolete reference catalogs

These reference catalogs are for hscPipe version ≤ 4.x. If you use the latest version, you don't need them.

astrometry_net_data
(http://astrometry.net/)
SDSS DR9 (about 20GB)
converted by the HSC software team
Note a further photometric calibration is done with the use of PS1.
SDSS DR8 (about 50GB)
converted by the HSC software team
Or, build one by yourself

Install

Install the program

(Case I) If you can be root, be root, first of all. Then expand the program archive into /opt. ↓

[root@example ~]# cd /opt
[root@example opt]# tar xvaf ~/hscPipe-VERSION-centos-6-x86_64.tar.xz

When you have installed the program here, there is nothing else you have to do as to program installation. Go on to data installation.

(Case II) If you cannot be root, expand the program archive in any directory you like. Only, you should avoid too deep a location in the directory hierarchy. As you will understand when you finish installation and say “setup-hscpipe && echo $PATH”, $PATH would surpass the capacity of the shell in its length if the program location were deep. In fact, csh's capacity is usually surpassed.

For instance, if you are to place it in ~/ana (All files will go under ~/ana/hscpipe/VERSION), do as follows:

[user@example ~]# mkdir ~/ana
[user@example ~]# cd ~/ana
[user@example ana]# tar xvaf ~/hscPipe-VERSION-centos-6-x86_64.tar.xz

In addition, run a script in the package as:

[user@example ana]# cd hscpipe/VERSION
[user@example VERSION]# ./install.sh

↑ This script will finish in about a minute. Go on to data installation. Beware that, in reading the rest of this page, you have to alter program paths in accordance with your installation.

Install data

Second, install the reference catalog. Unlike old versions, special commands are no longer required for recent hscPipe. Just expand the tarball in an arbitrary directory with sufficient room (~500GB).

[root@example ~]# cd SOME_DIRECTORY
[root@example dir]# : Expand the archive
[root@example dir]# tar xvaf ~/reference-ps1_pv3_3pi_20170110.tar.xz

Register a setup command

Finally, add to your ~/.bashrc (Be careful not overwrite root's): ↓

[user@example ~]$ vi ~/.bashrc
:
: (The original contents)
:

source /opt/hscpipe/VERSION/bashrc

That is all as to installation. Note that no piece of the software thus installed is ready to use unless you type setup-hscpipe into the terminal. Every time you log in, or you launch a new terminal, setup-hscpipe is necessary.

Usage (bash)

Use bash shell. Keep away from csh.

Set up the HSC pipeline

[user@example ~]$ setup-hscpipe

↑ This command sets huge lots of environment variables for the HSC pipeline to work.

Reset the environment variables.

[user@example ~]$ reset-env

↑ This command (perhaps) resets the environment back to the state it was on log-in.

Build from source

If you use an OS not listed in the list above, you can build the pipeline for yourself. Since version 5.x, the build process has been so stabilized that usual users can build the pipeline for themselves, given they are using Linux.

If you have gcc ≥ 4.8, we recommend following LSST's installation manual to build the pipeline. You should however take different steps not to install the LSST stack but hscPipe:

[user@example ~]$ cd /path/to/hscPipe6
[user@example hscPipe6]$ export EUPS_PKGROOT=http://master.ipmu.jp/~price/packages-hscPipe6/
[user@example hscPipe6]$ wget $EUPS_PKGROOT/newinstall.sh
[user@example hscPipe6]$ bash newinstall.sh # Answer yes to question
[user@example hscPipe6]$ source /path/to/hscPipe6/loadLSST.bash
[user@example hscPipe6]$ eups distrib install hscPipe VERSION

↑ With these commands, you will get “miniconda” version of the pipeline.

If you don't have gcc ≥ 4.8, or if you want exactly the same version as the distributed binary package, use the build scripts we provide.

Recommended building environment

2 GB of memory for each core will ease the build. 1 GB of memory for each core would do, but miseries await you.

Links

http://hsc-jira.astro.princeton.edu
Report bugs here.
Forum
Ask questions here.
Mailing list archive
The mailing list is: hsc_software.
http://hsca.ipmu.jp
Manuals etc. (Password asked; do not forward the question to me)
LSST Software User Guide
The manuals of the LSST Software, the basis of the HSC pipeline.