HSC Pipeline Binary Distribution

日本語

Here is the HSC analysis pipeline distributed in a binary format.

NOTE: The pipeline distributed here is not yet in a phase where users can simply trust its outputs for science. The help desk (Open-use support site) offers support for HSC open-use users if they use those versions of the pipeline which were used in producing the (internally or publicly) released SSP data. The other versions are for engineering use only, and we cannot provide support for them.

Suggestions or bug reports are welcome to helpdesk. (Do not send the address bug reports related to the pipeline itself, except those related to the compiled binary.)

This page is a translation from the Japanese original.

1. Environment
2. Download
3. Install
4. Usage (bash)
5. Tutorial (in different pages)
   • for hscPipe 7.x under construction
   • for hscPipe 6.x
   • for hscPipe 5.x
   • for hscPipe 4.x
   • for hscPipe 3.x
   • for hscPipe 2.x
6. Build from source
7. Links

Environment

Assume you want to process 300 shots that you have taken in 3 nights. Then, a typical environment required is as follows:

Download

Download the program and data.

Program

Download one of those in the following list, one that is appropriate to your environment. Recent versions of the binary packages, when expanded, are about 10GB.

The central part (hscPipe, LSST stack) of the program is distributed under GPLv3, and the rest is under other open source licenses like BSD. Scripts added by the binary distributor are under GPLv3.

Data

Reference catalog

The analysis pipeline requires a reference catalog for photometric and astrometric calibration. The catalog can be downloaded from community.lsst.org ("LSST-style format"). The linked page recommends that you take a subset instead of grabbing all the files. In case you want to get all, we have made a tarball containing all of them: reference-ps1_pv3_3pi_20170110.tar.xz.

Y-band straylight

The analysis pipeline >= v6.x requires straylight pattern files to process Y-band images. (https://community.lsst.org/t/y-band-stray-light-correction-for-hsc/2517) Here is the tarball of the straylight files: STRAY_LIGHT.tar.xz The contents of this tarball are the same as those in the linked page.

Raw image files for tutorial

If you want to try the tutorial, download image files: rawdata.tar.xz

Obsolete reference catalogs

These reference catalogs are for hscPipe version ≤ 4.x. If you use the latest version, you don't need them.

Install

Install the program

(Case I) If you can be root, be root, first of all. Then expand the program archive into /opt. ↓

[root@example ~]# cd /opt
[root@example opt]# tar xvaf ~/hscPipe-VERSION-centos-6-x86_64.tar.xz

When you have installed the program here, there is nothing else you have to do as to program installation. Go on to data installation.

(Case II) If you cannot be root, expand the program archive in any directory you like. Only, you should avoid too deep a location in the directory hierarchy. As you will understand when you finish installation and say “setup-hscpipe && echo $PATH”, $PATH would surpass the capacity of the shell in its length if the program location were deep. In fact, csh's capacity is usually surpassed.

For instance, if you are to place it in ~/ana (All files will go under ~/ana/hscpipe/VERSION), do as follows:

[user@example ~]# mkdir ~/ana
[user@example ~]# cd ~/ana
[user@example ana]# tar xvaf ~/hscPipe-VERSION-centos-7-x86_64.tar.xz

In addition, run a script in the package as:

[user@example ana]# cd hscpipe/VERSION
[user@example VERSION]# ./install.sh

↑ This script will finish in about a minute. Go on to data installation. Beware that, in reading the rest of this page, you have to alter program paths in accordance with your installation.

Install data

Second, install the reference catalog. Unlike old versions, special commands are no longer required for recent hscPipe. Just expand the tarball in an arbitrary directory with sufficient room (~500GB).

[root@example ~]# cd SOME_DIRECTORY
[root@example dir]# : Expand the archive
[root@example dir]# tar xvaf ~/reference-ps1_pv3_3pi_20170110.tar.xz

Register a setup command

Finally, add to your ~/.bashrc (Be careful not overwrite root's): ↓

[user@example ~]$ vi ~/.bashrc
:
: (The original contents)
:

source /opt/hscpipe/VERSION/bashrc

That is all as to installation. Note that no piece of the software thus installed is ready to use unless you type setup-hscpipe into the terminal. Every time you log in, or you launch a new terminal, setup-hscpipe is necessary.

Usage (bash)

Use bash shell. Keep away from csh.

Set up the HSC pipeline

[user@example ~]$ setup-hscpipe

↑ This command sets huge lots of environment variables for the HSC pipeline to work.

Reset the environment variables.

[user@example ~]$ reset-env

↑ This command (perhaps) resets the environment back to the state it was on log-in.

Build from source

If you use an OS not listed in the list above, you can build the pipeline for yourself. Since version 5.x, the build process has been so stabilized that usual users can build the pipeline for themselves, given they are using Linux.

We recommend following LSST's installation manual to build the pipeline. You should however take different steps not to install the LSST stack but hscPipe:

[user@example ~]$ cd /path/to/hscPipe8
[user@example hscPipe8]$ export EUPS_PKGROOT=https://tigress-web.princeton.edu/~HSC/hscPipe8/
[user@example hscPipe8]$ wget $EUPS_PKGROOT/newinstall.sh
[user@example hscPipe8]$ bash newinstall.sh -t # Answer yes to question
[user@example hscPipe8]$ source /path/to/hscPipe8/loadLSST.bash
[user@example hscPipe8]$ eups distrib install hscPipe VERSION

↑ With these commands, you will get “miniconda” version of the pipeline.

If you want exactly the same version as the distributed binary package, use the build scripts we provide.

Recommended building environment

• CPU: More than 4 cores.
• Memoy: More than 8 GB.

2 GB of memory for each core will ease the build. 1 GB of memory for each core would do, but miseries await you.

Links

http://hsc-jira.astro.princeton.edu

Report bugs here.

Forum

Ask questions here.

Mailing list archive

The mailing list is: hsc_software.

http://hsca.ipmu.jp

Manuals etc. (Password asked; do not forward the question to me)

LSST Software User Guide

The manuals of the LSST Software, the basis of the HSC pipeline.