How to make a catalog for astrometry.net

日本語

Here is explained how to make an astrometry_net_data by yourself. Proceed with this page in one hand, and the legitimate manual in another.

Get the latest astrometry.net

If you are using hscPipe 3.x or later, you should skip this section. But if you are using hscPipe 2.12.4d_hsc, there are some required tools that are not offered by the astrometry.net in that version of hscPipe because the version of the astrometry.net is old. You must build the latest astrometry.net by yourself.

Required libraries

yum install zlib-devel bzip-devel \
    libpng-devel libjpeg-turbo-devel \
    cairo-devel

Download the source

Download the latest source from the originator's download page. Version 0.49 is the latest as of 2014-06-18.

Build & install

[user@example ~]$ cd /dev/shm
[user@example shm]$ tar xvaf ~/astrometry.net-0.49.tar.bz2
[user@example astrometry.net-0.49]$ cd astrometry.net-0.49
[user@example astrometry.net-0.49]$ make \
    INSTALL_DIR=$HOME/usr/astrometry               $(: as you like) \
    CFITS_INC="-I $HOME/usr/cfitsio/include"       $(: where cfitsio resides) \
    CFITS_LIB="$HOME/usr/cfitsio/lib/libcfitsio.a" $(: where cfitsio resides) \
    install
[user@example astrometry.net-0.49]$ cd ..
[user@example shm]$ rm -rf astrometry.net-0.49
[user@example shm]$ cd ~
[user@example ~]$ export PATH="$HOME/usr/astrometry/bin:$PATH"

Prepare a catalog

Prepare a CSV file that looks like the following: (If you already have a FITS BinTable, you can skip this section.)

id,ra,dec,ra_err,dec_err,starnotgal,r,u,g,i,z,r_err,u_err,g_err,i_err,z_err
1237645879551066262,348.90252656,1.27188781,0.01384772,0.01628234,0,19.410612,18.237537,17.581324,17.201529,16.901588,0.046767,8.25278E-3,7.026487E-3,8.094831E-3,0.021641
1237645879562862699,15.89612394,1.26484677,0.01108913,7.81238953E-3,0,20.149837,19.282238,19.031107,18.95554,18.782871,0.046607,0.011408,0.012107,0.017041,0.054156
 :
 :
(Values are from SDSS DR8)

The names of filters and the number of filters are arbitrary within astrometry.net, but in hscPipe, you have to obey the following rule to prevent trouble.

• Fileter names are g, r, i, z, y, ... in small letters.

Note that the magnitudes will be used not only in astrometry but also in photometry calibration.

Convert it to a FITS BinTable

Next, convert the CSV to a FITS BinTable. (Instead of converting a CSV, you may prepare a FITS BinTable with your favorite tools.)

Download the conversion script csv-to-fits.py and do as follows:

[user@example ~]$ setup-hscpipe
[user@example ~]$ python csv-to-fits.py src-catalog.csv src-catalog.fits

Separate the catalog into fragments

Separate into fragments the FITS BinTable you have prepared above↑, so that each of the fragments will correspond to a Healpix pixel. Thus:

[user@example ~]$ mkdir workdir
[user@example ~]$ hpsplit -o workdir/cat-%i.fits -r ra -d dec -m 1 src-catalog.fits -n 2

Create index files

Create files that astrometry.net actually uses:

[user@example ~]$ filter=r # The default filter
[user@example ~]$ mkdir output
[user@example ~]$ # For all the catalog files that've been created in workdir:
[user@example ~]$ find workdir -name "cat-*.fits" -print0 | while read -r -d '' catalog
do
    # From catalog="workdir/cat-###.fits", generate index="output/index-###.fits"
    index="${catalog/"workdir/cat-"/output/index-}"
    # From index="output/index-###.fits", trim the extension ".fits"
    index="${index%.*}"

    # Build preset 0
    build-astrometry-index -v -i $catalog -o $index-0.fits -P 0 -S $filter -n 100 -r 1 -j 0.4 -E
    # preset 1, 2, 3
    for p in {1..3}
    do
        build-astrometry-index -v -1 $index-0.fits -o $index-$p.fits -P $p -S $filter -n 100 -r 1 -j 0.4 -E
    done
done

Create a config file

Describe in a config file what the index files are.

[user@example ~]$ vi output/andConfig.py
# Write as follows
import os.path
import os
import glob

root.starGalaxyColumn = "starnotgal"
filters = ("r", "u", "g", "i", "z") # Enumerate the filters in the catalog you prepared
root.defaultMagColumn = "r"
root.magColumnMap = dict([(f,f) for f in filters])
root.magErrorColumnMap = dict([(f, f + "_err") for f in filters])
root.indexFiles = [os.path.basename(path) for path in glob.glob(os.path.join(os.environ["ASTROMETRY_NET_DATA_DIR"], "index-*.fits"))]

Create a UPS directory

Eups requires a UPS directory so that it can use the output directory you have created above.

[user@example ~]$ mkdir -p output/ups
[user@example ~]$ touch output/ups/astrometry_net_data.table # empty file

Install

Rename the output directory an appropriate name, and move it into an appropriate directory.

[user@example ~]$ mv output MYCATALOG
[user@example ~]$ mv MYCATALOG /opt/hscpipe/astrometry_net_data/

Note that MYCATALOG must match one of the following wildcards:

sdss*

If the catalog is from SDSS, prefix "sdss" to the catalog name. The pipeline will take account of the difference in filter property between SDSS and HSC.

ps1*

If the catalog is from Pan-STARRS, prefix "ps1" to the catalog name. The pipeline will take account of the difference in filter property between Pan-STARRS and HSC.

hsc*

This is used when the catalog is from HSC itself, but it can also be used when you do not want color correction.

Register it in eups

[user@example ~]$ eups declare \
    -m /opt/hscpipe/astrometry_net_data/MYCATALOG/ups/astrometry_net_data.table \
    -r /opt/hscpipe/astrometry_net_data/MYCATALOG/ \
    astrometry_net_data  MYCATALOG

To make astrometry.net use the data you have created, set it up thus:

[user@example ~]$ setup-hscpipe
[user@example ~]$ setup astrometry_net_data MYCATALOG