GNU Astronomy Utilities NEWS -*- outline -*- Copyright (C) 2015-2024 Free Software Foundation, Inc. See the end of the file for license conditions. * Noteworthy changes in release X.XX (library XX.0.0) (YYYY-MM-DD) ** New publications - https://ui.adsabs.harvard.edu/abs/2023RNAAS...7..269E by Eskandarlou et al. on Gnuastro's zero point measurement script for calibration of astronomical images ('astscript-zeropoint'). - https://ui.adsabs.harvard.edu/abs/2024RNAAS...8...10I by Infante-Sainz & Akhlaghi on Gnuastro's script to visualize the full dynamic range of astronomical images ('astscript-color-faint-gray'). - https://ui.adsabs.harvard.edu/abs/2024RNAAS...8...22I by Infante-Sainz et al. on Gnuastro's script to measure the radial profile of a given source ('astscript-radial-profile'). ** New features *** New program - 'astscript-color-faint-gray': a new installed script to maximize the visualization of the contents within your astronomical images. It does this by showing the brighter parts of the image as color, intermediate regions as black and the noisy/faint regions as gray/white. *** All programs - The following options are now available in all programs to allow customization of which metadata is printed in the 0-th HDU of the FITS output of all Gnuastro programs. Also see the "Changed features" list for an important change in this regard to all Gnuastro output FITS files. --outfitsnoconfig: do not print any metadata in the 0-th HDU (including the used options and their values, the date, the versions and git commit of running directory). --outfitsnodate: do not write the 'DATE' keyword. --outfitsnoversions: do not write the versions of Gnuastro and mandatory dependencies. --outfitsnocommit: do not write the Git commit. *** Arithmetic --arguments: takes the name of plain-text file that contains the list of arguments to the program. This option is critical if you have very long (thousands) of operands and operators (which happen in large pipelines where the arguments are constructed automatically). Without this option, the shell is going to abort with an "Argument list too long" error message. - New operators: - rotate-coord: given a 2D point's coordinates, return the coordinates after it has been rotated by your requested angle around your requested center (see documentation for example). - mad: Median Absolute Deviation (MAD) stacking. - madclip-mad: MAD after MAD-clipping stacking. - madclip-std: Standard deviation after MAD-clipping stacking. - madclip-mean: Mean after MAD-clipping stacking. - madclip-median: Median after MAD-clipping stacking. - madclip-number: Number of elements after MAD-clipping stacking. - madclip-fill-mad: MAD after filled MAD-clipping stacking: this involves a two-phase clipping: after the first clipping, Arithmetic will "fill" the unclipped holes in each input and masks them for a second round. This is critical for clipping more diffuse outliers: where a pixel may not be clipped individually and will produce biased results, but due to proximity with many nearby clipped pixels, it can be discarded and thus its effect on the final stack be removed. - madclip-fill-std: Standard deviation after filled MAD-clipping stacking. - madclip-fill-mean: Mean after filled MAD-clipping stacking. - madclip-fill-median: Median after filled MAD-clipping stacking. - madclip-fill-number: Num. of elements after filled MAD-clipping stacking. - sigclip-fill-mad: MAD after filled MAD-clipping stacking. - sigclip-fill-std: Standard deviation after filled MAD-clipping stacking. - sigclip-fill-mean: Mean after filled MAD-clipping stacking. - sigclip-fill-median: Median after filled MAD-clipping stacking. - sigclip-fill-number: Num. of elements after filled MAD-clipping stacking. - sigclip-mad: MAD after sigma-clipping stacking. - collapse-madclip-mad: Collapse dim. by MAD-clipped MAD. - collapse-madclip-std: Collapse dim. by MAD-clipped STD. - collapse-madclip-mean: Collapse dim. by MAD-clipped mean. - collapse-madclip-median: Collapse dim. by MAD-clipped median. - collapse-madclip-number: Collapse dim. by MAD-clipped number. - collapse-madclip-fill-mad: Collapse dim. by filled MAD-clipped MAD. - collapse-madclip-fill-std: Collapse dim. by filled MAD-clipped STD. - collapse-madclip-fill-mean: Collapse dim. by filled MAD-clipped mean. - collapse-madclip-fill-median: Collapse dim. by filled MAD-clipped median. - collapse-madclip-fill-number: Collapse dim. by filled MAD-clipped num. - collapse-sigclip-mad: Collapse dim. by sigma-clipped MAD. - collapse-sigclip-fill-mad: Collapse dim. by filled sigma-clipped MAD. - collapse-sigclip-fill-std: Collapse dim. by filled sigma-clipped STD. - collapse-sigclip-fill-mean: Collapse dim. by filled sigma-clipped mean. - collapse-sigclip-fill-median: Collapse dim. by filled sigma-clip. median. - collapse-sigclip-fill-number: Collapse dim. by filled sigma-clipped num. *** Fits --arguments: takes the name of plain-text file that contains the list of arguments to the program. Currently this only works with '--keyvalue' and is necessary when you want to get the keywords of a very long list of files (~thousands!). Without this option, the shell is going to abort with an "Argument list too long" error message. --datasum-encoded: new option that will return an ASCII encoded 16-character string for the value of the 'DATASUM' concept in FITS data verification. *** Statistics --checkskynointerp: similar to '--checksky', but stop as soon as the good tiles are found (do not continue to interpolation and smoothing of the tiles which can be time consuming if not needed). --mad: Median Absolute Deviation (MAD) of input dataset. --madclip: MAD-clipping, showing intermediate results. --sigclip-mad: MAD of input after sigma-clipping. --madclip-number: Number of input elements after MAD-clipping. --madclip-median: Median of input elements after MAD-clipping. --madclip-mean: Mean of input elements after MAD-clipping. --madclip-std: Standard deviation of input elements after MAD-clipping. --madclip-mad: MAD of input elements after MAD-clipping. --mclipparams: Parameters of MAD-clipping, similar to sigma-clipping. Note that in a Gaussian distribution, MAD = 0.67 sigma, so if you want something similar to 3-sigma (the default), you should set the MAD multiple to 5 (the default). *** Makefile extension - 'ast-text-to-upper': convert input string to upper-case. - 'ast-text-to-lower': convert input string to lower-case. *** Library **** Functions - gal_dimension_collapse_mclip_mad: MAD-clipped MAD. - gal_dimension_collapse_mclip_fill_mad: filled MAD-clipped MAD. - gal_dimension_collapse_mclip_std: MAD-clipped STD. - gal_dimension_collapse_mclip_fill_std: filled MAD-clipped STD. - gal_dimension_collapse_mclip_mean: MAD-clipped mean. - gal_dimension_collapse_mclip_fill_mean: filled MAD-clipped mean - gal_dimension_collapse_mclip_median: MAD-clipped median. - gal_dimension_collapse_mclip_fill_median: filled MAD-clipped median. - gal_dimension_collapse_mclip_number: MAD-clipped number. - gal_dimension_collapse_mclip_fill_number: filled MAD-clipped number. - gal_dimension_collapse_sclip_mad: sigma-clipped MAD - gal_dimension_collapse_sclip_fill_mad: filled sigma-clipped MAD. - gal_dimension_collapse_sclip_fill_std: filled sigma-clipped STD. - gal_dimension_collapse_sclip_fill_mean: filled sigma-clipped mean - gal_dimension_collapse_sclip_fill_median: filled sigma-clipped median. - gal_dimension_collapse_sclip_fill_number: filled sigma-clipped number. - gal_fits_hdu_datasum_encoded: the 16-character encoded sting datasum. - gal_fits_key_list_add_date: add 'DATE' to current list of keywords. - gal_fits_key_list_add_git_commit: add the Git commit information of the running directory to the input list of keywords (if built with libgit2). - gal_fits_key_list_add_software_versions: add versions of Gnustro and its mandatory dependencies to the input list of keywords. - gal_statistics_clip_mad: MAD clipping of given input. - gal_statistics_mad: return median absolute deviation (MAD). - gal_statistics_median_mad: return median and MAD. - gal_txt_read_to_list: read all the space-separated words of the input plain-text file as a separate node in a linked list of strings. **** Macros - Used by 'gal_arithmetic': - GAL_ARITHMETIC_OP_SIGCLIP_MAD: Sigma-clipped MAD. - GAL_ARITHMETIC_OP_SIGCLIP_MAD: Sigma-clipped STD. - GAL_ARITHMETIC_OP_SIGCLIP_FILL_NUMBER: MAD-clipped num. of arrays. - GAL_ARITHMETIC_OP_SIGCLIP_FILL_MEAN: MAD-clipped mean of arrays. - GAL_ARITHMETIC_OP_SIGCLIP_FILL_MEDIAN: MAD-clipped median of arrays. - GAL_ARITHMETIC_OP_SIGCLIP_FILL_STD: MAD-clipped STD of arrays. - GAL_ARITHMETIC_OP_SIGCLIP_FILL_MAD: MAD-clipped STD of arrays. - GAL_ARITHMETIC_OP_MADCLIP_NUMBER: MAD-clipped number of mult. arrays. - GAL_ARITHMETIC_OP_MADCLIP_MEAN: MAD-clipped mean of multiple arrays. - GAL_ARITHMETIC_OP_MADCLIP_MEDIAN: MAD-clipped median of mult. arrays. - GAL_ARITHMETIC_OP_MADCLIP_STD: MAD-clipped STD of multiple arrays. - GAL_ARITHMETIC_OP_MADCLIP_MAD: MAD-clipped STD of multiple arrays. - GAL_ARITHMETIC_OP_MADCLIP_FILL_NUMBER: MAD-clipped num. of arrays. - GAL_ARITHMETIC_OP_MADCLIP_FILL_MEAN: MAD-clipped mean of arrays. - GAL_ARITHMETIC_OP_MADCLIP_FILL_MEDIAN: MAD-clipped median of arrays. - GAL_ARITHMETIC_OP_MADCLIP_FILL_STD: MAD-clipped STD of arrays. - GAL_ARITHMETIC_OP_MADCLIP_FILL_MAD: MAD-clipped STD of arrays. - Used by 'gal_statistics_clip_sigma' and 'gal_statistics_clip_mad': - GAL_STATISTICS_CLIP_OUTCOL_NUMBER_USED: index of final number in output. - GAL_STATISTICS_CLIP_OUTCOL_MEAN: index of clipped mean in output. - GAL_STATISTICS_CLIP_OUTCOL_STD: index of clipped STD in output. - GAL_STATISTICS_CLIP_OUTCOL_MEDIAN: index of clipped median in output. - GAL_STATISTICS_CLIP_OUTCOL_MAD: index of clipped MAD in output. - GAL_STATISTICS_CLIP_OUTCOL_NUMBER_CLIPS: index of clipped number in output. - GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MEAN: bit flag for measuring mean. - GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_STD: bit flag for measuring STD. - GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MAD: bit flag for measuring MAD. ** Removed features *** Library - gal_fits_key_write_version: redundant with the new 'gal_fits_key_list_add_software_versions'. - gal_fits_key_write_version_in_ptr: as in gal_fits_key_write_version. - gal_fits_key_write_config: redundant since all configuration (options) are stored as generic keys. ** Changed features *** All programs - Date and versions FITS keywords are only written in the 0-th HDU of output FITS files (along with the options names and values), not the HDU(s) containing data. Until now, the data and versions metadata were written in the data HDU. However, ultimately, these are also metadata and are better suited for the 0-th HDU. For example when everything else is identical, you expect an exactly reproducible HDU (to verify with the FITS 'CHECKSUM' for example), but every time you run, 'DATE' will be different! Similarly for the versions of the software or Git commit (they may change, while your dataset doesn't). To disable these in the 0-th HDU also, see the newly added '--outfitsno*' options in the new features. *** Arithmetic - Binary operators (like '+' or 'x') that are given two integers will crash with an error if the input operands have the same width but different signs. Until now, they would just report a warning and print the output. However, in large scripts, users could miss the warning and not be aware of a possibly wrong result. Therefore it is more robust to crash with an error rather than print a warning. This was suggested by Sepideh Eskandarlou and implemented by Faezeh Bidjarchian after a poll on Gnuastro's Matrix chat channel (#gnuastro:openastronomy.org). *** Crop - The 'ICF*' keywords are now written in the 0-th HDU of Crop's outputs (the extension with no data, only metadata). Therefore, the WCS is now the only metadata in the HDU containing data. *** astscript-psf-stamp - When no name is given to the temporary directory, the default name will contain the center coordinates as a suffix (to avoid being confused with other calls for other centers), and the contents of the temporary directy will have the same filename. Until now, the temporary directory name would be the same for different calls, and the coordinates suffix would be in the temporary file names. This could cause confusions and bugs (for example one call to this script could delete the temporary directory that was being used by a later call). *** Library - gal_fits_key_write: to generalize, the "title" argument has been removed (because it is a keyword). It can also create the file if it doesn't exist and can optionally free the list of keywords (until now, it would always free them). - gal_fits_key_write_in_ptr: similar to 'gal_fits_key_write'. - gal_fits_img_write: no more 'program_name'; it is a keyword title; added 'freekeys' argument to optionally free the input keyword list. - gal_fits_img_write_to_type: as in 'gal_fits_img_write'. - gal_fits_img_write_corr_wcs_str: as in 'gal_fits_img_write'. - gal_fits_tab_write: as in 'gal_fits_img_write'. - GAL_STATISTICS_CLIP_MAX_CONVERGE: new name for the old 'GAL_STATISTICS_SIG_CLIP_MAX_CONVERGE'. Since we now also have MAD-clipping. - gal_statistics_clip_sigma: new name for 'gal_statistics_sigma_clip'. - gal_table_write: as in 'gal_fits_img_write'. - gal_tile_full_values_write: as in 'gal_fits_img_write'. - gal_wcs_write: as in 'gal_fits_img_write'. ** Bugs fixed - bug #52674: Correcting Clang warnings on macOS during compilation; fixed with help of Thorsten Alteholz and Raul Infante-Sainz - bug #64825: astscript-fits-view only takes the last HDU provided; reported by Teet Kuumta and fixed by Raul Infante-Sainz. - bug #64852: astscript-zeropoint: crash when input is in 0-th HDU; reported by 'danc' in https://savannah.gnu.org/support/?110952; fixed by Sepideh Eskandarlou. - bug #64915: Segment's SKY_STD output HDU does not have WCS; reported by Sepideh Eskandarlou. - bug #64952: Table error message for incorrect column name was not complete; reported by Sepideh Eskandarlou. - bug #65005: astscript-psf-unit not accounting for --innerhdu; reported by Ignacio Ruiz Cejudo. - bug #65050: Use --libtool option in the TEMPLATE usage; found and fixed by Faezeh Bidjarchian. - bug #65055: Columns of empty FITS tables (with no rows) read in reverse. - bug #65084: Fits program's --skycoverage prints connected numbers for cubes; found with help of Rahna Payyasseri Thanduparackal. - bug #65106: Necessity of using --oversample with astscript-radial-profile when a 2D profile is to be created with '--customtable' of MakeProfile; reported by Sepideh Eskandarlou. - bug #65141: MakeCatalog crash when labled image is all blank; reported by Sepideh Eskandarlou. - bug #65149: MakeCatalog '--frac-max*-*' measurements done without '--frac-max'; reported by Helena Domínguez Sánchez. * Noteworthy changes in release 0.21 (library 19.0.0) (2023-10-20) ** New features *** New program - 'astscript-pointing-simulate': a new installed script that will simplify the process of designing a pointing pattern for your observing strategy. It takes into account the distortion of the camera, runs in parallel and allows customizations (for example to account for vignetting, or large blobs of bad pixels) through hooks at relevant steps. A new tutorial has been added in the Tutorials chapter of the book on how to use this new script. The following research note has also been published for this script: https://ui.adsabs.harvard.edu/abs/2023RNAAS...7..211A *** Book - New tutorial on a low surface brightness optimized design of a pointing pattern using the newly added installed script in Gnuastro for simulating the exposure map of a pointing pattern stack (it is called as 'astscript-pointing-simulate' on the command-line). - Smaller tutorials that were distributed within the documentation of different programs are brought into the "Tutorials" chapter. - New "Standard deviation vs. error" sub-section added under the MakeCatalog section. It uses real examples to clearly show the fundamental difference between the two (which are sometimes confused with each other). This was written with the help of Raul Infante-Sainz. *** Configuration files - To separate the option name and value, you can now also use the '=' character. This allows your custom configuration files to also be loaded into a shell (with 'source') or Make (with 'include'). Until now, only white-space characters were allowed as separators of option names and values. --config-prefix=STR: Given string can be a prefix of options in configuration files. Added with the point above, this allows loading the different configuration files of different instances of the same program without overwriting them. See the example in the book. *** Arithmetic - New operators (see book for a full description): - pool-min: Min-pooling to reduce the size of the input by calculating the minimum of a the pixels within the pooling window (accounting for a stride). See the new "Pooling operators" section of the book for more. The pooling operators were all implemented by Faezeh Bidjarchian. - pool-max: Similar to 'pool-min' but using maximum. - pool-sum: Similar to 'pool-min' but using sum. - pool-mean: Similar to 'pool-min' but using mean. - pool-median: Similar to 'pool-min' but using median. - to-1d: convert the input operand into a 1D array, no matter how many dimensions it has. Suggested by Faezeh Bidjarchian. - trim: remove all fully-blank outer regions of the input dataset. - eq-b1950-to-eq-j2000: Convert input equatorial coordinates (RA and Dec) in the B1950 equinox to equatorial J2000 equinox. - eq-b1950-to-ec-b1950: Same input as above; to ecliptic B1950. - eq-b1950-to-ec-j2000: Same input as above; to ecliptic J2000. - eq-b1950-to-galactic: Same input as above; to Galactic. - eq-b1950-to-supergalactic: Same input as above; to Supergalactic. - eq-j2000-to-eq-b1950: Convert input equatorial coordinates (RA and Dec) in the J2000 equinox to equatorial B1950 equinox. - eq-j2000-to-ec-b1950: Same input as above; to ecliptic B1950. - eq-j2000-to-ec-j2000: Same input as above; to ecliptic J2000. - eq-j2000-to-galactic: Same input as above; to Galactic. - eq-j2000-to-supergalactic: Same input as above; to Supergalactic. - ec-b1950-to-eq-b1950: Convert input ecliptic coordinates in the B1950 equinox to equatorial B1950 equinox. - ec-b1950-to-eq-j2000: Same input as above; to equatorial B1950. - ec-b1950-to-ec-j2000: Same input as above; to ecliptic J2000. - ec-b1950-to-galactic: Same input as above; to Galactic. - ec-b1950-to-supergalactic: Same input as above; to Supergalactic - ec-j2000-to-eq-b1950: Convert input ecliptic coordinates in the J2000 equinox to equatorial B1950 equinox. - ec-j2000-to-eq-j2000: Same input as above; to equatorial J2000. - ec-j2000-to-ec-b1950: Same input as above; to ecliptic B1950. - ec-j2000-to-galactic: Same input as above; to Galactic. - ec-j2000-to-supergalactic: Same input as above; to Supergalactic. - galactic-to-eq-b1950: Convert input Galactic coordinates to equatorial B1950 equinox. - galactic-to-eq-j2000: Same input as above; to equatorial J2000. - galactic-to-ec-b1950: Same input as above; to ecliptic B1950. - galactic-to-ec-j2000: Same input as above; to ecliptic J2000. - galactic-to-supergalactic: Same input as above; to Supergalactic. - supergalactic-to-eq-b1950: Convert input Supergalactic coordinates to equatorial B1950 equinox. - supergalactic-to-eq-j2000: Same input as above; to equatorial J2000. - supergalactic-to-ec-b1950: Same input as above; to ecliptic B1950. - supergalactic-to-ec-j2000: Same input as above; to ecliptic J2000. - supergalactic-to-galactic: Same input as above; to Galactic. *** ConvertType - It is now possible to write TIFF files in the output (until now, Gnuastro could only read TIFF files). This step was written by Fathma Mehnoor. *** Convolve --conv-on-blank: do not ignore blank pixels in the convolution. This will effectively expand the non-blank regions of your dataset into the blank regions and only works in spatial-domain convolution. This was suggested by Raul Infante-Sainz. *** MakeCatalog - New measurements: --river-min: minimum river value around a clump. --river-max: minimum river value around a clump. *** Table --info-num-cols: print the number of the input table's columns and abort. --info-num-rows: print the number of the input table's rows and abort. - '$_all' in column arithmetic: when an arithmetic expression contains this string, it will be repeated independently for all the columns of the input table. - New operators in Table's column arithmetic: - eq-j2000-to-flat: convert the RA and Dec columns in a table to a flat-RA and flat-Dec (accounting for a reference point as well as any type of projection that is available in the FITS WCS standard). This is necessary when you are plotting points in a report or paper cover more than a couple of square degrees on the sky. Without this operator, significant distortions will appear when you plot spherical coordinates (RA,Dec) on a flat plane (your paper's plot). See the documentation for this operator in the book for more. - eq-j2000-from-flat: the inverse of 'eq-j2000-to-flat' (useful when designin dither patterns for example (where you know the final shape/distanced after flatting; and want coordinates on the sphere). *** astscript-zeropoint --mksrc: use a custom Makefile for estimating the zeropoint, not the default installed Makefile. This is primarily intended for debugging or developing this script, not for normal usage. *** Library - gal_blank_not_minmax_coords: returns the minimum/maximum coordinates of non-blank regions of input dataset. - gal_blank_trim: trim all outer blank regions from the input dataset. - gal_polygon_area_sky: area of polygon defined on celestial coordinates. - gal_pool_min: min-pooling function, see 'pool-min' above. - gal_pool_max: max-pooling function, see 'pool-min' above. - gal_pool_sum: sum-pooling function, see 'pool-min' above. - gal_pool_mean: mean-pooling function, see 'pool-min' above. - gal_pool_median: median-pooling function, see 'pool-min' above. - gal_statistics_mad: Calculate the Median Absolute Deviation (MAD). - gal_statistics_median_mad: Calculate the Median and MAD. - gal_statistics_clip_mad: Median Absolute Deviation (MAD) clipping. - gal_tiff_write: write 'gal_data_t' to a TIFF file; written by Fathma Mehnoor. - gal_wcs_projection_name_to_id: convert projection names to IDs. - gal_wcs_projection_name_from_id: convert projection IDs to their name. - gal_wcs_coordsys_convert_points: convert the input set of points from one celestial coordinate system to another. - gal_wcs_coordsys_sys1_ref_in_sys2: return the longitude reference point of the first celestial coordinate system in the second coordinate system's longitude and latitude. ** Removed features *** MakeNoise - MakeNoise (with executable name 'astmknoise') has been removed from Gnuastro. This is because the 'mknoise-*' operators of Arithmetic do the same job, but in a much more customizable, modular and manageable manner; allowing to mix with other operators. The "Noise basics" sub-section of the book has been moved under the Arithmetic program's documentation also. ** Changed features *** Arithmetic - mknoise-sigma-from-mean: new name for the old 'mknoise-poisson' random number distribution. - mknoise-poisson: now produces correct Poisson distribution output, which can be non-symmetric in low background values and is always integers. For a complete description and hands-on example of the differences with 'mknoise-sigma-from-mean' and 'mknoise-sigma', see the description of these three operators in the "Random number generators" operators sub-section of the "Arithmetic operators" section of the Gnuastro book. *** MakeCatalog - The dash in the column names of the following measurement names has been replaced by underscore to conform with the general standard of using underscores between words in column names (reported by Raul Infante-Sainz). Also, in the case of '--sigclip-mean-sb-delta' the column name didn't correspond to the option name. -------------------------------------------------------------- Option Column name Old column name -------------------------------------------------------------- --sigclip-mean SIGCLIP_MEAN SIGCLIP-MEAN --sigclip-median SIGCLIP_MEDIAN SIGCLIP-MEDIAN --sigclip-number SIGCLIP_NUMBER SIGCLIP-NUMBER --sigclip-std SIGCLIP_STD SIGCLIP-STD --sigclip-mean-sb SIGCLIP_MEAN_SB SIGCLIP-MEAN-SB --sigclip-mean-sb-delta SIGCLIP_MEAN_SB_DELTA SIGCLIP-MEAN-SB-ERR -------------------------------------------------------------- - The default extension name for the values image ('--valueshdu') is now 'INPUT-NO-SKY'. Until now, it was simply '1' (without any hint on what what is expected by default). Recall that if the sky is not subtracted from the values image you can use '--sky' for MakeCatalog to subtract it before the measurements. Implemented by Sepideh Eskandarlou. - When the input clump labels do not have the 'NUMLABS' keyword (specifying how many clumps there are in the image, independent of the number of objects), MakeCatalog will no longer re-label the input clumps: the output catalog will have the same clump ids as the labels. Until now, it would generate new labels for the clumps in such cases (along with a new FITS image containing the clumps with the new labels to allow you to match them). The output catalog also had the new label, not the original labels (the old behavior was therefore very inconvenient!). *** Segment - The default name of the first extension of Segment's output is now 'INPUT-NO-SKY', until now, it was 'INPUT'. This was necessary because the first extension of the output is not the exact input file (if a value is given to '--sky' it will be subtracted from the input in the first extension). Implemented by Sepideh Eskandarlou. *** astscript-psf-stamp - After applying the signal-to-noise threshold (if it is requested), any extra pixels that are not connected to the central target are also masked. Such pixels can remain in rivers between bright clumps and would cause problem in the final stack and were not removed until now. Implemented by Sepideh Eskandarlou *** Library - gal_convolve_spatial: new argument to allow convolution over blank elements. - gal_convolve_spatial_correct_ch_edge: new argument to allow convolution over blank elements. - gal_polygon_area_flat: new name for the old 'gal_polygon_area' function. This was necessary because this function assumes flat coordinates and can lead to wrong results when used with a polygon defined by RA,Dec for example. - gal_statistics_clip_sigma: new name for 'gal_statistics_sigma_clip', also, a new argument has been added to measure statistics that were not automatically calculated during the clipping. - gal_wcs_distortion_name_to_id: new name of gal_wcs_distortion_from_string. - gal_wcs_distortion_name_from_id: new name of gal_wcs_distortion_to_string. - gal_wcs_coordsys_name_to_id: new name of gal_wcs_coordsys_from_string. ** Bugs fixed bug #64138: Arithmetic's mknoise-poisson only using first pixel value. Reported by Irene Pintos Castro. bug #64146: SIGCLIP column names contain dashes instead of underlines. Reported by Raul Infante-Sainz. bug #64148: MakeCatalog crash when using the options '--std' and/or 'sigclip-mean-sb-delta' for clumps catalog. Reported by Raul Infante-Sainz. bug #64153: astscript-ds9-region: '--namecol' gives a constant zero in image-mode. Reported by Zahra Sharbaf. bug #64186: Query: core dump when custom name given to '--dataset'. First reported by Zohreh Ghaffari as a problem in 'astscript-select-stars' (sr #110874) and identified as a problem in Query by Sepideh Eskandarlou. bug #64199: info astscript-zeropoint not working. Reported by Zahra Sharbaf. bug #64246: Query's --information gives a crash for Gaia datasets. Reported by Rashid Yaaqib. bug #64250: astscript-fits-view: no message printed when input file did not exist. Reported by Ryan Begley. bug #64274: MakeProfiles crash for points in 3D. Found by Teet Kuumta. bug #64303: Statistics fit --fitestimate crash when covariance matrix has NaN. Reported Sepideh Eskandarlou. bug #64357: Plain text metadata ignored when no name is present. bug #64392: Arithmetic's isnotblank operator producing zeros when no blanks present. bug #64420: Error message of wrongly specified HDU doesn't give the problematic option name; reported by Aaron Watkins. bug #64431: gal_wcs_to_cd does not account for altlin==3. Reported by Colin Orion Chandler. bug #64541: Segment's --keepmaxnearriver has no effect. bug #64544: Arithmetic crashes with tofilefree operator value has a directory. bug #64545: Accounting for CFITSIO silently appending '.gz' when file doesn't exist. Reported by Sepideh Eskandarlou. bug #64610: --copykeys of Fits program finds wrong key when the first characters of the keyword names match. bug #64615: Fits program's '--pixelareaonwcs' only accurate on equator. bug #64635: astscript-psf-stamp's '--snthresh' option produces fully NaN valued image when a saturated star is in the center. Found and fixed by Sepideh Eskandarlou. bug #64643: MakeCatalog not reporting the actual '--upmaskhdu' that was used on standard output. Found by S. Zahra Hosseini Shahisavandi. * Noteworthy changes in release 0.20 (library 18.0.0) (2023-04-29) ** New features *** New program - astscript-zeropoint: this script is used for calibrating the pixel values in one image (finding its "zero point") based on a reference catalog or (any number of) reference image(s). The script allows for checking many apertures in the same command to let you choose the best; and will operate in parallel where necessary. The documentation of this script contains two tutorials on how to optimally use it and interpret its results; please read and run them once before using it. This script was written as a team effort lead by Sepideh Eskandarlou and including (in alphabetical order): Elham Saremi, Giulia Golini, Raul Infante-Sainz, Samane Raji, Zahra Sharbaf, Zohreh Ghaffari. *** Book - New tutorial on detection and spectrum extraction from 3D data has been added in the "Tutorials" chapter. It uses a MUSE cube and demonstrates how to subtract the continuum, run NoiseChisel, Segment and MakeCatalog for obtaining spectra and creating pseudo narrow-band images of emission-lines. - New "Shell tips" sub-section added (under the "Command-line" section of the "Common program behavior" chapter). This sub-section contains useful shell tips and tricks that can be useful when using Gnuastro's programs. *** Arithmetic --writeall: Write all datasets on the stack as separate HDUs in the output; this is useful in debugging incomplete Arithmetic commands. - New operators (also available in Table). - isnotblank: same as 'isblank not', but slightly more efficient. This was suggested by Sepideh Eskandarlou. - swap: swap the top two datasets on the stack of operands. - index: return dataset of same size, with pixel values that are integers starting from 0 and incrementing one by one. This operator doesn't pop the top dataset from the stack, it only adds one (to avoid extra RAM usage in most usage scenarios). - counter: similar to 'index', but integers start with 1. - indexonly: similar to 'index', but pops the top stack dataset. - counteronly: similar to 'counter', but pops the top stack dataset. - constant: replace all elements of input image or column with the given constant value (for example to set all pixels in an image to NaN or zero), or to give identifiers to each source catalog when adding rows from different catalogs, see the documentation for complete examples. - counts-to-nanomaggy: convert counts to Nanomaggy (with fixed zeropoint of 22.5, used as the pixel units of many surveys like SDSS). This was suggested by Giulia Golini. - nanomaggy-to-counts: convert nanomaggy to counts. This was suggested by Giulia Golini. - box-vertices-on-sphere: calculate the RA,Dec of vertices of the four vertices of a rectangle from its center and width along RA and Dec. This takes into account the curved nature of the coordinate system. Added after discussion with Martin Kuemmel. - number-neighbors: Return the number of non-zero neighbors of each non-zero pixel in a binary image. - New operators (only in the Arithmetic program): - interpolate-meanngb: interpolate blank values with mean of the requested number of nearest neighbors. - Alternative (shorter) names for existing operators, added after discussion with Samane Raji. - u8: same as 'uint8' (to convert to unsigned 8-bit integers). - i8: same as 'int8' (to convert to signed 8-bit integers). - u16: same as 'uint16' (to convert to unsigned 16-bit integers). - i16: same as 'int16' (to convert to signed 16-bit integers). - u32: same as 'uint32' (to convert to unsigned 32-bit integers). - i32: same as 'int32' (to convert to signed 32-bit integers). - u64: same as 'uint64' (to convert to unsigned 64-bit integers). - i64: same as 'int64' (to convert to signed 64-bit integers). - f32: same as 'float32' (to convert to 32-bit floating point). - f64: same as 'float64' (to convert to 64-bit floating point). *** CosmicCalculator - Spectral line database has been increased to 235 spectral lines in the UV and optical (previously only had 41 lines!) from: http://astronomy.nmsu.edu/drewski/tableofemissionlines.html To see the full list, run 'astcosmiccal --listlines'. --lineunit: allows you to specify the units of the displayed spectral line wavelengths. Currently, it take four units: 'm', 'micron', 'nm' and 'angstrom'. We can add any other unit easily, just let us know. *** Crop --append: if the output file already exists, append the cropped image HDU to the already existing HDUs of the file. Without this option, any existing HDUs in the output file will be removed (default behavior). --metaname: Specify the name of the cropped output HDU (value to the 'EXTNAME' keyword in FITS). *** MakeCatalog - Book: with the increasing number of possible measurements the "MakeCatalog measurements" section of the Gnuastro book has been broken into separate sub-section and contextually similar measurements are grouped separately there. --sigclip-mean-sb: surface brightness (over one pixel's area in arcsec^2) of the sigma-clipped mean of the values. This is useful in scenarios where you want to compare surface brightness values independent of the object's area, and by removing outliers (for example in a radial profile). See the description of this option in the book for more. This was added after a discussion by Ignacio Trujillo and Zahra Sharbaf. --sigclip-mean-sb-err: the error in '--sigclip-mean-sb' but using the sigma-clipped standard deviation of the values as the error, not the standard deviation image. --sigclip-std-sb: the surface brightness (over one pixel's area in arcsec^2) of the sigma-clipped standard deviation of the values. This can be used to find the reliable surface brightness of a radial profile for example. --sum-in-slice: [3D in; vector out] Sum of values in each slice. --sum-err-in-slice: [3D in; vector out] Error in '--suminslice'. --area-in-slice: [3D input; vector out] Number of labeled in each slice. --sum-proj-in-slice: [3D input; vector out] Sum of projected area in each slice. --area-proj-in-slice: [3D in; vector out] Number of voxels that are used in '--sum-proj-in-slice'. --sum-proj-err-in-slice: [3D in; vector out] Error of '--sum-proj-in-slice'. --area-other-in-slice: [3D in; vector out] Area of other label in projected area on each slice. --sum-other-in-slice: [3D in; vector out] Sum of other label in projected area on each slice. --sum-other-err-in-slice: [3D in; vector out] Area in '--sum-other-in-slice'. *** NoiseChisel --outliernumngb: the number of neighboring tiles to reject those that have passed (the mean-median quantile difference criteria) because of being on the wings of bright stars/galaxies. Until now, this number was the same as the number given to '--interpnumngb', but it is confusing and can cause problems (because interpolation is done after outlier rejection). From this version, '--interpnumngb' is only in charge of the number of neighboring tiles for interpolation outliers are defined based on the number of tiles given to this option. Therefore, if you are not detecting the wings of large galaxies, THE BEST solution is most-probably to increase '--outliernumngb'. This was done after a discussion with Elham Saremi. *** Statistics --outliernumngb: see description of same option in NoiseChisel. *** Table - Vector columns with multiple values per column are now supported. The following features have been added to help working on vector columns: - Book: a new "Vector columns" section has been added under the Table program which describes the core concepts and usage examples of the options below. --information: also identifies vector columns (a column with more than one value), by placing an '(N)' after the type. Where 'N' is the number of elements within the vector. --tovector: merge multiple single-valued columns into a new vector column. --fromvector: extract identified elements/tokens of a vector column to separate single-valued columns. --keepvectfin: do not delete the input columns to '--tovector' and '--fromvector'. --rowfirst: do row-based operations before column-based operations (by default column-based operations are done first). --transpose: column-based operator to transpose vector columns into a new table that has the inverse number of rows and columns. --txteasy: (or '-Y') when output is a plain-text file or just gets printed on standard output (terminal), all floating point columns are printed in fixed point notation (as in '123.456') instead of the default exponential notation (as in '1.23456e+02'). For 32-bit floating points, use a precision of 3 digits and for 64-bit floating points use a precision of 6 digits. This can be useful for human readability, but be careful with some scenarios (for example '1.23e-120', which will show only as '0.0'!). For more, see the changes in Table in this version. - New column arithmetic operator: - sorted-to-interval: return two columns from a single (sorted) input, containing the minimum and maximum values of an interval. The intervals are defined such that the intervals of every row fully cover the full range of values in the given column. This can be useful in scenarios where you need to build the inputs of the '--customtable' feature in MakeProfiles (to build a 2D image a custom profile). *** astscript-radial-profile --precision: sample the radial profile at precision less than one pixel. This is useful when you need to sample the profile within the central few pixels more accurately than integer pixels. See the documentation of this option in the book for a complete explanation with examples. *** Library - GAL_ARITHMETIC_OP_SWAP: swap the top two operands. - GAL_ARITHMETIC_OP_INDEX: An index (counting from 0) for every element. - GAL_ARITHMETIC_OP_INDEXONLY: Similar to 'GAL_ARITHMETIC_OP_INDEX'. - GAL_ARITHMETIC_OP_COUNTER: A counter (counting from 1) for every element. - GAL_ARITHMETIC_OP_COUNTERONLY: Similar to 'GAL_ARITHMETIC_OP_COUNTER'. - GAL_ARITHMETIC_OP_COUNTS_TO_NANOMAGGY: convert counts to nanomaggy. - GAL_ARITHMETIC_OP_NANOMAGGY_TO_COUNTS: convert nanomaggy to counts. - GAL_ARITHMETIC_OP_BOX_VERTICES_ON_SPHERE: calculate the coordinates of vertices of a rectangle on a sphere from its center and width/height. - gal_binary_number_neighbors: num. non-zero neighbors of non-zero pixels. - gal_blank_flag_not: binary dataset with 1 for those input pixels that were blank. - gal_data_alloc_empty: Allocate an empty dataset with a given number of dimensions. - gal_list_f64_to_data: convert list of float64s to a 'gal_data_t' dataset with the requested type. - gal_list_data_remove: Remove the given dataset from the given list. - gal_list_data_select_by_id: find/select a dataset from a list of datasets using an identification string (either counter or name). - gal_permutation_apply_onlydim0: When we have a 2D input, apply permutation for all the elements of each row (along dimension-0 in C). - gal_statistics_has_negative: see if input has a negative value. - gal_table_col_vector_extract: extract the given elements of a vector column into separate columns. - gal_table_cols_to_vector: merge multiple columns into a vector column. - gal_units_counts_to_nanomaggy: Convert counts to nanomaggy. - gal_units_nanomaggy_to_counts: Convert nanomaggy to counts. - gal_wcs_box_vertices_from_center: calculate the coordinates of vertices of a rectangle on a sphere from its center and width/height. ** Removed features *** MakeCatalog - Removed the short option format for the following measurements (columns). This was done because the short options affect the sorting in the output of '--help', making it hard to find the options visually. Because most measurements do not have a short format, we have seen that the short format is rarely used. Only the most commonly used measurements keep their short format, and only when they don't affect the alphabetic sorting of the output of '--help': --area (short option was 'a') --axis-ratio (short option was 'Q') --magnitude-error (short option was 'G') --num-clumps (short option was 'c') --position-angle (short option was 'p') --semi-major (short option was 'A') --semi-minor (short option was 'B') --sn (short option was 'n') --sum (short option was 'b') --upperlimit-mag (short option was 'u') ** Changed features *** Configuration --with-python: this has replaced the old '--without-python' option. The Python extension features in the Gnuastro library are no longer built by default. To have them, you need to explicitly configure Gnuastro with this option. This was necessary because Python has many peculiarities that were causing problems for users that don't need it in Gnuastro. These features are only necessary for those who compile pyGnuastro from source, most future users of pyGnuastro will be installing pre-built binaries it through services like PyPI, so they won't be needing it either. *** CosmicCalculator - Given the major increase of the spectral line database, the old line names have been changed to accommodate the full database. For example 'H-alpha' has replaced 'halpha'. Please run 'astcosmiccal --listlines' to see the new names for the previous lines. *** MakeCatalog - "Sum" used instead of "brightness" --sum: new name for the old '--brightness' column. "Brightness" has a specific meaning in astronomy/physics and has units of energy/time. However, astronomical images aren't in this unit, they are usually in counts, or can have an arbitrary units (like nanomaggy). The new name is derived from what this column does (sum of pixel values) to avoid confusions. --sum-error: new name for '--brightnesserr'. --clumps-sum: new name for '--clumpbrightness'. --sum-no-river: new name for '--brightnessnoriver'. --checkuplim: now returns the center of each label's tile, not the first pixel (the bottom-left pixel position of the tile). The old behavior to return the first pixel was un-intuitive, would require several manual steps (to measure the minimum and maximum positions of each tile, measure the width along each dimension and add it), and could lead to confusing results when ignored. See the updated documentation of this option for more on custom usage in special situations. - Using a hyphen to help in readability and usability: Old name New name ======== ======== --areaarcsec2 --area-arcsec2 --areamaxv --area-max-val --areaminv --area-min-val --areaxy --area-xy --axisratio --axis-ratio --clumpsarea --clumps-area --clumpsgeow1 --clumps-geo-w1 --clumpsgeow2 --clumps-geo-w2 --clumpsgeow3 --clumps-geo-w3 --clumpsgeox --clumps-geo-x --clumpsgeoy --clumps-geo-y --clumpsgeoz --clumps-geo-z --clumpsmagnitude --clumps-magnitude --clumpsw1 --clumps-w1 --clumpsw2 --clumps-w2 --clumpsw3 --clumps-w3 --clumpsx --clumps-x --clumpsy --clumps-y --clumpsz --clumps-z --fracmax --frac-max --fracmaxarea1 --frac-max1-area --fracmaxarea2 --frac-max2-area --fracmaxradius1 --frac-max1-radius --fracmaxradius2 --frac-max2-radius --fracmaxsum1 --frac-max1-sum --fracmaxsum2 --frac-max2-sum --geoarea --geo-area --geoareaxy --geo-area-xy --geoaxisratio --geo-axis-ratio --geopositionangle --geo-position-angle --geosemimajor --geo-semi-major --geosemiminor --geo-semi-minor --geow1 --geo-w1 --geow2 --geo-w2 --geow3 --geo-w3 --geox --geo-x --geoy --geo-y --geoz --geo-z --halfmaxarea --half-max-area --halfmaxradius --half-max-radius --halfmaxsb --half-max-sb --halfmaxsum --half-max-sum --halfsumarea --half-sum-area --halfsumradius --half-sum-radius --halfsumsb --half-sum-sb --hostobjid --host-obj-id --idinhostobj --id-in-host-obj --magnitudeerr --magnitude-error --maxvx --max-val-x --maxvy --max-val-y --maxvz --max-val-z --maxx --max-x --maxy --max-y --maxz --max-z --minvx --min-val-x --minvy --min-val-y --minvz --min-val-z --minx --min-x --miny --min-y --minz --min-z --numclumps --num-clumps --objid --obj-id --positionangle --position-angle --riverave --river-mean --rivernum --river-number --sberror --sb-error --semimajor --semi-major --semiminor --semi-minor --sigclip-mean-sb-err --sigclip-mean-sb-delta --skystd --sky-std --surfacebrightness --sb --upperlimitmag --upperlimit-mag --upperlimitonesigma --upperlimit-onesigma --upperlimitquantile --upperlimit-quantile --upperlimitsb --upperlimit-sb --upperlimitsigma --upperlimit-sigma --upperlimitskew --upperlimit-skew --weightarea --weight-area *** MakeNoise --bgnotmag: new name for the old '--bgisbrightness' option. See the description of changed '--sum' in MakeCatalog (above) for more. *** MakeProfiles --mcolissum: new name for the old '--mcolisbrightness' option. See the description of changed '--sum' in MakeCatalog (above) for more. - Units of the output FITS image (value to 'BUNIT' keyword) is now "counts", until now, it was "brightness". See the description of changed '--sum' in MakeCatalog (above) for more. *** Table - Given the newly added vector columns, the precedence on operations has been updated. Please read the "Operation precedence in Table" section of the book. - To avoid potential loss of information in floating point columns, when printing the columns to standard output (in the terminal) or saving in a plain-text file, the default floating point formats are now printed in exponential (scientific) notation (e.g., 1.234e-5). Until now the default notation was fixed-point notation (e.g., 0.00001234). A new sub-section called "Printing floating point numbers" has been added to the book (under the "Table" section) to clarify this important point. -A: new short format for --txtf64format. The '-d' short format was conflicting with the short option name for '--descending'. *** astscript-psf-select-stars - Now uses the Gaia DR3 dataset by default (until now it was using eDR3). *** astscript-psf-stamp *** astscript-psf-unite *** astscript-radial-profile --axis-ratio: new name for old '--axisratio' --position-angle: new name for old '--positionangle' *** Library: - gal_blank_remove_rows: new 'onlydim0' argument to ignore vector columns when checking for blanks. - gal_list_str_cat: new 'delimiter' argument to specify the character for printing between the separate string nodes in the single output string. - gal_txt_write: new 'tab0_img1' argument. Until now, this function would distinguish between images and tables using the dimensions of the input. But with the addition of vector columns in tables (that have 2 dimensions) this argument becomes necessary. ** Bugs fixed bug #63266: Table ignores a value of 0 given to '--txtf32precision' or '--txtf32precision=0' (happens when floating point columns need to be rounded to integers). Reported by Sepideh Eskandarlou. bug #63270: Table aborts when called with '--descending' with complaint on '--txtf64format' (that was added in Gnuastro 0.19). Reported by Giulia Golini. bug #63285: Table not printing anything in rows of integer columns with value of zero. bug #63340: Statistics printing zero for values smaller than 1e-6 and 1e-14 for 32-bit or 64-bit floats respectively. Reported by Elham Saremi. bug #63345: WCS coordinate change not accounting for new equinox (only relevant for equatorial outputs). Reported by Alejandro Serrano Borlaug. bug #63399: Table's '--noblank' or '--noblankend' crashing when input has no rows. Reported by Elham Saremi. bug #63485: Oversampling in radial profile has incorrect center. Reported by Nafiseh Sedighi. bug #63543: Programs delete input when it has same name as output. Reported by Faezeh Bidjarchian. bug #63969: make distcheck error due to temporary files of astscript-psf-select-stars. Reported by Siyang He. bug #64076: MakeCatalog prints --upnum warning even with --quiet. Reported by Teet Kuutma. * Noteworthy changes in release 0.19 (library 17.0.0) (2022-10-24) ** New features *** Book - Two new sections added to the "General program usage tutorial" for describing how to prepare a FITS image for high quality publication (in PDF), and drawing vector graphics marks from a catalog over it (for example to show your selected galaxies in the field of view). The use the newly added features of ConvertType. *** Arithmetic - Added new type operators and physical constants. All are also available in Table's column arithmetic also). - e: Base of the natural logarithm (no units). - pi: Fraction of Circle cirumference to diameter (no units). - c: speed of light in vaccume (in units of m/s). - G: Gravitational constant (in units of m^3/kg/s^2). - h: Plank's constant (in units of J/Hz). - au: Astronomical Units (in units of meters). - ly: Light years (in units of meters). - avogadro: Avogadro's constant (in units of 1/mol). - fine-structure: Fine structure constant (no units). - counts-to-sb: convert counts to surface brightness (mag/arcsec^2). - sb-to-counts: convert surface brightness (mag/arcsec^2) to counts. - mag-to-sb: convert magnitudes to surface brightness over an area. - sb-to-mag: convert surface brightness to magnitudes over an area. - New operators that are specific to Arithmetic: - collapse-median: collapse input dataset by calculating the median along the given dimension. - collapse-sigclip-std: Collapse with sigma-clipped standard deviation. - collapse-sigclip-mean: Collapse with sigma-clipped mean. - collapse-sigclip-median: Collapse with sigma-clipped median. - collapse-sigclip-number: Collapse with number remaining after sigma-clip. *** ConvertType - It is now possible to draw vector graphics marks from a catalog over the output PDF images. The following options have been added to ConvertType for doing this. See the "General program usage tutorial" for a fully working example. --marks: name of table containing mark information. --markshdu: HDU of table if file given to '--marks' is FITS. --markcoords: name or number of two columns containing coordinates. --mode: if the coordinates are in 'img' (image) or 'wcs' (RA/Dec). --markshape: name or number of column containing the shape of each mark. --markrotate: name or number of column containing rotation of each mark. --marksize: name or number of column containing the size of the mark. --sizeinpix: interpret the values in the size column as pixels. --sizeinarcsec: interpret the values in the size column as arc-seconds. --sizeinarcmin: interpret the values in the size column as arc-minutes. --marklinewidth: name or number of column containing mark's line width. --markcolor: name or number of column containing mark's color. --listcolors: List all the 140 available colors, and show the colors on 24-bit (true color) terminal. --marktext: name or number of column containing text under each mark. --marktextprecision: number of decimals to print as text when the text column (given to '--marktext') is floating point. --markfont: name or number of column containing the font to use for the the mark text (given to '--marktext'). --markfontsize: name or number of column containing the size of the font to use for the mark text (given to '--marktext'). --showfonts: build a demo PDF with one page per font to show the various available fonts on the system. --listfonts: List the names of the available fonts on the terminal. - It is now possible to select the color of the border of images produced in vector graphics outputs (EPS and PDF) with '--bordercolor'. *** Fits --pixelareaarcsec2: print the image pixel area in units of arcsec^2 to standard output. Among other things, this is useful in creating a surface brightness image using the new 'counts-to-sb' operator of Arithmetic. --pixelareaonwcs: Ouput an image with the same number of pixels as the input. But each pixel's value shows its area on the sky (in degrees-squared). This area is calculated after accounting for distortion, projection or rotation. Implemented by Pedram Ashofteh-Ardakani. --edgesampling: extra sampling along each pixel's edge used to configure the output of '--pixelareaonwcs', similar to Warp. *** Statistics - Linear and Polynomial least squares fitting are now available and very easy to call on the command-line. They are wrappers over the respective least squares fitting functions of the GNU Scientific Library. The interface is pretty simple, like the example below: aststatistics table.fits -cX,Y,Yerr --fit=linear-weighted It is also possible to estimate values and errors of the fitted model on a new X axis. A complete example has been added to the newly added "Least squares fitting" section of the book (under the Statistics program documentation). Please see that tutorial to easily use this feature. The following new options have been added to the Statistics program for this purpose: --fit: the model to use. Currently the following models are supported: linear linear-weighted linear-no-constant linear-no-constant-weighted polynomial polynomial-weighted polynomial-robust --fitweight: nature of the "weight" column (default: standard dev). --fitmaxpower: maximum power of X in polynomial models. --fitrobust: weight function to use in the "robust" polynomial model. --fitestimate: File name, or number to estimate the fit on. --fitestimatehdu: HDU containing table in file given to '--fitestimate'. --fitestimatecol: Column containing X axis values for '--fitestimate'. *** Table - It is now possible to customize the format of floating point numbers in the plain-text outputs: when output is printed on the standard output (command-line) or in plain-text files. The following new options have been added for this new feature: --txtf32format (or '-f'): Format of 32-bit floating point columns. This can be either 'fixed' (for fixed-point notation) or 'exp' (for exponential/scientific notation). --txtf32precision (or '-p'): number of digits following the decimal-point of 32-bit floating point columns. --txtf64format (or '-d'): Format of 64-bit floating point columns. This can be either 'fixed' (for fixed-point notation) or 'exp' (for exponential/scientific notation). --txtf32precision (or '-B'): number of digits following the decimal-point of 32-bit floating point columns. *** Warp - Can correct distortions (with any standard recognized by WCSLIB) and simultaneously align the image to the coordinate system. When no named linear operation (like '--rotate', '--scale' or etc) is requested, Warp will go into this mode. It is highly customizable through the following options. See the "Invoking Warp" section of the book for more. This feature has been written by Pedram Ashofteh-Ardakani. --center: RA, DEC of the center of the central pixel of output. --width: Width of output in degrees or pixels (see '--widthinpix'). --widthinpix: interpret values of '--width' as pixels. --cdelt: Pixel scale of output ('CDELTi' keywords in FITS). --ctype: Coordinates and projection algorithm. Default: RA/Dec and Gnomonic or 'TAN'). --edgesampling: extra sampling of pixel polygon to account for strong non-linear projection or distortion effects, when necessary. --gridfile: warp the input to the exact WCS and pixel grid of the file given to this option. This is very useful when matching images from differetn surveys. Using this option, you can also insert distortions in an image (for example on a mock image, to make it match an observed exposure with dithering+distortion). --gridhdu: HDU containing image to be matched in '--gridfile'. --checkmaxfrac: visualize the Moiré pattern of the warp in the second extension of the output. This is the maximum fraction of a single input pixel's area in the output pixel. When the output pixel scale is similar to the input, the Moiré pattern can cause varying artificial smoothing of the noise level. See the newly added "Moiré pattern and its correction" section of the book for more on its basics and how to reduce it in your outputs. *** astscript-fits-view --ds9colorbarmulti: show a separate color-bar for each image in DS9. By default this script will show a single color-bar for all the images to help save space on the monitor when there are many images. *** astscript-psf-stamp - sub-pixel warping is applied to ensure that your coordinate is at the center of the central pixel of the output image. This results in a _major_ improvement when estimating the center of the PSF. --nocentering: disable sub-pixel warping when creating the PSF stamp. As described above, the sub-pixel warping is critical for the central part of the PSF, but for the outer parts it is statistically negligible. So to avoid slowing down you pipeline, you can disable sub-pixel warping with this option. --snthresh: if given, the value to this option is assumed to be a signal-to-noise ratio (S/N) threshold and all pixels below that S/N will be masked. This is useful because we are often forced to stack stars of differing magnitudes. While the fainter ones are good for the inner parts of the star, they degrade the stack's outer parts. With this option, the fainter stars won't harm the outer parts. *** GNU Make extensions (in a Makefile) It is now possible to use custom Gnuastro functions in GNU Make, using its extension facilities with Dynamic libraries. GNU Make is a very powerful workflow manager that is also used for data analysis (not just for compilation). With the Gnaustro Make functions, (astronomical) data analysis becomes even more easier and faster. In the following, you can see the first set of such functions (they all begin with 'ast-'). For more, see the newly added chapter in the Gnuastro manual. - ast-version-is: will return '1' if the running Gnuastro has the given version (argument of this function). This can be used to ensure reproducibility in combination with Make's conditional features, see the minimal working example in the manual. - ast-text-contains: will return space-separated words within a larger list that contain a certain string. The to-contain string can be anywhere within the words of the larger list. - ast-text-not-contains: will return space-separated words within a larger list that DO NOT contain a certain string. The to-not-contain string can be anywhere within the words of the larger list. - ast-fits-with-keyvalue: takes a keyword name, a list of keyword values, a HDU and a list of FITS files. It will return only those FITS files that have the requested value(s) in the requested keyword of the requested HDU. - ast-fits-unique-keyvalues: takes a keyword name, a HDU and a list of FITS files. It will return all the unique values given to that keyword within the FITS files. *** Library: - GAL_CONFIG_HAVE_PYTHON: non-zero if Python3+Numpy features included. - GAL_CONFIG_HAVE_GNUMAKE_H: non-zero if GNU Make extensions can be made. - gal_box_border_rotate_around_center: width of box after rotation. - gal_color_id_to_name: return the name of a color from its ID. - gal_color_in_rgb: return the fraction of red-green-blue in a color. - gal_color_name_to_id: return the ID of a color from its name. - gal_dimension_collapse_median: collapse input along dim. using median. - gal_dimension_collapse_sigclip_mean: collapse with sig-clipped mean. - gal_dimension_collapse_sigclip_std: collapse with sig-clipped STD. - gal_dimension_collapse_sigclip_median: collapse with sig-clipped median. - gal_dimension_collapse_sigclip_number: collapse with num. after sig-clip. - gal_eps_shape_id_to_name: return the name of a shape from its ID. - gal_eps_shape_name_to_id: return the ID of a shape from its name. - gal_fit_name_to_id: Convert string name to ID of fitting model. - gal_fit_name_from_id: Convert ID of fitting model to string name. - gal_fit_name_robust_to_id: Convert name of robust weights to ID. - gal_fit_name_robust_from_id: Convert ID of robust weights to name. - gal_fit_1d_linear: linear fit of input columns. - gal_fit_1d_linear_no_constant: linear fit with no constant. - gal_fit_1d_linear_estimate: estimate a linear fit on a new X column. - gal_fit_1d_polynomial: polynomial fit of input columns. - gal_fit_1d_polynomial_robust: robust polynomial fit of input columns. - gal_fit_1d_polynomial_estimate: estimate a polynomial fit on new X column. - gal_fits_unique_keyvalues: extract all unique values to a certain keyword in many files. - gal_fits_with_keyvalue: select FITS image with a certain key value. - gal_list_data_select_by_name: select a dataset from a list by its name. - gal_list_str_cat: Concatenate (append) list to a space-separated string. - gal_list_str_extract: Extract space-separated tokens to a list. - gal_python_type_from_numpy: Convert Numpy's type id. to Gnuastro's. - gal_python_type_to_numpy: Convert Gnuastro's type id. to Numpy's. - gal_txt_contains_string: Check a certain string within in a larger one. - gal_units_counts_to_sb: SB from counts, zeropoint and area. - gal_units_mag_to_sb: surface brightness (SB) from magnitude and area. - gal_units_sb_to_counts: counts from SB, zeropoint and area. - gal_units_sb_to_mag: magnitude from SB and area. - gal_warp_pixelarea: return image of same size, but with area on sky. - gal_warp_wcsalign_init: initialize the WCS aligning structure. - gal_warp_wcsalign_onpix: Per-pixel filling of output. - gal_warp_wcsalign_onthread: function to give to pthreads. - gal_warp_wcsalign: high-level function to align input by its WCS. - gal_warp_wcsalign_free: free the contents of the WCS aligning structure. - gal_wcs_free: free a WCS structure that is created or read by Gnuastro. ** Removed features *** Statistics --refcol has been removed because it breaks the modularity principle (given that it is the job of Gnuastro's Table program to limit rows from a larger table based on many different criteria). The output of Table can be directly piped to Statistics to achieve the same (and much more feature-rich effect). *** Warp --align: has been removed. This is because aligning an image (while correcting for any possible distortion) is now the default behavior of Warp (when no linear operations have been requested). ** Changed features *** Book - The "General program usage tutorial" section is now the first section of the Tutorial chapter, since it introduces the tools at a more basic level. The "Sufi simulates a detection" (which was previously first) has been moved to the fourth section. *** Warp - The short format of the '--centeroncorner' option has been removed. The '-c' is now the short format for the new '--center' option to Warp. *** astscript-psf-scale-factor --widthinpix: new name for the old '--stampwidth' option. This was done to have the same name to a similar option in Crop and help in remembering. *** astscript-psf-stamp --widthinpix: new name for the old '--stampwidth' option. This was done to have the same name to a similar option in Crop and help in remembering. *** Library - gal_eps_write: two new arguments have been added to draw marks, and to set the border color. - gal_pdf_write: similar to 'gal_eps_write'. - gal_fits_hdu_open: new argument to optionally exit program if HDU couldn't be opened. - GAL_TABLE_DISPLAY_FMT_FIXED: new name for GAL_TABLE_DISPLAY_FMT_FLOAT since it corresponds to the fixed-point notation of printing floating points in plain-text (the '_FLOAT' suffix was too generic and unclear). ** Bugs fixed bug #62861: '--printkeynames' of Fits program gets caught in an infinite loop on FITS files that have empty keywords before 'END'. Found by Pedram Ashofteh-Ardakani. bug #62892: Installed scripts don't account for differing LANG and LC_NUMERIC. Found by Teet Kuutma and fixed by Raul Infante-Sainz. bug #62937: psf-scale-factor not being quiet, when requested. Found and fixed by Sepideh Eskandarlou. bug #62943: Couldn't read the value of width with '--snthresh' is called. Found and fixed by Sepideh Eskandarlou. bug #62944: No warning when the option value isn't immediately after the equal sign in long format. Found by Faezeh Bijarchian. bug #63013: Sigma clip segmentation fault when input has an integer type with values close to saturation-level. bug #63022: psf-scale-factor not saving the result in the output file, found and fixed by Raul Infante-Sainz. bug #63189: MakeProfiles custom profiles become NaN with a single row being NaN, reported by Nafise Sedighi. bug #63207: Match crashes when one input has no rows. Found by Sepideh Eskandarlou. bug #63257: Fits program's '--skycoverage' gives unreasonable outputs when image crosses the RA=0 hour circle. Found by Irene Pintos Castro. bug #63261: Radial profile script ignores central pixel in azimuthal profiles. Found and fixed by Sepideh Eskandarlou. * Noteworthy changes in release 0.18 (library 16.0.0) (2022-07-21) ** New features *** Book - New "Measuring the dataset limits" section has been added in second tutorial. Using the catalogs produced during the analysis, it shows how to derive a dataset's magnitude limit, completeness limit (crudely: without simulations), and surface brightness limit. All are important measures in any scientific analysis! This section was written with the help of S. Zahra Hosseini Shahisavandi and Sepideh Eskandarlou. *** Arithmetic - New operators (also available in Table's column arithmetic): - stitch: connect any number of input images along the given dimension (for example to stitch the images of separate amplifiers of a CCD into one image). - noblank: Remove blank elements from the input and return a 1D output (irrespective of how many dimensions the input had). - jy-to-counts: convert Janskys to counts using an AB-magnitude-based Zeropoint (the inverse operator 'counts-to-jy' already existed). - mag-to-jy: convert AB magnitudes to Janskys. - jy-to-mag: convert Janskys to AB magnitudes. - load-col-%-from-%-hdu-%: Load a certain column into the operands stack. The name or number of the column is the first '%'. The file name hosting that column is the second '%', and HDU (if the file is a FITS file), is identified by third '%'. This can be used to load columns outside the main table in Table's column arithmetic or by operators that need columns as input parameters (like the new 'random-from-hist' operator). - random-from-hist-raw: return a set of random points based on a custom distribution defined by its histogram. See the description and complete examples in the book for more. - random-from-hist: Similar to 'random-from-hist-raw', but within each bin, the final returned value is itself the result of a uniform sampling (to make the final distribution less descrete). See the example in the book for selecting random star magnitudes based on Gaia's observed magnitude distribution. - New options to add metadata (standard FITS keywords) to your output for easy understanding of the data in the future, or by your colleagues if you share the output). Until now, to have this feature, you needed to call Fits on the output of Arithmetic (for setting the HDU name to "my-sum" for example): astarithmetic img-a.fits img-b.fits + --output=sum.fits astfits sum.fits --write=EXTNAME,"my-sum" But with the following commands, you can add important metadata to the output of Arithmetic in the same command and simplify your scripts: --metaname: Metadata name of the output (FITS keyword: 'EXTNAME'). --metaunit: Unit of the output's pixels (FITS keyword: 'BUNIT'). --metacomment: Description of the data (FITS keyword: 'COMMENT'). Like the following for the example above: astarithmetic img-a.fits img-b.fits + --metaname="my-sum" \ --output=sum.fits *** Crop --oneelemstdout: when a crop has a single pixel and this option is called, the single pixel's value will be printed on the standard output instead of creating a FITS file. This option can be useful in labeled images like detection maps (when you want to see what label a certain coordiante/pixel corresponds to). This option was suggested by Raul Infante-Sainz. *** Fits - When no option is specified (thus printing the HDU metadata in the given file), the units of the HDUs are also printed as a new column. For tables, it will be blank ('n/a'); because tables have separate units on each column (use 'asttable table.fits -i'). --copykeys: can now take any number of keyword names also. For example '--copykeys=KEYNAME1,KEYNAME2,KEYNAME3'. Until now, it was only possible to give this option a range of keyword positions (for example '--copykeys=10:13' to copy the 10th to 13th keywords (inclusive). But now, you can also select non-contiguous set of keywords to copy from one HDU into another HDU (possibly in another file). This was suggested by Juan Antonio Fernández Ontiveros and implemented with the help of Jash Shah. *** MakeProfiles - It is now possible to insert a custom image as a new (10th) 'custom-img' profile. This is useful when you have no-noise FITS images from numerical simulations for example, and you want to insert those at a certain location within a larger image (while optionally adding existing Sersic, point or other profiles in the same command). The magnitude of the inserted image can be set in the magnitude column of the input catalog. This can be disabled (to use the original image pixel values) using the '--mcolnocustimg' option. --mcolnocustprof: new option to disable scaling of custom radial profiles to required magnitude when building a 2D image from them. *** Query - Short name for Gaia's DR3 dataset has been added for both ESA's Gaia database and VizieR, for example with the following commands: astquery gaia --dataset=dr3 --center=1.234,5.678 --radius=0.01 astquery vizier --dataset=gaiadr3 --center=1.234,5.678 --radius=0.01 *** astscript-radial-profile --zeroisnotblank: new option to account for zero-valued pixels in the profile measurement. By default, during the internal cropping phase, zero-valued pixels are set to blank (and thus not considered during the profile measurement). When this option is called, it is directly passed to the internal cropping command ('astcrop' has an option with the same name). This was suggested by Ignacio Ruiz Cejudo and implemented by Sepideh Eskandarlou. *** Library - GAL_ARITHMETIC_OP_UNIQUE: to find unique elements in dataset. - GAL_ARITHMETIC_OP_NOBLANK: to remove blank elements from dataset. - GAL_ARITHMETIC_OP_STITCH: to stitch multiple datasets. - GAL_ARITHMETIC_OP_RANDOM_FROM_HIST: the new 'random-from-hist' operator. - GAL_ARITHMETIC_OP_RANDOM_FROM_HIST_RAW: the new 'random-from-hist-raw' op. - gal_arithmetic_load_col: Low-level function for parsing the string of the newly added 'load-col' operator, mentioned in the Arithmetic section above. - gal_statistics_std_from_sums: return the standard deviation using the sum, sum of squares and number of a distribution. ** Removed features ** Changed features *** MakeCatalog --std: measures the standard deviation of the object/clump pixels of the values image. Until now, there was no column to calculate this, and the '--std' column returned the _sky_ standard deviation over the object/clump labels (from the standard deviation image, ignoring the values image). But that was confusing, especially since we have columns like '--mean' or '--median' which only use the values image (see bug #62685). --skystd: new name for old '--std', see description of '--std' above. --sfmagnsigma: default value changed to 3 (more commonly used). --sfmagarea: default value set to 100 arcsec^2 (more commonly used). --upnum: MakeCatalog won't abort when '--upnum' is too small (less than 20). '--upnum' defines the number of random positions for upper-limit measurements, so when its too small, the statistical precision of the result will badly affected and MakeCatalog would abort with an error message. However, in some scenarios (for example when the positions of the random apertures from '--checkuplim' are needed) the actual upper-limit measurements aren't relevant! So from this version, instead of aborting MakeCatalog, it prints a warning and continues. This was suggested by S. Zahra Hosseini Shahisavandi and Ignacio Trujillo. *** MakeProfiles - The string identifier for custom radial profiles is now called 'custom-prof' (until now, it was called 'custom'). This was necessary because of the new custom image feature that has been added. The code for this custom radial profile function (8) hasn't changed. - After constructing a custom radial profile (the 'custom-prof' profile), MakeProfiles will scale it to have the desired total magnitude in the input catalog of all profiles. Until now, the 2D images created from a custom radial profile wouldn't be touched and the pixel values would exactly come from the custom radial profile (ignoring the magnitude column of the input catalog). To have the old behavior, use the new '--mcolnocustprof' option. This new functionality was proposed by Elham Saremi. *** Table --catcolumnfile: the '-N' suffix (where 'N' is a counter for the files with concatenated, or appended, column) is only added if a column with a identical column name (not case sensitive) existed in the table. Until now, this suffix would be added to all the newly concatenated columns (recall that the suffix addition can be disabled in general with the '--catcolumnrawname' option). --rowrange: new name for the old '--rowlimit'. This option takes the range of desired rows based on their position (for example '--rowrange=2,5' will only print rows 2, 3, 4 and 5 of the table). But the old name was very confusing and would not directly convey this behavior. *** astscript-radial-profile --azimuth: if the first azimuthal angle is larger than the second, the outer region between the two angles will be used. For example '--azimuth=80,40' will use the full azimuthal range except for the region with azimuthal angles between 40 and 80 degrees. This can be useful to ignore parts of the profile. See the example in the description of this option on how to visually check the pixels used for the profile. ** Bugs fixed bug #62216: MakeProfiles crash when a 3D cube is requested and input catalog is from a pipe. Found by Irene Pintos Castro. bug #62220: astscript-psf-stamp crashes when coordinate not in image. Found and fixed by Sepideh Eskandarlou. bug #62250: Convolve crashes when doing PSF matching. Found with help of Raul Infante-Sainz. bug #62252: Radial profile giving one larger radius in high axis-ratio profile. Found and fixed by Sepideh Eskandarlou. bug #62253: Arithmetic: segmentation fault when invalid input given for 'ra-to-degree' or 'deg-to-degree' operators. Found by Pedram Ashofteh-Ardakani. bug #62305: Not reading the 'MemFree' keyword in '/proc/meminfo' to find the available RAM (and thus printing an annoying warning). Reported by Juan Miro. bug #62309: MakeCatalog segmentation fault when clumps image given as main input (objects should be given). Reported by Sepideh Eskandarlou. bug #62429: Segment prints one line status-progress line when called with '--quiet'. Reported by Teet Kuutma. bug #62548: Table doesn't recognize plain-text string column when defined width is larger and it is the last column. Found with the help of Sepideh Eskandarlou. bug #62563: Some checks of installed scripts get skipped on macOS. Found and fixed by Raul Infante-Sainz. bug #62564: Arithmetic not parsing numbers in scientific notation (for example '1e5'). Found by Elham Saremi. bug #62590: Failure to build on macOS when building in debugging mode ('./configure --enable-debug'). Found by Raul Infante-Sainz. bug #62597: Arithmetic not writing single-valued output into a file when called with '--output'. Reported by Raul Infante-Sainz. bug #62636: Table ignoring '--range' when '--head' is also called. Reported by Sepideh Eskandarlou. bug #62662: Table crashes when column arithmetic is applied after adding rows from another file (with '--catrowfile'). bug #62674: Column arithmetic can't use columns from --catcolumnfile. bug #62678: Radial profile script using values image instead of STD image when '--instd' not used. Found and fixed by Raul Infante-Sainz. bug #62679: MakeProfiles kernel only normalized when zeropoint is zero. Found by Raul Infante-Sainz. bug #62680: MakeProfiles using WCS-options when --background is given and the background doesn't have WCS. Found with the help of Raul Infante-Sainz. bug #62683: Arithmetic's 'set-' operator not working after the 'makenew' operator. Found by Raul Infante-Sainz. bug #62685: MakeCatalog's '--std' option is wrongly interpreted as the standard deviation of the values image, while it is actually the root mean square of the Sky standard deviation pixels. Found by Raul Infante-Sainz. bug #62694: Radial profile script overestimates the surface brightness error. Found and fixed with the help of Raul Infante-Sainz. bug #62710: Plain-text integers starting with 0 are read in the octal base (which is common in software engineering, but not in data analysis). Found by Manuel Sánchez-Benavente. bug #62718: Table goes into an infinite loop on some old ASCII FITS tables. Found by Hilderic Browne. bug #62720: Table not reading string columns that are shorter than the defined width in the metadata. bug #62721: Radial profile script can't deal with angles around azimuth zero (for example '--azimuth=355,5' to get only the azimuthal range of 10 degrees around the major axis). Found and fixed by Samane Raji. bug #62784: Conversion of sexagesimal RA or Dec ignores sign when first digit is zero (for example '-00d12m34'). Found by Manuel Sánchez-Benavente. bug #62794: psf-stamp scripts produces inconsistant outputs on failure. Found by Sepideh Eskandarlou and Nafise Sedighi. * Noteworthy changes in release 0.17 (library 15.0.0) (2022-03-20) ** New features New programs: - astscript-fits-view: Given any number of FITS files, this script will either open SAO DS9 (for 2D images or 3D cubes) or TOPCAT (for tables) to visualize their contents in a graphic user interface (GUI). Both SAO DS9 and TOPCAT are free software packages that are heavily used in astrophysics. When opening in DS9, files will be opened as "match"ed and "lock"ed frames, enabling easy visual comparison. For GUIs that conform to the freedesktop.org standards, a '.desktop' file is also provided that you can easily activate (as described in the manual). Once activated, simply clicking on FITS file(s) in your graphic environment will open DS9 or TOPCAT depending on their contents. This script was suggested and partly written by Sepideh Eskandarlou. - A set of installed scripts to enable easy estimation and subtraction of the extended PSF (in a highly modular and easily scalable manner). They are primarily based on the method described in Infante-Sainz et al. 2020 (https://ui.adsabs.harvard.edu/abs/2020MNRAS.491.5317I) and primarily maintained by Raul Infante-Sainz himself and the help of Sepideh Eskandarlou. A full tutorial on real data from the J-PLUS survey (http://j-plus.es) has also been added to the tutorials chapter to describe in full detail how to run these scripts in a generic way (on any survey). These installed scripts all have a 'astscript-psf-*' prefix in their names: - astscript-psf-select-stars: Find all the stars within an image that are suitable for constructing an extended PSF. If the image has WCS, this script can automatically query Gaia to find the good stars. - astscript-psf-stamp: build a crop (stamp) of a certain width around a star at a certain coordinate in a larger image. This script will do sub-pixel re-positioning to make sure the star is centered and can optionally mask all other background sources). - astscript-psf-scale-factor: Given a PSF model, and the central coordinates of a star in an image, find the scale factor that has to be multiplied by the PSF to scale it to that star. - astscript-psf-unite: Unite the various components of a PSF into one. Because of saturation and non-linearity, to get a good estimate of the extended PSF, its necessary to construct various parts from different magnitude ranges. - astscript-psf-subtract: Given the model of a PSF and the central coordinates of a star in the image, do sub-pixel re-positioning of the PSF, scale it to the star and subtract it from the image. Book: - New tutorial on how to extract the extended Point Spread Function (PSF) from the actual science data and how to subtract it. Written with the help of Raul Infante-Sainz and Sepideh Eskandarlou. - New section called "Skewness cased by signal and its measurement" added to the third tutorial (Detecting large extended targets). It describes the significant advantages of using the quantile of the mean as a measure of skewness and why its a very important factor when understanding how NoiseChisel works. It was originally written by Sepideh Eskandarlou, with edits by Elham Saremi and Pedram Ashofteh-Ardakani. - New section called "Integer benefits and pitfalls" added under the Arithmetic program's documentation. It describes the running time, storage and RAM consumption benefits if you use integers (where possible), and the issues/solutions you may confront when doing so. All programs: - Coordinate-related columns in all programs now also accept sexagesimal values, not just degrees. Therefore, '--center=113.8729761,31.9027152' and '--center=07h35m29.51,31d54m9.77' are now equivalent in the Crop or Query programs that have this option! Other improved options include the '--polygon' options of Table and Crop (that also take region files produced by DS9), the '--coord' option of Match, the '--crval' option of MakeProfiles. - Columns of FITS tables are now read in parallel (when '--numthreads' isn't set to '1', and if the dependency CFITSIO library was built with '--enable-reentrant'). In scenarios where many columns may be necessary (for example the Table or Match programs), this will greatly improve the running time of the programs. This task was implemented based on feedback from Andrés Del Pino Molina. Arithmetic: - New operators: - add-dimension-fast: Add a new dataset along the "fastest" dimension of the first dataset (in a FITS image, the "fast" dimension is the horizontal one). For example if you have N one-dimensional datasets with M elements, you can use this operator to construct a 2-dimensional FITS image that has N pixels along the horizontal and M pixels along the vertical. - repeat: Put a copy of the given dataset onto the top of the stack of operators for the requested number of times. Crop: --widthinpix: when in WCS-mode, the value given to '--width' will be interpreted in units of pixels. This will allow having crops of a fixed pixel size, even when the mode of the central coordinate is in WCS. This was suggested by Jesús Varela. Fits: - If a 'CHECKSUM' keyword exists in a HDU and any of the keyword modification keywords are called, Fits will automatically update the checksum after all the changes are done. This is important to keep the checksum valid after the change (recall that you can verify the FITS checksums with the '--verify' option to Fits). This task was suggested by Tamara Civera Lorenzo. Table: - Options that accept strings ('--colmetadata', '--equal' and '--notequal') can now accept a comma within the string: to avoid confusing the comma with a separator of values, you should put a '\' before it. For example '--equal=AB,cd\,ef' will select all rows where the 'AB' column has a value of 'cd,ef'. This task was suggested by Zohreh Ghaffari. --catrowfile: File to concatenate (i.e., add or append) rows into the main input table. With this option, you can add the rows of another table into the final output. This option can be called multiple times, allowing you to merge rows of any number of tables. This job is done after concatenating columns, but before any of the row-selection operations. This was suggested by Raúl Infante-Sainz and Manuel Sánchez Benavente. --catrowhdu: The HDU(s) of the FITS file(s) given to '--catrowfile'. --noblankend: remove blank rows from selected columns of the _output_ table, for example after after adding columns by column arithmetic or from other tables (this is the new name for the old '--noblank' option, see the changed features below). Like the new '--noblank', this option can also be called multiple times, so '--noblankend=1 --noblankend=2' is equivalent to '--noblankend=1,2'. - New operator for column arithmetic: - date-to-millisec: convert the formatted date string into the number of milli-seconds (as a 64-bit integer) since Unix-time (00:00:00 of 1970-01-01). Match: - k-d tree based matching has been added for finding the matching rows, and is set to the default mode. It will do the matching in parallel, which will improve the speed when many rows are present. The k-d tree can also optionally be saved in a file for matching new tables with the same one (further improving running speed). For a full description of Match's new behavior, please see the new "Matching algorithms" section of the book. This feature was implemented with the help of Sachin Kumar Singh. --kdtree: new option to specify the algorithm used by Match. It can take four values: 'internal' (which is the default), 'build' (to build a k-d tree), 'FILE.fits' (name of k-d tree to import), or 'disable' (to do the matching with the old sort-based method). For more, please see the new "Matching algorithms" section of the book. --kdtreehdu: the HDU of the external k-d tree (if a FITS file was given to '--kdtree'). Library: - gal_fits_key_exists_fptr: Check if a certain keyword exists in the already-open FITS file pointer. ** Removed features Nothing was removed in this release. ** Changed features Arithmetic: - 'add-dimension-slow' is the new name for the old 'add-dimension'. Segment: - '--noobjects' is the new name for the old '--onlyclumps' option. This is because the name Only-Clumps can lead to a wrong expectation that the output will _only_ contain a 'CLUMPS' HDU. However, with'--onlyclumps', the input image (without a Sky level) and the Sky standard deviation image are also present as HDUs in the output. To only get the 'CLUMPS' HDU, it is necessary to also use the '--rawoutput' option. This renaming was done to avoid such confusions and was raised by Sepideh Eskandarlou. Match: - By default (when '--quiet' isn't called), Match will print the names of the inputs and timings of important steps to the standard output (on the command-line). Table: - The '--noblank' option's logic is very similar to other row-selection (by value) options (like '--range', '--polygon' or '--equal'). However, until now, it was applied at the end of Table, just before writing the output (so it would apply to the output table, that may have new/different column names/orders, not the input table like the other similar options). This was very inconvenient and non-intuitive. From this version, '--noblank' option now behaves similarly to the other input row-selection (by value) options. So it can be called multiple times (to give many columns in multiple calls) and it can be called for a column that is not necessarily one of the output columns (for example you want the RA and Dec of all rows without a blank magnitude, so you don't need to have the magnitude column any more!). The old behavior of this option is now available in the new '--noblankend' option. This point was raised by Samane Raji. - 'date-to-sec' operator now only returns the number of seconds and is always in a 64-bit signed integer format. Until now, if the input had sub-second precision it would return a 64-bit floating point type; however, that resulted in bug #61976 (where the loss of precision due to the floating-point's logarithmic scale for too many significant digits caused incorrect conversions). Library: - gal_fits_tab_read: now takes the number of threads to read the desired columns in parallel (this will greatly improve reading speed when there are many columns and many rows). - gal_table_read: similar to 'gal_fits_tab_read' (only relevant for reading FITS tables). - gal_blank_initialize: also works on string data, but initializing only a tile over a larger block of elements is only supported for numeric data types. ** Bugs fixed bug #61329: make check crash in macOS in convolve/spectrum-1d.sh, found and fixed with the help of Sebastian Luna-Valero. bug #61378: Table crash when input is sexagesimal RA/DEC and no metadata. bug #61462: Crash on FITS binary table columns with repeat of 0. bug #61468: DS9 region file can't be read in sexagesimal coordinates, reported by Juan Antonio Fernández Ontiveros and Alejandro Lumbreras Calle. bug #61490: Extra columns in the metadata causes crash in plain-text tables, found by Zohreh Ghaffari. bug #61493: NoiseChisel's --checksn not showing pseudos equal to --snminarea (only larger ones), found by Raúl Infante-Sainz. bug #61584: When FITS file doesn't end with '.fits' and the output file already existed, Arithmetic would write the output in new extension while it should have been deleted. bug #61598: ds9-region script not using precise WCS values, found and fixed by Raúl Infante-Sainz. bug #61698: Mean operator of Arithmetic (for stacking many images) not returning NaN for blank regions when input is integer. This fix also resolves the same problem in the 'sum', 'std', 'median' and all 'sigclip-*' operators); reported by Zohreh Ghaffari. bug #61740: WCSLIB conflicting with system's wide-character string (WCS) library in Gnulib checks for some systems; reported by Martin Guerrero Roncel. bug #61940: Numbers with many decimals (like '330624.3918430004', common in Julian dates for example) incorrectly read as float32 (thus loosing precision); reported by Zohreh Ghaffari, fixed with help of Pedram Ashofteh-Ardakani. bug #61967: DS9 polygon region files not read when they have width and color; reported by Zohreh Ghaffari, fixed by Pedram Ashofteh-Ardakani. bug #61976: Table arithmetic date-to-sec produces same result for different times (separated by about 1 second). This bug was reported by Zohreh Ghaffari and Pedram Ashofteh-Ardakani. bug #62008: Arithmetic not deleting existing output when the 'makenew' is used (no FITS file exists in the reverse polish notation). bug #62052: WCS decomposition of CD into PC+CDELT not setting internal values; reported by Ignacio Ruiz Cejudo and Raul Infante-Sainz. bug #62054: Crash in Table's '--catrowfile' when string column is present; reported by Manuel Sánchez-Benavente. bug #62069: Wrong Arithmetic result in binary operators when both input operands are integers of same width, but different sign (for example 'int32' and 'uint32'); reported by Zohreh Ghaffari and Pedram Ashofteh-Ardakani. bug #62070: Segmentation fault in Table's '--tail' option, when string columns are requested and the requested number of rows given to '--tail' is more than half of the number of rows; reported by Manuel Sánchez-Benavente. bug #62096: 'astarithmetic 250 1 +' not producing correct result; reported by Raul Infante-Sainz. bug #62112: NoiseChisel crash when '--checktiles' and '--continueaftercheck' called together; reported by Giulia Golini. bug #62127: GNU Scientific Library's 1D Steffen interpolation method (that guarantees monotonicity) not used in Gnuastro's library, even if it is present and found at configure time; found and fixed by Pedram Ashofteh-Ardakani. bug #62204: Column arithmetic crashes when there are no rows; reported by Zohreh Ghaffari. * Noteworthy changes in release 0.16 (library 14.0.0) (2021-10-10) [stable] ** New features Book: - New section added to ConvertType's documentation for annotating the PDF output images with coordinates on the edges of the images, or tangential size at certain redshift. This is done using the PGFPlots package of LaTeX, directly with the same fonts and graphical settings as your paper/slide, while being in vector graphics quality, and not requiring hundreds of dependencies (only LaTeX is necessary beyond Gnuastro). - New section to describe the precedence of Table operators in one call to Table. Since the number of operations on table columns/rows has greatly increased in the last few versions, knowing this precedence can minimize the number of calls to Table. - The section on reverse polish notation (used in Arithmetic, for images, and column arithmetic in Table) has been edited and extended to be more useful for a new user. - Due to the great expansion of the number of operators in the Arithmetic program and Column Arithmetic in the table program, the "Arithmetic operators" section of the book has now grouped similar operators in sub-sections to greatly help in finding a certain option. Arithmetic: - New operands (also available in Table's column arithmetic, the distance conversion operators were suggested by Markus Schaney): - box-around-ellipse: width and height of the box covering an ellipse. - au-to-pc: Convert Astronomical Units (AUs) to Parsecs (PCs). - pc-to-au: Convert Parsecs (PCs) to Astronomical Units (AUs). - ly-to-pc: Convert Light-years (LYs) to Parsecs (PCs). - pc-to-ly: Convert Parsecs (PCs) to Light-years (LYs). - ly-to-au: Convert Light-years (LYs) to Astronomical Units (AUs). - au-to-ly: Convert Astronomical Units (AUs) to Light-years (LYs). CosmicCalculator: - A warning is printed if the requested redshift is lower than 0.007 (corresponding to ~30Mpc in Plank 2018 cosmology). Because at these scales the peculiar velocity of galaxies may be significant compared to the Hubble flow (which is the basis of the calculations here). The warning can be suppressed with the '--quiet' option. This was suggested by Ignacio Trujillo. MakeCatalog: --areaerror: spatial resolution of image specified by user, used in estimating the surface brightness error. --sberror: error in measuring the surface brightness (mag/arcsec^2). MakeProfiles: - New type of profile showing the azimuthal angle (in degrees, along the elliptical circumference of fixed radius) of each pixel. In combination with the radial distance profile, you can now create complex features in polar coordinates, such as tidal tails or tidal shocks (using the Arithmetic program to mix the radius and azimuthal angle through a function to create your desired features). Implemented after discussions with Fernando Buitrago and Matthias Kluge. Match: - When called with '--notmatched --outcols=AAA,BBB', Match will append non-matching rows of second table into first table's rows (for columns 'AAA' and 'BBB' in example above, which are assumed to exist in both inputs, with same numeric datatype). This allows easy/clean merging of two catalogs that may have matching objects into one catalog without repetition. See description of '--outcols' in the Match manual for more. This feature was proposed by Juan Molina Tobar and Leslie Hunt. Sort-by-night (installed script) --stdintimeout: new option to set internal delay time for input from pipes (standard input). This is not for inputs to the script (which should always be files), but for the script's internal pipes. Statistics: --quantofmean: the quantile of the mean of the input dataset. this is a very good statistic to measure skewness in a distribution, see the description of this option in the book for more. Library: - Arithmetic macros: - GAL_ARITHMETIC_OP_BOX_AROUND_ELLIPSE - gal_wcs_write_wcsstr: wrapper of wcshdo of WCSLIB to simplify generic access to that low-level operation (has some extra sanity checks). - gal_units_au_to_pc: Convert Astronomical Units to Parsecs. - gal_units_pc_to_au: Convert Parsecs to Astronomical Units. - gal_units_ly_to_pc: Convert Light-years to Parsecs. - gal_units_pc_to_ly: Convert Parsecs to Light-years. - gal_units_ly_to_au: Convert Light-years to Astronomical Units. - gal_units_au_to_ly: Convert Astronomical Units to Light-years. ** Removed features ** Changed features All programs: --stdintimeout: default value changed from 0.1 seconds to 1.5 seconds to avoid too many crashes when command before the pipe takes longer (thus needing a manual change of this value). MakeProfiles: - The default output suffix (when no '--output' is given) is now '_profiles.fits'. Until now, when no output name was given, MakeProfiles just replaced the suffix with '.fits' (effectively assuming the input is a plain-text file). As a result, if the input catalog was a FITS file, it would be automatically written-over (causing an error: bug #60989). Match: - The two '--notmatched' and '--outcols' can be called together (to create a single catalog that appends the non-matching rows of second table with the rows of the first. Until this version, this would cause an error. Table - Column concatenation (adding columns from other files) is now done before all row selection or row re-ording operations (if called). Until now, adding columns from other files was done after all row selection or re-ordering, but this was not too practically useful. - Column arithmetic operators: - 'wcs-to-img' new name for old 'wcstoimg' operator. The new unit-conversion operators generally have a '-to-' between the units to help in readability. So to avoid confusion, this operator has also been changed. - 'img-to-wcs' new name for old 'imgtowcs' operator (see description of 'wcs-to-img' above). Library - gal_fits_key_write_filename: now has 'quiet' argument to avoid printing a warning. ** Bugs fixed bug #60725: MakeCatalog doesn't put comment on --halfsumsb column. bug #60776: Radial profile script not using standard deviation image, fixed by Zahra Sharbaf. bug #60778: Brightness error not NaN when all STD pixels are blank, reported by Zahra Sharbaf. bug #60826: Arithmetic won't delete existing file with tofile operators. bug #60881: Query segmentation fault when NED is called without a dataset. bug #60901: sort-by-night crash during make check due to stdintimeout, reported by Zahra Sharbaf. bug #60903: Increasing value of stdintimeout not effective beyond 1000000, reported by Zahra Sharbaf. bug #60909: WCS coordinate conversion not working with TPV distortion, fixed with the help of Mark Calabretta. bug #60923: Standard deviation calculation giving NaN values due to floating point errors, found and fixed by Natáli Anzanello. bug #60958: Warning on long filename not written in FITS keyword printed even when '--quiet' is called, found by Raúl Infante-Sainz. bug #60989: MakeProfiles deletes input fits catalog when no output name specified. bug #60999: No content check when plain-text table doesn't have metadata. bug #61007: Crash due to CFITSIO 4.0.0 version format change, reported by Vincenzo Testa and Zohreh Ghaffari. bug #61108: Plain text table metadata ignored when numeric data type was not understood, reported by Sepideh Eskandarlou. bug #61128: Radial profile: --instd and --stdhdu error when space used before value, found and fixed by Sepideh Eskandarlou. bug #61192: BuildProgram custom program linking problems on Debian-based OSs, reported by Raúl Infante-Sainz. bug #61244: Sum operator in arithmetic ignoring floating point images with blank values, found and reported by Giulia Golini and Raúl Infante-Sainz. bug #61287: Two "switch" blocks miss "break" in MakeCatalog when setting output column information, found and fixed by Vladimir Markelov. bug #61293: MakeCatalog not properly initializing the '--minvy' column when parsing clumps, found and fixed by Vladimir Markelov. * Noteworthy changes in release 0.15 (library 13.0.0) (2021-05-30) [stable] ** New features New programs: - astscript-ds9-region: Given a table (either as a file or from standard input), create an SAO DS9 region file from the requested positional columns (WCS or image coordinates). For example with the command below you can select certain rows of a given table, and show them over an image: asttable table.fits --range=MAGNITUDE,18:20 --column=RA,DEC \ | astscript-ds9-region --column=1,2 --radius=0.5 \ --command="ds9 image.fits" This script was written with the help of Samane Raji. - astscript-radial-profile: Measure the radial profile of an object on an image. The profile can be centered anywhere in the image and any circular, or elliptical distance can be defined. The output is a table with the profile's value in one column and any requested measure in the other columns (any MakeCatalog measurement is possible). This script was written and is maintained by Raúl Infante Sainz. Zahra Sharbaf and Carlos Morales Socorro also contributed. All programs: - FITS files that don't have a recognized FITS suffix (e.g., '.fits', '.fit', '.fits.gz', etc) will also be recognized. Generally, it is good practice to keep a suffix to be more human and computer friendly (checking three or four characters is much faster than opening the file and checking the contents to conform with the FITS standard), but when you are given such a file (by someone else), this new feature can be handy. Suggested by Clotilde Laigle and Leigh Smith. --wcslinearmatrix: new option in all programs that lets you select the output WCS linear matrix format. It takes one of two values: 'pc' (for the 'PCi_j' formalism) and 'cd' (for 'CDi_j'). Until now, the outputs were always stored in the 'PCi_j' formalism (which is still the recommended format). Book: - New "Image surface brightness limit" section added to the third tutorial (on "Detecting large extended targets"). It describes the different ways to measure a dataset's surface brightness limit and upper-limit surface brightness, while discussing their differences. This tutorial was written with the help of Ignacio Trujillo and Raúl Infante Sainz. Arithmetic: - New operators (all also available in Table's column arithmetic). - sin: Trigonometric sine (input in degrees). - cos: Trigonometric cosine (input in degrees). - tan: Trigonometric tangent (input in degrees). - asin: Inverse of trigonometric sine (output in degrees). - acos: Inverse of trigonometric cosine (output in degrees). - atab: Inverse of trigonometric tangent (output in degrees). - sinh: Hyperbolic sine. - cosh: Hyperbolic cosine. - tanh: Hyperbolic tangent. - asinh: Inverse of hyperbolic sine. - acosh: Inverse of hyperbolic cosine. - atanh: Inverse of hyperbolic tangent. - mknoise-sigma: Add Gaussian noise with the fixed sigma. - mknoise-poisson: Add Poisson noise with the given background. - mknoise-uniform: Add uniform noise around existing value. - counts-to-mag: Convert counts to magnitudes with given zero point. - mag-to-counts: Convert magnitudes to counts with given zero point. - counts-to-jy: Convert counts to Janskys through a zero point based on AB magnitudes. Trigonometric operators proposed by Zahra Sharbaf and Samane Raji. --envseed: new option to get random number generator settings for the new 'mknoise-sigma' and 'mknoise-poisson' operators from the environment for reproducibility (see "Generating random numbers" section in manual). ConvertType: --globalhdu: Use a single HDU identifier for all the input files files. Its operation is identical to the similarly named option in Arithmetic. Until now it was necessary to call '--hdu' three times if you had three input FITS files with input in the same HDU. Crop: --polygon: now also accepts an SAO DS9 region file as input (as well as coordinates). This enables you to draw your polygon over the image through the GUI interface of SAO DS9, save your polygon as a "region" file, and feed the region file into Crop directly. Until now, the only way to define the polygon was to give all the polygon vertice coordinates, one-by-one to the '--polygon' option (which was tedious and buggy), but is still possible. This was implemented by Natáli Anzanello. Fits: --hastablehdu: print 1 if at least one table HDU exists in file. --hasimagehdu: print 1 if at least one image HDU exists in file. --listallhdus: print all HDU names (or numbers when no name exists). --listtablehdus: print table HDU names (or numbers when no name exists). --listimagehdus: print image HDU names (or numbers when no name exists). --printkeynames: Print all keyword names in current HDU. --wcscoordsys: convert the WCS coordinate system of the input into any recognized coordinate system. It currently supports: equatorial (J2000, B1950), ecliptic (J2000, B1950), Galactic and Supergalactic. For example if 'image.fits' is in galactic coordinates, you can use this command to convert its WCS to equatorial (J2000): astfits image.fits --wcscoordsys=eq-j2000 This option only works with WCSLIB 7.5 and above (released in March 2021), otherwise it will abort with an informative warning. This was done with the help of Mark Calabretta. --keyvalue: Print only the values of the FITS keywords given to this option in separate columns. This option can take multiple keyword names and many FITS files. Thus generating a table of keyword values (with one row per file where the first column is the file name). Its output can thus be written as a Table file or be piped to the Table program for selecting a certain sub-set of your FITS files based on key values, or sorting them for example. This was added after a discussion with Alberto Madrigal. MakeCatalog: - Newly added measurement columns: --upperlimitsb: upper-limit surface brightness for the given label (object or clump). This is useful for measuring a dataset's realistic surface brightness level for each labeled region by random positioning of its footprint over undetected regions (not extrapolated from the single-pixel noise level like the "surface brightness limit"). NoiseChisel: - Can now work on 3D datacubes. Since the configuration parameters are different from images, it is recommended to manually set the 3D configuration (the '...' can be the input image and options): astnoisechisel --config=/usr/local/etc/astnoisechisel-3d.conf ... Alternatively, you can set an 'astnoisechisel-3d' alias like below and always easily run 'astnoisechisel-3d' on cubes. alias astnoisechisel-3d="astnoisechisel --config=/usr/local/etc/astnoisechisel-3d.conf" Segment: - Can now work on 3D datacubes. Similar to NoiseChisel, it requires a separate set of default configurations, so please see the note under NoiseChisel above. Table: - When given a value of '_all', the '--noblank' option (that will remove all rows with a blank value in the given columns) will check all columns of the final output table. This is handy when you want a "clean" (no NaN values in any column) table, but the table has many columns. Until now, '--noblank' needed the name/number of each column to "clean". --rowlimit: new option to specify the positional interval of rows to show. Until now, the '--head' or '--tail' options would just allow seeing the first or last few rows. You can use this new option to view a contiguous set of rows in the middle of the table. --rowrandom: Make a random selection of the rows. This option is useful when you have a large table and just want to see a random sub-set of the rows. It takes an integer, selects that many rows from the input randomly. --polygon: Similar to same option in Crop (mentioned above). - New column arithmetic operators: - 'set-AAA' operator (which allows storing the popped operand into a named variable for easy usage in complex operations) is also usable in Table's column arithmetic. Until now this operator was only available in the Arithmetic program (for operation on images). - 'date-to-sec' Convert FITS date format ('YYYY-MM-DDThh:mm:ss') into seconds from the Unix epoch (1970-01-01,00:00:00 UTC). This can be very useful in combination with the new '--keyvalue' option of the Fits program to operate on FITS dates (for example sort your FITS images based on observation time). Query: - The Galactic extinction calculator of the NASA/IPAC Extragalactic Database (NED) is now available for any coordinate with a command like below. For more, see the manual (the description of the 'extinction' dataset of NED in the "Available datasets" section). astquery ned --dataset=extinction --center=49.9507,41.5116 This feature was suggested by Ignacio Trujillo and implemented with the help of Joseph Mazzarella. Library: - New arithmetic operator macros (for the 'gal_arithmetic' function): - GAL_ARITHMETIC_OP_SIN: sine (input in deg). - GAL_ARITHMETIC_OP_COS: cosine (input in deg). - GAL_ARITHMETIC_OP_TAN: tangent (input in deg). - GAL_ARITHMETIC_OP_ASIN: Inverse sine (output in deg). - GAL_ARITHMETIC_OP_ACOS: Inverse cosine (output in deg). - GAL_ARITHMETIC_OP_ATAN: Inverse tangent (output in deg) - GAL_ARITHMETIC_OP_ATAN2: Inverse tangent (with two inputs, out deg). - GAL_ARITHMETIC_OP_SINH: Hyperbolic sine. - GAL_ARITHMETIC_OP_COSH: Hyperbolic cosine. - GAL_ARITHMETIC_OP_TANH: Hyperbolic tangent. - GAL_ARITHMETIC_OP_ASINH: Inverse hyperbolic sine. - GAL_ARITHMETIC_OP_ACOSH: Inverse hyperbolic cosine. - GAL_ARITHMETIC_OP_ATANH: Inverse hyperbolic tangent. - GAL_ARITHMETIC_OP_MKNOISE_SIGMA: Add fixed-sigma noise. - GAL_ARITHMETIC_OP_MKNOISE_POISSON: Add Poisson noise. - GAL_ARITHMETIC_OP_COUNTS_TO_JY: Convert counts to Janskys. - New arithmetic flag macros: - GAL_ARITHMETIC_FLAG_ENVSEED: read random number generator seed from env. - GAL_ARITHMETIC_FLAG_QUIET: don't print any warnings by some operators. - WCS coordinate system macros: - GAL_WCS_COORDSYS_EQB1950: 1950.0 (Besselian-year) equatorial coords. - GAL_WCS_COORDSYS_EQJ2000: 2000.0 (Julian-year) equatorial coords. - GAL_WCS_COORDSYS_ECB1950: 1950.0 (Besselian-year) ecliptic coords. - GAL_WCS_COORDSYS_ECJ2000: 2000.0 (Julian-year) ecliptic coords. - GAL_WCS_COORDSYS_GALACTIC: Galactic coordinates. - GAL_WCS_COORDSYS_SUPERGALACTIC: Supergalactic coordinates. - gal_array_file_recognized: For FITS images, check contents also. - gal_ds9_reg_read_polygon: Parse the polygon from an SAO DS9 region file. - gal_fits_file_recognized: Check the file contents when suffix isn't FITS. - gal_units_counts_to_mag: Convert counts to magnitudes. - gal_units_mag_to_counts: Convert magnitudes to counts. - gal_units_counts_to_jy: Convert counts to Janskys. - gal_wcs_coordsys_from_string: WCS coordinate system from string. - gal_wcs_coordsys_identify: Parse WCS struct to find coordinate system. - gal_wcs_coordsys_convert: Convert the coordinate system of the WCS. ** Removed features ** Changed features astscript-sort-by-night: - Thanks to the new features in the Fits and Table programs (described above), the efficiency of this script has improved dramatically (from 19 seconds to 0.42 seconds for about 650 FITS files used in the test!). - The default end to a "night" is set to 11:00a.m. Until now it was 9:00a.m. But in some cases, calibration images may be taken after that. So to be safer in general it was incremented by 2 hours. MakeCatalog: - Surface brightness limit (SBL) calculations are now written as standard FITS keywords in the output catalog/table. Until now, they were simply stored as 'COMMENT' keywords with no name so it was hard to parse them automatically. From this version, the following keywords are also written into the output table(s), see the "MakeCatalog output" section of the book for more: 'SBLSTD', 'SBLNSIG', 'SBLMAGPX', 'SBLAREA', 'SBLMAG'. - Upper-limit (UP) settings are also written into the output tables as keywords (like surface brightness limit numbers above): 'UPNSIGMA', 'UPNUMBER', 'UPRNGNAM', 'UPRNGSEE', 'UPSCMLTP', 'UPSCTOL'. Library: - GAL_ARITHMETIC_FLAG_FREE new name for GAL_ARITHMETIC_FREE. - GAL_ARITHMETIC_FLAG_NUMOK new name for GAL_ARITHMETIC_NUMOK. - GAL_ARITHMETIC_FLAG_INPLACE new name for GAL_ARITHMETIC_INPLACE. - GAL_ARITHMETIC_FLAGS_BASIC new name for GAL_ARITHMETIC_FLAGS_ALL. - gal_fits_key_write_wcsstr: also takes WCS structure as argument. - gal_fits_key_read_from_ptr: providing a numerical datatype for the desired keyword's value is no longer mandatory. When not given, the smallest numeric datatype that can keep the value will be found and used. - gal_wcs_read: allows specifying the linear matrix of the WCS. - gal_wcs_read_fitsptr: allows specifying the linear matrix of the WCS. ** Bugs fixed bug #60082: Arithmetic library crash for integer operators like modulo bug #60121: Arithmetic segfault when multi-operand output given to set- bug #60368: CosmicCalculator fails --setdirconf when redshift isn't given, reported by Sepideh Eskandarlou bug #60483: No warning when file given to '--config' doesn't exist, reported by Sepideh Eskandarlou bug #60484: Match crashes when called with --coord and --ccol2 (together) bug #60603: Table crashes with an empty input and --range, reported by Sepideh Eskandarlou. bug #60619: Crop crashes with differing image and WCS dimension bug #60634: Crop's '--hstartwcs' and '--hendwcs' wrong with comment keys bug #60644: MakeCatalog's Brightness error over-estimated in images with noise STD less than 1. This was not an issue for S/N or Magnitude error on the same image (already corrected there). * Noteworthy changes in release 0.14 (library 12.0.0) (2021-01-25) [stable] ** New features Book: - Tutorial on "Detecting large extended targets" improved with better NoiseChisel configuration, and more clear description. - New sub-section on "Memory management" in the "Common program behavior" chapter. It fully describes how to optimally deal with large datasets that may exceed your system's RAM. - Examples and better description added to many operators in the "Arithmetic operators" subsection of the Arithmetic program's section. New program: - Query ('astquery') is a new program to query to external datasets and retrieve the resulting datasets from the command-line. It is possible to get list of datasets or column names from the databases. The basic spatial query (finding objects in the vicinity of a certain point) can be managed without knowing the query language of the database: with basic options (like '--center' with '--radius' or '--width'), or an input image that has WCS. Currently Query supports VizieR (containing +20500 datasets, making it the largest database in astronomy), NED, as ESA's Gaia database and ASTRON. See the new "Query" section of the book (under the "Data containers" chapter) for more. All programs: - Plain text table inputs can have floating point columns that are in sexagesimal format of '_h_m_s' or '_d_m_s' (where '_' is a number). These are the classical format to respectively represent Right Ascension (RA) and Declination (Dec). They will be directly read as a single floating point number (in units of degrees) into memory. Therefore the same column of a plain-text table, can be degrees in some rows and sexagesimal in others. Besides large tables, with this feature, conversion to sexagesimal coordinates to degrees becomes very easy, for example: echo "7h34m35.5498 31d53m14.352s" | asttable Recall that the inverse can also be done with the more general column arithmetic: echo "113.64812416667 31.88732" \ | asttable -c'arith $1 degree-to-ra $2 degree-to-dec' - If input is a HEALpix grid (1D table column that represents the 2D spherical representation of datasets), the programs will print a warning, suggesting to use the 'HPXcvt' utility of WCSLIB. Arithmetic: - New operators: - 'interpolate-maxofregion': interpolate connected blank regions with the maximum value that is immediately touching it. This can be used to fill the blank centers of saturated stars for example. - 'interpolate-minofregion': similar to 'interpolate-maxofregion', but for the minimum. - 'makenew': new operator to create an empty (zero-valued) new dataset with given dimension and size (given as operands). Crop: --primaryimghdu: Write the final cropped image into the primary (or 0-th) extension of the output FITS file, so the output only has one extension. Fits: - New '--skycoverage' option will report the area of the input image in RA/Dec, both in units of center/width, and box minimum/maximum format. This is paritcularly useful in combination with the new 'astquery' option, to easily search the contents of external databases within the image. MakeCatalog: --maximum: maximum value of labeled regions pixels (clump/object). --areaarcsec2: area of labeled region (clump/object) in arcsec^2. --surfacebrightness: the surface brightness of the labeled region. --fwhm: observed FWHM in pixels (non-parametric), along the major axis. --halfmaxarea: number of pixels with a value larger than half the maximum. --halfmaxradius: radius of region that is larger than half the maximum. --halfmaxsum: sum of pixels with a value larger than half the maximum. --halfmaxsb: surf. brightness within half of the maximum. --fracmax: fractions to use in '--fracmaxarea1' or '--fracmaxarea2'. --fracmaxsum1: sum of pixels brighter than first given fraction of max. --fracmaxsum2: sum of pixels brighter than second given fraction of max. --fracmaxarea1: number of pixels brighter than first given fraction of max. --fracmaxarea2: number of pixels brighter than second given fraction of max. --fracmaxradius1: radius derived from '--fracmaxarea1'. --fracmaxradius2: radius derived from '--fracmaxarea2'. --halfsumsb: Surface brightness within area reported by '--halfsumarea'. --halfsumarea: area containing half of the summed object or clump values. --halfsumradius: radius derived from '--halfsumarea', underestimates r_e. --areaminv: the number of pixels that are equal to the minimum value. --areamaxv: the number of pixels that are equal to the maximum value. - New columns to return the position of pixel with minimum or maximum value: '--minvx', '--maxvx', '--minvy', '--maxvy', '--minvz', '--maxvz'. MakeNoise: --bgisbrightness: new option to say that the value of '--background' (used to simulate Poisson noise) should be interpreted as brightness, not magnitude. MakeProfiles: - It is now possible to make any custom radial profile with the 'custom' profile (with code '8'). A table should be given to the new '--customtable' option which will define each radial interval and the value to use for that radial interval. See the description of '--customtable' for more. Statistics: - 2D histograms can now be built as a FITS image with a linear WCS that contains axis values and box size. This allows using the power of FITS viewers for plotting/inspecting distributions of two columns in a table relative to each other (for example color-magnitude plots). You can also convert these 2D histogram images to PDF or JPEG with Gnuastro's ConvertType to directly use in your papers/reports (see the newly added "2D histogram as an image" section of the book for more). Table: - New '--noblank' option will remove all rows in output table that have at least one blank value in the specified columns. For example if 'table.fits' has blank values (NaN in floating point types) in the 'magnitude' and 'sn' columns, with '--noblank=magnitude,sn', you can be sure that all rows with blank values in these columns have been removed. - New trigonometric operators for column-arithmetic (inputs in units of degrees): 'sin', 'cos' and 'tan'. Their inverse trigonometric (outputs in units of degrees) have also been added: 'asin', 'acos' and 'atan'. The 'atan2' operator (inverse tangent that preserves the quadrant, see its description in the book) is also now available. - New hyperbolic operators for column-arithmetic: 'sinh', 'cosh' and 'tanh'. Their inverse has also been added: 'asinh', 'acosh' and 'atanh'. Library: - GAL_ARITHMETIC_OP_MAKENEW: new 'makenew' operator. - gal_binary_connected_adjacency_list: finding connected components without a square adjacency matrix (which can consume major RAM). - gal_blank_flag_remove: Remove all flagged elements in a dataset. - gal_blank_remove_rows: remove all rows that have at least one blank. - gal_dimension_dist_elliptical: Elliptical dist. of a point from center. - gal_fits_hdu_is_healpix: Return 1 if HDU is a HEALpix grid. - gal_fits_hdu_datasum: calculate DATASUM of given HDU in given FITS file. - gal_fits_hdu_datasum_ptr: calculate DATASUM of opened FITS file pointer. - gal_fits_key_list_comment_add: add a COMMENT keyword to the keyword list. - gal_fits_key_list_comment_add_end: add a COMMENT keyword to end of list. - gal_pointer_allocate_ram_or_mmap: allocate in RAM or memory-mapped file. - gal_pointer_mmap_free: "free" (actually delete) the memory-mapped file. - gal_wcs_create: create WCSLIB-compatible WCS from raw values. - gal_wcs_coverage: Return the sky coverage of given image HDU. - gal_wcs_dimension_name: return the name of the requested WCS dim. ** Removed features ** Changed features All programs: - Memory management: Until now, an internal array was only allocated in the RAM when its size was smaller (in bytes) than the value given to the '--minmapsize' option. But this was annoying/buggy when the system has enough RAM to keep large files. From this version, all Gnuastro programs will first attempt to write the array in RAM, only when it fails (there is no more RAM left), will they use a memory-mapped file (which can dramatically slow down the program). Please see the newly added "Memory management" section of the book for a complete explanation of Gnuastro's new memory management strategy. - When an array needs to be memory-mapped (read into a file on HDD/SSD, not RAM, usually due to RAM limitations), it is written in a 'gnuastro_mmap' directory of the running directory, not the hidden '.gnuastro_mmap' directory. Since the files in this directory are usually large, it is better that the user sees them by default (in case the program crashes and they aren't deleted). --interpnumngb: the default value has been increased to 15 (from 9). The reason for this is that we now have a more robust outlier removal algorithm (see description under "NoiseChisel & Statistics"). Crop: - When cropping a single image in WCS mode, there is no longer any limitation on the WCS. Until now for all WCS mode crops, it was necessary for the WCS to be aligned to the celestial coordinates. But from this version, this is only necessary when cropping from many files (and stitching them together where necessary). For WCS-mode crops of a single image, any WCS that is recognized by WCSLIB is fine. Fits: - The '--pixelscale' option also prints the pixel area (for 2D inputs, or 2D slices of 3D inputs) and the voxel volume (for 3D inputs). Until now, it would only print the pixel scale along each dimension. - When printing FITS file HDU information (no options given), a new "Comments" column may be printed for each HDU in the end of the line. It will be printed if special situations are found (for example a 2D HEALPix grid, that is usually stored as a 1D array/column). NoiseChisel & Statistics: - New algorithm used to reject outlying tiles. In NoiseChisel this is done when estimating the quantile threshold, the pseudo-detection threshold and the final Sky value. In Statistics, its just the Sky value. Unlike the previous method that used the global distribution of tile values to identify outliers, the new algorithm uses a relative measure. See the book for more. Since outliers are now rejected more robustly, the default value of '--meanmedqdiff' has been increased to 0.01 (was 0.005) and '--outliersigma' has been decreased to '5' (was 10). Also 'smoothwidth' has been increased from '3' to '5' to have smoother tessellation values. Statistics: - The '--histogram2d' now takes a string argument: either 'image' or 'table'. For the old behavior please use '--histogram2d=table'. See the new features above for the 'image' mode. Table: - Column arithmetic operators 'degree-to-ra' and 'degree-to-dec' will return the sexagesimal format of '_h_m_s' and '_d_m_s' respectively. Until this version, they would both use colons as delimiters ('_:_:_'). Library: - gal_pointer_mmap_allocate: new name for 'gal_pointer_allocate_mmap'. - gal_threads_dist_in_threads: now accounts for billions of threads, thus includes memory management options. - gal_threads_spin_off: now accounts for memory management. - gal_units_degree_to_ra: new 'usecolon' argument to optionally format output string with colons as delimiters ('_:_:_'). When this option is zero, the string will be in the '_h_m_s' format. - gal_units_degree_to_dec: similar to 'gal_units_degree_to_ra', but when 'usecolon' is zero, the string will be in the '_d_m_s' format. ** Bugs fixed bug #59017: Segment's object IDs are not thread-safe (i.e., reproducible). bug #59105: Column arithmetic operator degree-to-ra, returning to dec. bug #59136: Makeprofiles with --replace is not thread-safe. bug #59155: Match cannot find the proper row when coordinates have NaN. bug #59371: MakeCatalog crash with clumps on non-contiguous object labels. bug #59400: CosmicCalculator fails --printparams when redshift isn't given. bug #59459: Unclear WCS when both PC and CD exist in input, but conflict. bug #59625: MakeProfiles uses last --kernel, if it is called more than once. bug #59700: Segment's excessive RAM usage when many clumps over a detection. bug #59765: MakeCatalog crash when to-be-subtracted Sky is a single-element per tile tessellation (e.g., NoiseChisel's '--oneelempertile'). * Noteworthy changes in release 0.13 (library 11.0.0) (2020-09-06) [stable] ** New features All programs: - When reading plain-text tables, the blank value for numeric columns can be any string (specified in the special comment-line format described in the "Gnuastro text table format" of the manual). Until now, it had to be a number in the same type. Arithmetic: - New operators: - interpolate-minngb: Fill blanks with minimum of nearest neighbors. - interpolate-maxngb: Fill blanks with maximum of nearest neighbors. This can be useful to fill the blank values of saturated stars for example. - To force integers to floats, you can also put a '.' or '.0' after them. Until now, it was only possibly by putting an 'f' after them. Hence while '5' will be read as an integer, '5.', '5.0' or '5f' will be read as floating point. This also applies to column arithmetic in Table. ConvertType: - New colormap: 'sls-inverse' is the inverse of the SLS color, good for printing because of a white background. CosmicCalculator: --velocity: Velocity (in km/s) to use instead of input redshift. --usedvelocity: Print the velocity (in km/s) at input redshift. --listlinesatz: Print the wavelength of all pre-defined spectral lines at given redshift as a simple table with the line names. This is very convenient and can be used in conjunction with '--obsline' for example to print the observed wavelength of all lines when Lyman-alpha is at 4000 Angstroms with this simple command: astcosmiccal --obsline=lyalpha,4000 --listlinesatz FITS: - New '--pixelscale' option will return the size of pixels in each dimension in the "world coordinates". - New '--wcsdistortion' option allows conversion between the various WCS distortions. For example if you have a FITS image with the TPV-based WCS distortion, and you would like to convert it to a SIP-based distortion, you can simply run 'astfits --wcsdistortion=SIP' on the file. The inverse conversion is also supported (from SIP to TPV). Statistics - New feature to create a 2D-histogram using two input columns (useful when you have lots of points that are too dense and may hide important features). This mode can be activated with the new '--histogram2d' option. The binning of the first (X axis) column is specified with the same 1D histogram options. The second column's binning is configured with the following options: --numbins2: Number of bins along the second column. --greaterequal2: Only second column points that are larger than this. --lessthan2; Only second column points that are less than this. --onebinstart2: Make sure one bin starts at the value given here. Table: - New '--catcolumns' to specify which columns to concatenate (or append) to the output. You can specify the file name containing the columns to append with the '--catcolumnfile' option and '--catcolumnhdu' (see changed features because until now they had different names). - New '--catcolumnsrawname' will leave the name of concatenated (appended) columns unchanged. By default the names of the appended columns will be appended with a '-N' (where 'N' is a counter for the file that is used to append columns). The default behavior is to avoid multiple columns having the same name. - New '--colmetadata' option to add/update column metadata (name, units or comments) just before writing the output. This is a very useful feature in combination with column arithmetic or column concatenation because it will allow you to update the new column metadata in the same command. See the manual for more. Library: - Spectral lines library: SiIII, OIII, CIV, NV and rest of Lyman series. - GAL_CONFIG_HAVE_WCSLIB_DIS_H: if the host's WCSLIB supports distortions. - GAL_CONFIG_HAVE_WCSLIB_MJDREF: if the host's WCSLIB reads MJDREF keyword. - gal_cosmology_velocity_from_z: Calculate velocity from redshift. - gal_cosmology_z_from_velocity: Calculate redshift from velocity. - gal_data_array_ptr_calloc: Allocate array of pointers to gal_data_t - gal_data_array_ptr_free: Free all the datasets within the array and itself. - gal_fits_key_list_title_add: Add a title key word to the list. - gal_fits_key_list_title_add_end: Add a title key word to the list's end. - GAL_INTERPOLATE_NEIGHBORS_METRIC_RADIAL: Radial metric for interpolation. - GAL_INTERPOLATE_NEIGHBORS_METRIC_MANHATTAN: Mahattan distance. - GAL_INTERPOLATE_NEIGHBORS_METRIC_INVALID: For error-handling/completeness. - GAL_INTERPOLATE_NEIGHBORS_FUNC_MIN: Use minimum for interpolation. - GAL_INTERPOLATE_NEIGHBORS_FUNC_MAX: Use maximum for interpolation. - GAL_INTERPOLATE_NEIGHBORS_FUNC_MEDIAN: Use median for interpolation. - GAL_INTERPOLATE_NEIGHBORS_FUNC_INVALID: for error-handling/completeness. - gal_kdtree_create: Create a k-d tree for optimal spatial searching. - gal_kdtree_nearest_neighbour: Find the nearest neighbor using a k-d tree. - gal_statistics_histogram2d: Generate 2D histogram from two columns. - GAL_WCS_FLTERROR: Limit to identify a floating point error for WCS. - gal_wcs_write: Write the given WCS into a FITS extension with no data. - gal_wcs_clean_errors: clean major WCS components from errors specified by the FITS keyword 'CRDER', or floating point errors. - gal_wcs_distortion_from_string: Return distortion string/name from ID. - gal_wcs_distortion_to_string: Return distortion ID from string/name. - gal_wcs_distortion_identify: Identify the distortion of given WCS. - gal_wcs_distortion_convert: Convert between various WCS distortions. ** Removed features ** Changed features Arithmetic: - The 'pow' operator can also accept integer inputs. This also applies to column arithmetic in Table. MakeProfiles: - The status of every created profile (along with the number of remaining profiles) is no longer printed when there are more than 50 profiles. This is done because printing itself can slow down the program an in a general/automated script this info is redundant. Table: --catcolumnfile ('-L') is new name for '--catcolumn' ('-C'). --catcolumnhdu is new name for '--catcolhdu' (short option name hasn't changed). Library: - gal_fits_key_list_add: new 'ufree' argument to optionally free units. - gal_fits_key_list_add_end: similar to 'gal_fits_key_list_add'. - gal_interpolate_neighbors: new name for gal_interpolate_close_neighbors. - gal_statistics_outlier_bydistance: new name for the old 'gal_statistics_outlier_positive'. It can now use the same algorithm for negative outliers with a new argument. - gal_txt_write: Now accepts a new argument for keyword lists. - gal_type_string_to_number: Numbers ending in '.' or '.0' will be parsed as floating point. Until now, it would only parse numbers as floating point if they had non-zero decimals. ** Bugs fixed bug #58434: MakeCatalog crash when ordering is required and no usable pixels. bug #58455: Timezone is not portable and uses flag instead of seconds. bug #58696: Warp with --centeroncorner --scale making wrong size. bug #58774: Warp' s output on a cube is a 2D image or wrong size. bug #58809: NoiseChisel not removing negative outlier tiles. bug #58833: Segment crashes when detection map has blank pixels. bug #58835: Floating point errors when comparing pixel scale in Crop. bug #58898: Plain text string columns touching next, clear first character. bug #58901: Blank values for non-standard integer types in FITS tables. bug #58974: WCS conversion not reasonable on processed TPV data. bug #59019: FITS Table crash when TFORM comes before TNULL. * Noteworthy changes in release 0.12 (library 10.0.0) (2020-05-20) [stable] ** New features Arithmetic: - New 'quantile' operator for coadding datasets. - New 'size' operator to report length of dataset in requested dimension. - When '--wcsfile' is given the value 'none', output will not have any WCS. CosmicCalculator: --listlines: list the pre-defined spectral line wavelengths and names (which you can use with the '--obsline' and '--lineatz' options). This is convenient when you forget the specific name of the spectral line used within Gnuastro. Crop: --polygon: can now also crop concave polygons (when atleast one inner angle is more than 180 degrees). Concave polygons occur a lot in deep astronomical imaging: in the shape of the deepest regions. --polygonsort: Sort the given set of vertices to the '--polygon' option. For a concave polyton, the sorting will be correct, but for a convex polygon, there is no unique solution/sorting, so it may not be what you expect, see the manual. Fits: --datasum: Calculate and print the given HDU's "datasum" to stdout. --datetosec: Can also account for 'Z' in the end of the date-time string. According to 'https://www.w3.org/TR/NOTE-datetime', a 'Z' effectively means no time zone, or UTC time (which is the default in FITS). It still doesn't account for time zone hours of the w3.org standard. MakeCatalog: --sigmaclip: define sigma-clipping parameters for the new '--sigclip-*' columns. --forcereadstd: Read the standard deviation image even if not needed by any columns. This is useful when you want the surface brightness limit, but don't need any error-related columns. New output columns: --sigclip-number: Number of sigma-clipped pixels in object/clump. --sigclip-median: Sigma-clipped median of pixels in object/clump. --sigclip-mean: Sigma-clipped mean of pixels in object/clump. --sigclip-std: Sigma-clipped standard deviation of pixels in object/clump. Table: --equal: Can now work on columns with string type also. --notequal: Can now work on columns with string type also. --polygon: Polygon to use in '--inpolygon' or '--outpolygon'. --inpolygon: Select rows that are inside the polygon of '--polygon'. --outpolygon: Select rows that are outside the polygon of '--polygon'. --catcolumn: Concatenate tables by column (keeping number of rows fixed). --catcolhdu: Specify the HDU/extension of the FITS files of --catcolumn. - New operators in column arithmetic: - 'ra-to-degree': Convert Right Ascension (HH:MM:SS) to degrees. - 'dec-to-degree': Convert Declination (DD:MM:SS) to degrees. - 'degree-to-ra': Convert degrees to Right Ascension (HH:MM:SS). - 'degree-to-dec': Convert degrees to Declination (HH:MM:SS). - 'distance-flat': Distance between two points, assuming flat space. Library: - GAL_SPECLINES_INVALID_MAX: Total number of spectral lines, plus 1. - GAL_ARITHMETIC_OP_QUANTILE: operator for 'gal_arithmetic'. - gal_txt_trim_space: trim white space before and after a string. - gal_polygon_is_convex: identify if a polygon is convex or concave. - gal_polygon_is_inside: if point is inside polygon (convex or concave). - gal_polygon_is_counterclockwise: check if polygon is counter-clockwise. - gal_polygon_to_counterclockwise: convert to counter-clockwise if it isn't. - gal_polygon_vertices_sort: un-ordered vertices to concave/convext polygons. - gal_units_extract_decimal: Extract numbers from strings like "A:B:C". - gal_units_ra_to_degree: Convert RA (HH:MM:SS) to degrees. - gal_units_dec_to_degree: Convert Dec (DD:MM:SS) to degrees. - gal_units_degree_to_ra: Convert degrees to RA (DD:MM:SS). - gal_units_degree_to_dec: Convert degrees to Dec (DD:MM:SS). ** Removed features ** Changed features All programs and libraries: --minmapsize: Gnuastro's programs no longer attempt to write memory-mapped files under '.gnuastro'. They will only attempt to write them under the '.gnuastro_mmap' directory. Until now, when an internal array needed to be memory-mapped, Gnuastro's programs (through the 'pointer.h' library) would first try writing the mmap files in the '.gnuastro' directory. When it failed it would attempt writing in the '.gnuastro_mmap' directory. However, '.gnuastro' is also used to store configuration files (which are hand-written and thus valuable). Mixing the two types of source (configuration files) and automatically generated (memory-mapped) files is very problematic. - FITS ASCII tables: When a column has a floating point type, but its ASCII string can't be parsed as a number, it will be read as a NaN. Until now, the corresponding program/library would abort, printing the problematic string and its location. Crop: --polygon: by default it will no longer attempt to sort the polygon vertices, sorting can be requested with the new '--polygonsort' option. --polygonout: is the new name for '--outpolygon'. Having 'polygon' at the start of the option name, makes it easier to find in the help list and also to understand generally. MakeCatalog: - Until now, if no standard deviation image was requested, MakeCatalog wouldn't include any surface brightness limit metadata in its output. Now, those two lines are filled, but with a notice on the cause (that there is no standard deviation image), and suggesting solutions. NoiseChisel: - Until now, when NoiseChisel didn't detect any pixels, it just printed a message and wouldn't not make any output file. This was very inconvenient in general scripts. From now on, in this scenario, an output file will be created and the detection map will only have a value of zero. As a result, the Sky and Sky standard deviation extensions will be measured over all the tiles. Table: - In Column arithmetic, when columns must be specified by their number, that number should be distinguished with a '$' before it (for example '$1' means the first column). Until now, this character was 'c', but the new identifying character is very similar to AWK, allowing easier adoption and is also more clear. It is just important to put the total 'arith' string within single quotes, not double quotes. - Operators: - 'distance-on-sphere': New name for old `angular-distance' operator. Library: - gal_polygon_is_inside_convex: new name for 'gal_polygon_pin'. ** Bugs fixed bug #57300: MakeCatalog memory crash when input dataset has units. bug #57301: MakeCatalog using river sum instead of mean times by clump area. bug #57921: Arithmetic's interpolation operator not reading metric. bug #57989: Warp not accounting for translation in pixel grid. bug #57995: Fits lib's date to second function affected by host's timezone. bug #58315: Some NaNs with sigma-clip operators in Arithmetic and one input. bug #58371: Table crashes with a commented newline in the columns. * Noteworthy changes in release 0.11 (library 9.0.0) (2019-11-25) [stable] ** New features Book: - The "General program usage tutorial" now has a section on how to write scripts effectively to automate your analysis. Arithmetic: - The new 'add-dimension' operator will stack the popped operands in a higher-dimensional dataset. For example to build a 3D cube from individual 2D images/slices. --onedonstdout: when the output is one-dimensional, print the values on the standard output, not into a file. BuildProgram: - Will use common environment variables like LDFLAGS, CPPFLAGS and CC to help in customizing the build of your program. --cc: custom C compiler to use. Until now, 'gcc' was hard-coded into the source and there was no way to choose a custom C compiler. --noenv: With this option, no environment variables will be read. ConvertType: - New 'viridis' colormap (value for the '--colormap' option). This is the default colormap of Python's Matplotlib, and is available in many other plotting tools like LaTeX's PGFPlots. Convolve: - Spatial domain convolution now possible on 3D data cubes (with a 3D kernel). CosmicCalculator: --lineatz: return the observed wavelength of a line if it was emitted at the redshift given to CosmicCalculator. You can either use known line names, or directly give a number as any emitted line's wavelength. MakeCatalog: - Catalogs from 3D inputs now available, with the following new options, see book for more. --spectrum: label's spectrum (across the third dimension). --z: Flux-weighted position in 3rd dimension. --geoz: Geometric center in third FITS axis. --minz: Minimum third FITS axis position. --maxz: Maximum third FITS axis position. --clumpsz: Flux weighted center of all clumps in object in 3rd dim. --clumpsgeoz: Geometric center of all clumps in object in 3rd dim. --w3: Flux weighted center in third WCS axis. --geow3: Geometric center in third WCS axis. --clumpsw3: Flux wheighted center of all clumps in 3rd dim. --clumpsgeow3: Geometric center of all clumps in 3rd dim. --areaxy: Projected area in first two dimensions. --geoareaxy: Projected geoarea in first two dimensions. --inbetweenints: output will contain one row for all integers between 1 and the largest label in the input (irrespective of their existance in the input image). This was the default/only behavior of MakeCatalog until now. However, there are situations where the labeled input image integers may not be contiguous. For example if the input's only labeled pixel values are 11 and 13 from this release MakeCatalog's output will only have two rows. If you want the old behavior (of one row per integer, even if its not in the image), you can use this option. MakeProfiles: - Can produce mock ellipsoids in a datacube (using X-Z-X Euler angles for 3D orientation), the following options have been added, see the book for more details. --p2col: Second Euler angle (X-Z-X order). --p3col: Third Euler angle (X-Z-X order). --q2col: Axis ratio (major/dim3 in 3D). - The '--kernel' option can build 3D kernels, see the description of this option in the book for examples and details on how to run it. Match: - Matching of catalogs now possible using 3 coordinates (on catalogs generated from 3D data cubes), see book for more. NoiseChisel: - arXiv:1909.11230 added in papers to cite (with the '--cite' option): this paper describes the major changes made to NoiseChisel in the last 10 stable releases since the 2015 paper, most importantly how Segment has been separated and the new growth strategy. It is therefore necessary to cite it along with the initial 2015 paper when using NoiseChisel. Segment: - arXiv:1909.11230 added in papers to cite (with the '--cite' option): this paper describes why Segment has been separated from NoiseChisel and some important updates to it compared to the 2015 paper, it is therefore necessary to cite it along with that paper when using Segment. Statistics: --contour: compute a contour plot which can be directly fed into the PGFPlots package of LaTeX for plotting the contours. Support for more formats will be added based on the need/request. Table: --equal: Output only rows that have a value equal to the given value in the given column. For example '--equal=ID,2,4,5' will select only rows that have a value of 2, 4 and 5 in the 'ID' column. --notequal: Output only rows that have a different value compared to the values given to this option in the given column. - Column Arithmetic operators: - 'angular-distance': a new operator to easily find the angular distance (along a great circle) between points in various table columns, or the distances of all the points in the table rows with a fixed point. See the book for examples and better explanation. Library: - gal_binary_connected_indexs: store indexs of connected components. - gal_blank_remove_realloc: Remove blanks and shrink allocated space. - gal_box_bound_ellipsoid_extent: Extent of 3D ellipsoid. - gal_box_bound_ellipsoid: Bounding box for a 3D ellipsoid. - gal_statistics_unique: Return unique (non-blank) elements of the input. ** Removed features ** Changed features ** Bugs fixed bug #56736: CosmicCalculator crash when a single value given to --obsline. bug #56747: ConvertType's SLS colormap has black pixels which should be orange. bug #56754: Wrong sigma clipping output when many values are equal. bug #56999: Compilation error on macOS 10.9 not recognizing AT_FDCWD. bug #57057: BuildProgram not using environment LDFLAGS or CPPFLAGS. bug #57101: Crop segmentation fault when no overlap exists in image-mode. bug #57164: MakeCatalog crashes when a label isn't in the dataset. bug #57180: MakeCatalog reporting infinity S/N when --instd isn't an image. bug #57200: Generated pkgconfig must request wcslib, not wcs. bug #57293: NaN value for brightness-related columns when values have NaN. * Noteworthy changes in release 0.10 (library 8.0.0) (2019-08-03) [stable] ** New features Installation: - With the the following options at configure time, its possible to build Gnuastro without the optional libraries (even if they are present on the host system): '--without-libjpeg', '--without-libtiff', '--without-libgit2'. All programs: - When an array is memory-mapped to non-volatile space (like the HDD/SSD), a warning/message is printed that shows the file name and its size. Later, when its deleted, a warning/message is also printed, informing you that it has been deleted. These warnings can be very useful when you actually have enough RAM, but forget to increase the '--minmapsize' value (therefore significantly slowing down the program). When you don't have enough RAM, but don't want to be annoyed with the warnings, you can use the new '--quietmmap' option to disable them. Arithmetic: - 'unique' operator removes all duplicate (and blank) elements from the dataset and returns a single-dimension output, containing only the unique values in the dataset. Crop: - Can also crop 3D datasets (data cubes). A 3D crop has the same syntax as the old 2D mode, only when the dataset is 3D, three coordinates (values, ranges or catalog-columns) should be given to the relevant option. Just note that '--polygon' crops are still not supported in 3D. CosmicCalculator: --obsline: alternative way to set the used redshift. With this option instead of explicitly giving the redshift, you can give a rest-frame and observed wavelength and the redshift to use will be calculated internally. For many lines, it is possible to give the line name instead of its rest-frame wavelength. For example '--obsline=lyalpha,6000' will use the redshift where the Lyman-alpha line has been shifted to 6000 Angstroms. --usedredshift: Print the used redshift as a "Specific calculation" (in line with other single-valued calculations). Fits: --primaryimghdu: Copy/cut the given image HDU to the zero-th/first HDU of the output file that doesn't yet exist. Statistics: --sigclip-number, --sigclip-median, --sigclip-mean, --sigclip-std: Do sigma-clipping and only print the desired value as a single-value measurement. Until now sigma-clipping results included a lot of visually useful information, which also made automatic usage of results hard. These options fix this issue. Please see the example in the book under '--sigclip-median' for a nice use case. Table: - Column arithmetic. It is now possible to apply many operations on the input table columns before printing them in the output. Similar to Arithmetic, but on table columns. The operators and notation is just like the Arithmetic program. See the "Column Arithmetic" section of the book for a detailed discussion and several examples. - WCS to Image coordinate conversion with 'wcstoimg' and 'imgtowcs'. For example if the input catalog has at least an 'ID' column and two 'RA' and 'DEC' columns, the set of options below will produce 5 columns where the last two columns are the image coordinates for each row based on the WCS in 'a.fits': '-cID,RA,DEC -c"arith RA DEC wcstoimg" --wcsfile=a.fits' --head: Only output the given number of rows from the top of columns. --tail: Only output the given number of rows from the bottom of columns. Library: - New 'speclines.h' library functions and macros related to spectral lines. It has many macros with line wavelengths, and several functions for using them in combination with their names. - list.h: Functions to return the last element in linked lists. For example 'gal_list_sizet_last' or 'gal_list_data_last'. - gal_arithmetic_operator_string: Return operator string from code. - gal_arithmetic_set_operator: Return operator code from string. - gal_blank_initialize_array: Initialize an array with blank values. - gal_dimension_remove_extra: Remove extra (length 1) dimensions. - gal_list_data_to_array_ptr: Make an array of pointers from the list. - gal_fits_img_info_dim: Only return the size information of a dataset. - GAL_TYPE_INT: Corresponding to respective width based on system. - GAL_TYPE_UINT: Corresponding to respective width based on system. - GAL_BLANK_INT: Blank value for 'int' (can be 16-bit or 32-bit). - GAL_BLANK_UINT: Blank value for unsigned 'int' (can be 16-bit or 32-bit). ** Removed features ** Changed features Installation: - Better './configure' tests (using Gnulib's 'AC_LIB_HAVE_LINKFLAGS') to avoid some crashes during 'make' when the host had multiple conflicting versions of some dependencies (GSL in particular). Arithmetic: - The output of co-adding operators is no longer the same type as the input in general. The output of the 'min' and 'max' operators are still the same type as the input. However the 'number' and 'sigclip-number' operators will output an unsigned 32-bit integer type and the rest ('sum', 'mean', 'std', 'median', 'sigclip-median', 'sigclip-mean' and 'sigclip-std') return 32-bit floating point datasets MakeCatalog: - When a clumps catalog is requested, MakeCatalog will automatically deduce the total number of clumps (at a small cost in performance). Until now, it was mandatory for the clumps label dataset to contain the total number of clumps in the 'NUMLABS' keyword. Library: - gal_statistics_outlier_flat_cfp: Improved implementation with new API. - New 'quietmmap' argument added to the following functions (as the argument following 'minmapsize'). For more, see the description above of the new similarly named option to all programs: 'gal_array_read' 'gal_array_read_to_type', 'gal_array_read_one_ch', 'gal_array_read_one_ch_to_type', 'gal_data_alloc', 'gal_data_initialize', 'gal_fits_img_read', 'gal_fits_img_read_to_type', 'gal_fits_img_read_kernel', 'gal_fits_tab_read', 'gal_jpeg_read', 'gal_label_indexs', 'gal_list_data_add_alloc', 'gal_match_coordinates', 'gal_pointer_allocate_mmap', 'gal_table_read', 'gal_tiff_read' and 'gal_txt_image_read' Book: - The two larger tutorials ("General program usage tutorial", and "Detecting large extended targets") have been broken into subsections for easier readability. - The "Hubble visually checks and classifies his catalog" tutorial has been removed because it didn't come with a dataset, so it was hard for people to use. Also, all its operations were already present in the general tutorial. ** Bugs fixed bug #56195: astscript-sort-by-night crashing because of AWK. bug #56246: Single-valued measurement must abort with no value in Statistics. bug #56256: Segmentation fault when reading plain text array/image. bug #56257: ConvertType: Values not preserved when converting text to FITS. bug #56299: CosmicCalculator fails at z=0. bug #56324: Column metadata not usable when input is from pipe/stdin. bug #56424: Warp crashes with empty string given to options. bug #56480: Segfault in statistics library's histogram function. bug #56641: MakeProfile's center position changes based on precision. bug #56635: Update tutorial 3 with bug-fixed NoiseChisel. bug #56662: Converting -R to -Wl,-R causes a crash in configure on macOS. bug #56671: Bad sorting with asttable if nan is present. bug #56709: Segment crash when input has blanks, but labels don't. bug #56710: NoiseChisel sometimes not including blank values in output. * Noteworthy changes in release 0.9 (library 7.0.0) (2019-04-17) [stable] ** New features All programs: --checkconfig: print the names and values given to options as they are parsed on the command-line or in various configuration files (the configuration file name is also printed). This can be very useful in debugging (finding which configuration file is responsible for a given option's value). Arithmetic: - The new 'tofile-' and 'tofilefree-' operators will save the top operand into a file. They can be very handy in debugging/understanding an Arithmetic command (especially as it gets complicated), or to produce multiple files/extensions with a single call to Arithmetic. - Four new operators have been added to allow coadding multiple datasets into one using sigma-clipping: 'sigclip-number', 'sigclip-mean', 'sigclip-median', and 'sigclip-std'. These are very useful when several inputs have strong outliers that affect the median, or the mean is required. - Multithreaded operation for the following operators that combine/co-add multiple inputs into one output with same size: 'min', 'max', 'number', 'sum', 'mean', 'std', 'median', 'sigclip-number', 'sigclip-median', 'sigclip-mean', 'sigclip-std'. --wcsfile and --wcshdu: these two options can be used to specify a different file for reading the WCS of the output. This is useful when the default (the WCS of the first dataset that is read) is not the required one. --interpmetric: new option that is necessary with the 'interpolate-medianngb' operator. For more, see the description of this option in NoiseChisel. Fits: - Add "title" to group FITS keywords with '--write=/,"title name". This "title" is composed of two keyword records/lines: a blank one (all whitespace), followed by another starting with '/' and ending in any string given to this option. This visually separates the keywords and acts as a title. Classifying the keywords into contextually similar groups greatly helps in visual inspection and is encouraged. - Calculate and write 'CHECKSUM' and 'DATASUM' integrity keywords into the specified header using '--write=checksum' (for both) or '--write=datasum' (only for 'DATASUM'). --datetosec: Convert the FITS date format (old or new) to number of seconds since since the Unix epoch time (1970-01-01,00:00:00). The FITS date format (for example 'YYYY-MM-DDThh:mm:ss') is hard to use for automatic processing (requires calendar peculiarities like number of days in each month, or leap years and etc). With this option a single integer is returned that can be used for example to sort FITS files by date keywords without worrying about calendar peculiarities. --verify: confirm if the 'DATASUM' and 'CHECKSUM' keyword values agree with the specified HDU's content and/or data. --copykeys: Copy several keyword records (in a given range) from one FITS HDU/extension into another (possibly in another file). --outhdu: The name/number of the output HDU (for '--copykeys'). Match: - All the columns from one of the input catalogs can now be merged with any of the columns of the second using the special '_all' name of '--outcols'. For example the output of '--outcols=a_all,b5' will contain all the columns from the first input and the 5th column of the second input. This greatly simplifies the merging of different table columns into one. --coord: manually specify coordinates to match on the command-line. Until now, if you only wanted to make check a specific coordinate's matching with a catalog. It was necessary to make a single-row catalog as a file and feed that into Match. With this option, you can now specify the coordinates to match against another catalog with the command-line. NoiseChisel: --interpmetric: Set the metric to use to identify the nearest neighbors for tile interpolation (quantile threshold, initial Sky, and final Sky). Until now only the manhattan/taxicab metric was available, which is fast, but could cause 45-degree lines in the interpolation. From this version, with this option, its also possible to use the radial distance (which is now the default). Just note that if there are many tiles over the image, the radial distance will be slower. --snthresh: Manually set the signal-to-noise ratio of true pseudo-detections. With this option, NoiseChisel will not attempt to find pseudo-detections over the noisy regions of the dataset, but will directly go onto applying the manually input value. - Several new options have been added to remove NoiseChisel's dependence on values that were hard-coded in its source and thus not modifiable at run-time by the user. To allow full configurability, these steps can also be configured by the user. --pseudoconcomp: allows setting the connectivity (4 or 8, in a 2D image) to define separate pseudo-detections. If its stronger, pseudo-detections that are touching on the corner will be identified as one. --dopening: The number of openings to do after applying '--dthresh'. --dopeningngb: The connectivity (4 or 8) to use for '--dopening'. Statistics: --interpmetric: Similar to NoiseChisel. Table: --range: Limit the output rows to those with a value within the given numeric range with this format: '--range=COL,low,high'. This is very useful when only certain rows of the input are required not the output. The advantage over piping to AWK is that you can save the output directly to FITS (preserving the metadata). See the book for more. --sort: Sort the output rows based on the value of the given column in ascending order. --descending: When called with '--sort', will arrange the output rows in descending order. Installed scripts: With this release, Gnuastro also installs Bash scripts for common higher-level usage of (possibly multiple) programs. These scripts have a 'astscript-*' name, to easily show up on the command-line as Gnuastro executables with the other Gnuastro programs, but are identifiable from them. They support options just like the programs (which can be listed with '--help'). Please see the new "Installed scripts" chapter of the book for more. - astscript-sort-by-night: New Gnuastro executable, using Gnuastro's Fits program to identify files with dates in the same night (possibly spanning two calendar dates). Library: GAL_BLANK_LONG: new macro for the 'long' type blank value. GAL_BLANK_ULONG: new macro for the 'unsigned long' type blank value. gal_blank_number: Return the number of blank elements in a dataset. gal_dimension_dist_radial: Radial distance between two coordinates. gal_fits_key_date_to_struct_tm: FITS date format to C broken-down time. gal_fits_key_date_to_seconds: FITS date format to Unix epoch time. gal_qsort_index_single_TYPE_i: Set of functions to sort indexs ascending. gal_qsort_index_single_TYPE_d: Set of functions to sort indexs descending. gal_statistics_outlier_cumulative: Uses flatness of the cumulative distribution to find outliers. gal_table_list_of_indexs: returns the list of indexs matching columns. gal_type_is_int: to see if we have an integer (any width, any sign). ** Removed features ** Changed features Arithmetic: - 'num' operator is renamed to 'number'. - 'numvalue' operator is renamed to 'numbervalue'. - '--dontdelete' will append the output to any existing file. Note that this change is only in Arithmetic, other programs will still just complain and abort. ConvertType: --forcemin & --forcemax: until now, '--flminbyte' and '--flmaxbyte' were used to force the range of conversion to color channels (even if the range is beyond the limits of the dataset). With the introduction of color maps in Gnuastro 0.8, it is also necessary to force a range on non-byte datasets. It is thus necessary to use a more generic name. MakeCatalog: --std: Until now, this option would measure the mean standard deviation under the label. But this is not a statistically meaningful measure for the Sky standard deviation and could be incorrectly used. From now on, this option measures the square root of the mean variance, or root mean square (the correct definition of the Sky standard deviation). NoiseChisel: --ignoreblankintiles: Until now '--ignoreblankinsky', would specify if blank values should also be written into the tiled Sky and Sky standard deviation outputs. But NoiseChisel can optionally produce many more tiled outputs (for example with '--checkqthresh'). So the option was renamed to '--ignoreblankintiles' to highlight that the status of blank elements can be set in all tiled outputs. Statistics: --ignoreblankintiles: similar to same option in NoiseChisel. Table: --colinfoinstdout: now corresponds to the '-O' short option. Until this version, the '-s' short option was used for it. But with the new '--sort' option, '-s' may cause confusion. Library gal_arithmetic: new argument: number of threads to use (when relevant). gal_eps_write: new argument: optional bit-optimization with 'dontoptimize'. gal_pdf_write: new argument: optional bit-optimization with 'dontoptimize'. ** Bugs fixed bug #55313: Fits program writing --write values in reverse order bug #55333: Fits program crash when --write or --update have no value. bug #55439: Arithmetic segmentation fault when reusing dataset name. bug #55478: Memory mapping crashes when .gnuastro is not writable. bug #55491: NoiseChisel crash when no tiles good for quantile thresholding. bug #55544: Arithmetic's output WCS with where operator is not as expected. bug #55740: Diamond shapes in nearst-ngb interpolation affecting NoiseChisel. bug #55763: Crop not keeping Blank pixels on unsigned types. bug #55844: WCS library (and thus all programs) can't deal with CROTAi values. bug #55845: Crash when necessary column not found from standard input. bug #55988: MakeProfiles segfault when input catalog has no rows. bug #56001: Bad tile sizes with only one tile and small remainder. bug #56048: Crash when WCS cannot be written to FITS file. * Noteworthy changes in release 0.8 (library 6.0.0) (2018-12-28) [stable] ** New features All programs: - Input files and parameters written as keywords in the first extension of output when it is FITS. This is only relevant for some programs, (for example not the Fits or Table programs). - Standard input (for example from pipes) is now available to feed input to all programs that accept plain text input (ConvertType, Convolve, Match, MakeProfiles, Statistics, Table). - Updated acknowledgment statement (output of '--cite' option). Arithmetic: --onedasimage: write output as an image if it has one dimension, not table. ConvertType: --colormap: color map to display a single-channel dataset (for example FITS image) with a range of colors in formats that support color (like JPEG, or PDF). Until now, the only available mapping was grayscale. Now "Hue, Saturation, Value" (HSV) and "SLS" (from SAO DS9) colormaps are also supported. --rgbtohsv: Convert the RGB input channels to HSV (when the output is in FITS format). Convolve: - Convolves 1D arrays (table columns, for example spectra) also. Therefore two new options have been added to it: '--column' ('-c', similar to other programs that read table columns), and '--kernelcolumn' (to specify the column of the kernel in its own file/extension). Fits: --numhdus: prints the number of HDUs in the given FITS file. NoiseChisel: - New outlier identification algorithm for quantile thresholds. This is very useful when there are extended and bright sources in the dataset: the tiles containing very faint signal that pass the general pixel-value distribution test due to the flatness of the extended profiles, can be identified and removed as outliers in comparison with the other passed tiles. The outlier finding algorithm ('gal_statistics_outlier_positive': a new library function) uses the distribution of distances between the sorted elements and is configured with these options. --outliersclip: Sigma-clipping parameters for the process. --outliersigma: Multiple of sigma to define an outlier. --blankasforeground: Treat blank elements as foreground (regions above the threshold) in the binary processing steps: initial erosion and opening as well as the filling holes and opening of pseudo-detections. From this version, by default, blank elements in the dataset are considered to be background, so if a foreground pixel is touching it, it will be eroded. This option is irrelevant if the datasets contains no blank elements, but can help remove false positives that are touching blank elements. --holengb: allows defining the connectivity of the holes that are filled when defining pseudo-detections. Until now, this was hard-wired into the code (=8) and not modifiable at run-time. --skyfracnoblank: Ignore blank pixels when estimating the fraction of undetected pixels for Sky estimation. NoiseChisel only measures the Sky over the tiles that have a sufficiently large fraction of undetected pixels. This is done to decrease the bias caused by faint un-detected wings of bright galaxies or stars, see description of '--minskyfrac' for more. Until now the reference for this fraction was the whole tile size (irrespective of how many blank elements it contains). With this option, it is now possible to ask for ignoring blank pixels when calculating the fraction. This is useful when blank/masked pixels are distributed across the image. Statistics: - If an input table has only one column, Statistics won't complain and abort when no '--column' ('-c') is given: there is only one column to use anyway, so it will be used. In the absence of which column to use, it will still complain and abort if the input has more than one column. - Sky estimation: new outlier estimation algorithm similar to NoiseChisel. - Input can be given using the standard input (for example a pipe). Library: - gal_blank_flag_apply: Set all flagged/masked elements to blank. - gal_fits_key_list_reverse: Reverse the given list of FITS keywords. - gal_fits_key_write_title_in_ptr: Write a two-line title FITS keyword. - gal_fits_key_write_in_ptr: New name of gal_fits_key_write. - gal_fits_key_write_version_in_ptr: old gal_fits_key_write_version. - gal_fits_key_write_config: write key list and version as config header. - gal_statistics_outlier_positive: Find the first positive outlier. - gal_txt_stdin_read: Read lines in standard input into a list of strings. ** Removed features NoiseChisel: --mirrordist: mean quantile is now used (not mode), see changes below. --qthreshtilequant: removed due to new outlier rejection algorithm. Statistics: --mirrordist: mean quantile is now used (not mode), see changes below. ** Changed features Arithmetic: - If the output has one dimension, it will be written as a table, not a FITS image/array. This can be changed with the new '--onedasimage' option. Convolve: - The short option for '--numchannels' is now '-n'. Until now, it was '-c', but that would conflict with the short option used for '--column' in all the other programs that also read from a table. MakeProfiles: --mergedsize: new name for the old '--naxis' option. Since the option names and values are now written into the FITS header of the output, this option's name would get confused with the mandatory FITS keyword 'NAXIS'. NoiseChisel: - Until now, the mode's quantile was used to identify tiles with no significant signal. But from this version, the mean's quantile in each tile is used instead. The reason is that the mean is more sensitive to outliers (signal). Therefore the old '--modmedqdiff' is now called '--meanmedqdiff' . Statistics: --meanmedqdiff: new name for '--modmedqdiff'. Similar to NoiseChisel. Library: - gal_array_read: list of strings (from standard input) acceptable. - gal_array_read_to_type: list of strings (from stin) acceptable. - gal_array_read_one_ch: list of strings (from stdin) acceptable. - gal_array_read_one_ch_to_type: list of strings (from stdin) acceptable. - gal_data_copy_to_allocated: Also copies string metadata (e.g., name). - gal_fits_key_write: filename and hdu instead of FITS pointer. - gal_fits_key_write_version: filename and hdu instead of FITS pointer. - gal_fits_key_write_filename: write at the top or end of the input list. - gal_statistics_outlier_positive: Window-size is now adjustable (new argument). - gal_table_info: list of strings (from stdin) acceptable. - gal_table_read: list of strings (from stdin) acceptable. - gal_txt_table_info: list of strings (from stdin) acceptable. - gal_txt_image_info: list of strings (from stdin) acceptable. - gal_txt_table_read: list of strings (from stdin) acceptable. - gal_txt_image_read: list of strings (from stdin) acceptable. ** Bugs fixed bug #54493: Warp crashes when type isn't set. bug #54526: Invalid r, q and truncation of point profiles in MakeProfiles. bug #54579: NoiseChisel pseudo-detection failure when dataset is negative. bug #54782: Segment's check image not removing sky clumps some tiles. bug #54810: Arithmetic crash when previously named operand renamed. bug #55025: MakeCatalog's '--prepforconv' option being ignored. bug #55079: Blank EPS or PDF page when width options not given. bug #55157: No sanity check on values given to Crop's --section. bug #55295: Crash when more than one collapse operator called. bug #55298: Arithmetic reading 255 on command-line as blank. * Noteworthy changes in release 0.7 (library 5.0.0) (2018-08-08) [stable] ** New features Installation: --enable-debug: debugging flags, no optimization, no shared libraries. --enable-check-with-valgrind: Run 'make check' tests within Valgrind. Arithmetic: - 'set-A': Set a name ('A' in this case) for the popped dataset. This allows only reading the dataset it into memory once and possibly using it many times. - 'fill-holes': Flip background (0) pixels surrounded by foreground (1). - 'collapse-sum': collapse/remove a dimension by summing over it. - 'collapse-min': collapse/remove a dimension by using minimum value. - 'collapse-max': collapse/remove a dimension by using maximum value. - 'collapse-mean': collapse/remove a dimension by averaging over it. - 'collapse-number': Number of elements included in the collapse. CosmicCalculator: - Default cosmology set to Plank 2018 results (Paper VI). MakeCatalog: --minx: minimum position along first FITS axis. --maxx: maximum position along first FITS axis. --miny: minimum position along second FITS axis. --maxy: maximum position along second FITS axis. Table: --colinfoinstdout: column information when writing to standard output. Library: - gal_array_name_recognized_multiext: If format contains multiple datasets. - gal_dimension_collapse_sum: collapse/remove a dimension by summing. - gal_dimension_collapse_mean: collapse/remove a dimension by averaging. - gal_dimension_collapse_number: collapse/remove a dimension by number. - gal_dimension_collapse_minmax: collapse/remove a dimension by extremum. - gal_wcs_remove_dimension: Remove a dimension in the given WCS structure. ** Removed features ** Changed features Crop: --checkcenter: the units of value depend on mode (image or WCS). MakeCatalog: - '--checkuplim': new name for '--checkupperlimit'. - '--brightnessnoriver': new name for '--noriverbrightness'. Library: - gal_txt_write: new 'colinfoinstdout' argument. - gal_table_write: new 'colinfoinstdout' argument. ** Bugs fixed bug #54057: Building failure due to not finding gsl_interp_steffen. bug #54063: Match tests in make check fail randomly. bug #54186: MakeCatalog's --checkupperlimit not keeping output's name. bug #54188: MakeCatalog's Upperlimit not being sigma-clipped properly. bug #54284: Crop segfault when catalog contains no data. bug #54285: make check fails if g++ not present. bug #54286: BuildProgram's configuration file, not built by default. bug #54297: No Match output when --notmatched called and no match. bug #54298: Table not writing array when there are no rows. bug #54312: Crash when CFITSIO doesn't have fits_is_reentrant function. bug #54346: Non '-I' or non '-L' strings in CPPFLAGS or LDFLAGS cause crash. bug #54358: Arithmetic's where, not ignoring blank values in condition. bug #54406: Insufficient sanity checks in mode symmetricity calculation. bug #54430: BuildProgram fails with older Libtool versions on dash. * Noteworthy changes in release 0.6 (library 4.0.0) (2018-06-04) [stable] ** New features Building: - New optional dependency: The TIFF library (libtiff). All programs: - Input image dataset(s) can be in any of the formats recognized by Gnuastro (e.g., FITS, TIFF, JPEG), provided that their libraries (optional dependencies) were present at installation time. New program: - Segment: new program in charge of segmentation over detections. This operation was previously done by NoiseChisel. NoiseChisel is now ONLY in charge of detection. Arithmetic: - erode: Erode the foreground of a binary dataset. - dilate: Dilate the foreground of a binary dataset. - filter-sigclip-mean: sigma-clipped, mean filter operator. - filter-sigclip-median: sigma-clipped, median filter operator. - connected-components: label the connected elements of the input. - invert: subtract the maximum of unsigned types (absorption to emission). - interpolate-medianngb: Interpolate (only blanks) with nearest neighbors. ConvertType: - TIFF images can also be used as input. MakeCatalog: MakeCatalog will only read the datasets necessary for the requested columns. Until now, it would read all the possible datasets and all the intermediate measurements. This is thus major improvement in memory and CPU usage. As a result, the input argument is no longer assumed to be the values file, but the object labels file. Please see the "MakeCatalog inputs and basic settings" section of the book for more. Here is the summary of the new options: --insky: Sky value as a single value or file (dataset). --instd: Sky standard deviation as a single value or file (dataset). --valuesfile: filename containing the values dataset. --valueshdu: HDU/extension containing the values dataset. --clumpscat: Make a clumps catalog also. --noclumpsort: Don't sort the clumps catalog by host object ID. --subtractsky: Subtract the given Sky from the values dataset. --variance: input standard deviation image is actually variance. --checkupperlimit: make a table for random positions and measurements. --geoarea: the number of labeled pixels (irrespective of value). --brightnesserr: error in estimating the brightness. --mean: calculate the mean pixel value within an object or clump. --median: calculate the median pixel value within an object or clump. --upperlimitsigma: position in random distribution (in units of sigma). --upperlimitquantile: position in random distribution (quantile). --upperlimitonesigma: 1sigma value of the random distribution. --upperlimitskew: (mean-median)/sigma or skewness of random distribution. NoiseChisel: - New tutorial on detecting large and extended targets. --rawoutput: only output the detection labels and Sky and its STD. --ignoreblankinsky: don't set the pixels that are blank in the input to blank in the Sky and Sky standard deviation outputs (when '--oneelempertile' is not called). --label: label the connected detections. Until now this was the default behavior. However, from this release, NoiseChisel is only in charge of detection. Segmentation is done by a new program (Segment). Since detection is ultimately just a binary operator, the default output now has an 8-bit unsigned integer type with values of 0 or 1. With this option, you can ask for it to label/count the connected detections instead of the default binary output. Statistics: --manualbinrange: histogram or CFP range can be outside of distribution. --ignoreblankinsky: similar to same option in NoiseChisel. Libraries: gal_array_read: read array from any of known formats (FITS, TIFF, JPEG,...). gal_array_read_to_type: similar to 'gal_array_read', but to given type. gal_array_read_one_ch: Read a dataset, abort if it has multiple channels. gal_array_read_one_ch_to_type: Make sure input is in one channel and type. gal_binary_label_holes: label the holes within the foreground. gal_blank_is: check to see if argument is blank in its type or not. gal_eps_name_is_eps: Returns 1 if given filename is EPS. gal_eps_suffix_is_eps: Returns 1 if given suffix is EPS. gal_eps_to_pt: Converts dataset size to PostScript points. gal_eps_write: Writes a dataset into an EPS file. gal_interpolate_1d_blank: Fill blank elements through interpolation. gal_interpolate_1d_make_gsl_spline: Allocate and initalize 'gsl_spline'. gal_jpeg_name_is_jpeg: Returns 1 if given filename is JPEG. gal_jpeg_suffix_is_jpeg: Returns 1 if given suffix is JPEG. gal_jpeg_read: Reads input JPEG image into 'gal_data_t'. gal_jpeg_write: Writes a 'gal_data_t' into a JPEG file. gal_label_grow_indexs: grow known indexs into desired areas. gal_label_watershed: apply watershed algorithm on desired region. gal_label_clump_significance: measure significance of all clumps in region. gal_pdf_name_is_pdf: Returns 1 if given filename is PDF. gal_pdf_suffix_is_pdf: Returns 1 if given suffix is PDF. gal_pdf_write: Writes a dataset into an PDF file. gal_pointer_allocate_mmap: Allocate space in a file, not in RAM. gal_qsort_index_single_d: Sort indexs of single array in decreasing order. gal_qsort_index_single_i: Sort indexs of single array in decreasing order. gal_qsort_index_multi_d: Sort indexs in multiple arrays (different threads). gal_qsort_index_multi_i: Sort indexs in multiple arrays (different threads). gal_tiff_name_is_tiff: check if name contains a TIFF suffix. gal_tiff_suffix_is_tiff: check if suffix is a TIFF suffix. gal_tiff_dir_string_read: convert a string to a TIFF directory number. gal_tiff_read: Read the contents of a TIFF "directory" to 'gal_data_t'. ** Removed features NoiseChisel: - Segmentation (and thus the options below) moved to the new Segment program: --onlydetection, --segsnminarea, --checkclumpsn, --segquant, --keepmaxnearriver, --gthresh, --minriverlength, --objbordersn, --grownclumps, --checksegmentation. --skysubtracted: no longer necessary, included in noise measuremnts. MakeCatalog: --skysubtracted: no longer necessary, included in noise measuremnts. Library: - The macros 'GAL_STATISTICS_SORTED_NOT', 'GAL_STATISTICS_SORTED_INVALID', 'GAL_STATISTICS_SORTED_INCREASING', 'GAL_STATISTICS_SORTED_DECREASING': these macros are removed because we already have the 'GAL_DATA_FLAG_SORT*'' bit-flags in 'gal_data_t'. ** Changed features Arithmetic: - filter-mean: a blank value in the input can be non-blank in the output when non-blank elements present in filter. - filter-median: similar to filter-mean. Fits: --history: can be called/written multiple times in one run. --comment: can be called/written multiple times in one run. MakeCatalog: - The 'WCLUMPS' keyword in the objects labeled image is no longer used to see if a clumps catalog should also be made. To build a clumps catalog, you can now use the '--clumpscat' option. - Estimation of noise-level is now done per-pixel over the whole label. Until now the average noise level was used. --objectsfile has been removed. The main input argument is now assumed to be the objects file. NoiseChisel: From this release, NoiseChisel is only in charge of detection and won't do segmentation any more. The new Segment program is in charge of segmentation. Many of the changes below are due to this new limited scope of NoiseChisel. --kernel: value 'none' will disable convolution. - Renamed options: --convolvedhdu ==> --chdu --wkhdu ==> --whdu --detsnminarea ==> --snminarea --checkdetsn ==> --checksn --detquant ==> --snquant - By default the output detection map is a binary image (values of 0 or 1). - With no output name, the output has a '_detected.fits' suffix. Segment: - [Previously in NoiseChisel]: For finding true clumps, the difference in the peak of the clump and the highest valued river pixel, divided by the noise standard deviation are used. Until now, the total signal-to-noise ratio was used as a criteria. In initial tests, this algorithm was much more promising in detecting clumps over strong gradients and also on flatter gradients. Table: --column: multiple columns (comma separated) can be used in one instance of this option (multiple instances of this option are still acceptable also). Libraries: gal_binary_holes_fill: new name for 'gal_binary_fill_holes'. gal_dimension_is_different: new name for 'gal_data_dsize_is_different'. gal_fits_img_read: now only reads the data not the WCS, therefore it no longer needs the last two arguments. A subsequent call to 'gal_wcs_read' can be used to read the WCS information in the file. gal_pointer_increment: new name for 'gal_data_ptr_increment'. gal_pointer_num_between: new name for 'gal_data_ptr_dist'. gal_pointer_allocate: replaces 'gal_data_malloc_array' and 'gal_data_calloc_array', through an argument you can ask for the allocated memory to be cleared or not. gal_qsort_TYPE_i: new name for gal_qsort_TYPE_increasing. gal_qsort_TYPE_d: new name for gal_qsort_TYPE_decreasing. gal_statistics_is_sorted: can now also update the bit-flags regarding the sorted nature of the input (to optimize future calls to the function). gal_statistics_quantile_function: returns 'inf' or '-inf' if the given value is smaller than the minimum or larger than the maximum of the input dataset's range. Until now, it would return blank in such cases. gal_statistics_number: the output dataset now has a 'size_t' type. Until now it was 'uint64_t'. ** Bugs fixed bug #50957: --version output not possible on Mac OS X bug #52979: Many unused result warnings for asprintf in some compilers. bug #53122: Configure time CPPFLAGS and LDFLAGS don't pass to BuildProgram. bug #53142: Crash when printing values with the '--onlyversion' option. bug #53147: NULL value of onlyversion option causing a crash. bug #53226: Match output directory ignored when making multiple files. bug #53230: Statistics program bad results on integer columns with limits. bug #53268: NoiseChisel crash when no growth is possible. bug #53295: MakeCatalog parses area larger than clump. bug #53304: NoiseChisel crash when there is no detection. bug #53312: Fits crash on keyword editing (except --delete). bug #53407: Instrumental noise in MakeNoise should be squared. bug #53424: Sigma-clipping seg-faults when there are no more elements. bug #53580: Warp crash when no WCS present. bug #53825: NoiseChisel not accounting for fully zero-valued tiles. * Noteworthy changes in release 0.5 (library 3.0.0) (2017-12-22) [stable] ** New features Manual/Book: An extended tutorial is added showing some general applications of almost all the programs. This may be a good place to get a feeling of how Gnuastro is intended to be used and some of the programs. New Program and library: Match is a new program that will match two given inputs (currently catalogs). Its output is the re-arranged inputs with the same number of rows/records such that all the rows match. The main work is also done with the new low-level 'gal_match_catalog' library function which can also be used in more generic contexts. All programs: a value of '0' to the '--numthreads' option will use the number of threads available to the system at run time. Arithmetic: The new operators 'filter-median' and 'filter-mean' can be used to filter (smooth) the input. The size of the filter can be set as the other operands to these operators. BuildProgram: The new '--la' option allows the identification of a different Libtool '.la' file for Libtool linking information. BuildProgram: The new '--deletecompiled' option will delete the compiled program after running it. CosmicCalculator: all the various cosmological calculations can now be requested individually in one line with a specific option added for each calculation (for example '--age' or '--luminositydist' for the age of the universe at a given redshift or the luminosity distance). Therefore the old '--onlyvolume' and '--onlyabsmagconv' options are now removed. To effectively use these new features, please review the "Invoking CosmicCalculator" section of the book. Fits: when an extension/HDU is identified on the command-line with the '--hdu' option and no operation is requested, the full list of header keywords in that HDU will be printed (as if only '--printallkeys' was called). MakeCatalog: physical nature agnostic WCS column names. Previously the first WCS axis was always assumed to be RA and the second DEC. So for example even if you had a spectrum (with X and wavelength as the two WCS dimensions), you would have to ask for '--ra' and '--dec'. The new '--w1' and '--w2' options are now generic and don't assume any particular type only their order in the FITS header. MakeCatalog now also uses the CTYPE and CUNIT keywords to set the names and units of its output columns. The '--ra' and '--dec' options are now just internal aliases for '--w1' or '--w2' which will be determined based on the input's CTYPE keyword. Also the new '--geow1', '--geow2', '--clumpsw1', '--clumpsw2', '--clumpsgeow1', '--clumpsgeow2' options replace the old options '--geora', '--geodec', '--clumpsra', '--clumpsdec', '--clumpsgeora', '--clumpsgeodec'. No alias is currently defined for the latter group. MakeCatalog: the new '--uprange' option allows you to specify a range for the random values around each object. This is useful when the noise properties of the dataset vary gradually and sampling from the whole dataset might produce biased results. NoiseChisel: with the new '--convolved' and '--convolvedhdu' options, NoiseChisel will not convolve the input any more and use the given dataset instead. In many cases, as the inputs get larger, convolution is the most time consuming step of NoiseChisel. With this option, you can greatly speed up your tests (to find the best parameters by varying them, for a given analysis). See the book for more information and examples. NoiseChisel: with the new '--widekernel' option it is now possible to use a wider kernel to identify which tiles contain signal. The rest of the steps (identifying the quantile threshold on the selected tiles and etc) are done on the dataset convolved with '--kernel' as they were before. Since it is time consuming, this is an optional feature. NoiseChisel: with the new '--qthreshtilequant' option, it is now possible to discard high-valued (outlier) tiles before estimating qthresh over the whole image. This can be useful in detecting very large diffuse/flat regions that would otherwise be detected as background (and effectively removed). NoiseChisel: the finally selected true detections are now grown based on signal contiguity, not by blind dilation. The growth process is the same as the growing of clumps to define objects. Only for true detections, the growth occurs in the noise. You can configure this growth with the '--detgrowquant' and '--detgrowmaxholesize'. With this new feature it is now possible to detect signal out to much lower surface brightness limits and the detections don't look boxy any more. Cosmology library: A new set of cosmology functions are now included in the library (declared in 'gnuastro/cosmology.h'). These functions are also used in the CosmicCalculator program. 'gal_table_read' can now return the number of columns matched with each input column (for example with regular expressions), a new argument has been added to allow this feature. 'gal_fits_key_img_blank': returns the value that must be used in the BLANK keyword for the given type as defined by the FITS standard. 'gal_txt_write' and 'gal_fits_tab_write' now accept an extension name as argument to allow a name for the FITS extension they write. 'gal_box_bound_ellipse_extent' will return the maximum extent of an ellipse along each axis from the ellipse center in floating point. ** Removed features Installation: The '--enable-bin-op-*' configuration options that were introduced in Gnuastro 0.3 have been removed. By managing the arithmetic functions in a better manner (a separate source file for each operator), compilation for all types (when done in parallel) takes about the same time as it took with the default (only four) types until now. MakeCatalog: '--zeropoint' option doesn't have a short option name any more. Previously it was '-z' which was confusing because '-x' and '-y' were used to refer to image coordinate positions. NoiseChisel: The '--dilate' and '--dilatengb' options have been removed. Growing of true detections is no longer done through dilation but through the '--detgrowquant' and '--detgrowmaxholesize' options (see above). ** Changed features CosmicCalculator: The redshift is no longer mandatory. When no redshift is given, it will only print the input parameters (cosmology) and abort. MakeCatalog: when the output is a FITS file, the two object and clumps catalogs will be stored as multiple extensions of a single file. Until now, two separate FITS files would be created. Plain text outputs are the same as before (two files will be created). 'gal_binary_fill_holes' now accepts a 'connectivity' and 'maxsize' argument to specify the connectivity of the holes and the maximum size of acceptable holes to fill. 'gal_fits_img_read' and 'gal_fits_img_read_to_type' now also read the WCS structure of the extension/HDU in a FITS file and have two extra arguments: 'hstartwcs' and 'hendwcs'. With these options it is possible to limit the range of header keywords to read the WCS, similar to how they are used in 'gal_wcs_read'. 'gal_txt_write', 'gal_table_write_log', 'gal_fits_tab_write' and 'gal_txt_write' don't have the 'dontdelete' argument any more. The action they take if the file already exists depends on the file: for FITS, a new extension will be added and for text, they will abort with an error. 'gal_tile_block_write_const_value' and 'gal_tile_full_values_write' now accept a new 'withblank' option to set all pixels that are blank in the tile's block to be blank in the check image. 'gal_wcs_pixel_area_arcsec2' will return NaN (instead of aborting) when input is unreasonable (not two dimensions or not in units of degrees). 'gal_wcs_world_to_img' and 'gal_wcs_img_to_world': Until now, these two WCS conversion functions would explicitly assume RA and Dec and work based on input arrays (so for example it was also necessary to give the number of elements and etc). They now accept 'gal_data_t' as input for the input coordinates, thus their API has been greatly simplified and their functionality increased. ** Bugs fixed ConvertType crash when changing values (bug #52010). Arithmetic not accounting for integer blank pixels in binary operators (bug #52014). NoiseChisel segfault when memory mapping to a file (bug #52043). CFITSIO 3.42 and libcurl crash at Gnuastro configure time (bug #52152). MakeCatalog crash in upper-limit with full size label (bug #52281). NoiseChisel leaving unlabeled regions after clump growth (bug #52327). Arithmetic crash with no input tokens (bug #52422). Libtool checks only in non-current directory (bug #52427). * Noteworthy changes in release 0.4 (library 2.0.0) (2017-09-13) [stable] ** New features All programs: '.fit' is now a recognized FITS file suffix. All programs: ASCII text files (tables) created with CRLF line terminators (for example text files created in MS Windows) are now also readable as input when necessary. Arithmetic: now has a new '--globalhdu' ('-g') option which can be used once for all the input images. MakeNoise: with the new '--sigma' ('-s') option, it is now possible to directly request the noise sigma or standard deviation. When this option is called, the '--background', '--zeropoint' and other option values will be ignored. MakeProfiles: the new '--kernel' option can make a kernel image without the need to define a catalog. With this option, a catalog (or accompanying background image) must not be given. MakeProfiles: the new '--pc', '--cunit' and '--ctype' options can be used to specify the PC matrix, CUNIT and CTYPE world coordinate system keywords of the output FITS file. MakeProfiles: the new 'distance' profile will save the radial distance of each pixel. This may be used to define your own profiles that are not currently supported in MakeProfiles. MakeProfiles: with the new '--mcolisbrightness' ("mcol-is-brightness") option, the '--mcol' values of the catalog will be interpretted as total brightness (sum of pixel values), not magnitude. NoiseChisel: with the new '--dilatengb' option, it is now possible to identify the connectivity of the final dilation. Library: Functions that read data from an ASCII text file ('gal_txt_table_info', 'gal_txt_table_read', 'gal_txt_image_read') now also operate on files with CRLF line terminators. ** Removed features ** Changed features Crop: The new '--center' option is now used to define the center of a single crop. Hence the old '--ra', '--dec', '--xc', '--yc' have been removed. This new option can take multiple values (one value for each dimension). Fractions are also acceptable. Crop: The new '--width' option is now used to define the width of a single crop. Hence the old '--iwidth', '--wwidth' were removed. The units to interpret the value to the option are specified by the '--mode' option. With the new '--width' option it is also possible to define a non-square crop (different widths along each dimension). In WCS mode, its units are no longer arcseconds but are the same units of the WCS (degrees for angles). '--width' can also accept fractions. So to set a width of 5 arcseconds, you can give it a value of '5/3600' for the angular dimensions. Crop: The new '--coordcol' option is now used to determine the catalog columns that define coordinates. Hence the old '--racol', '--deccol', '--xcol', and '--ycol' have been removed. This new option can be called multiple times and the order of its calling will be used for the column containing the center in the respective dimension (in FITS format). MakeNoise: the old '--stdadd' ('-s') option has been renamed to '--instrumental' ('-i') to be more clear. MakeProfiles: The new '--naxis' and '--shift' options can take multiple values for each dimension (separated by a comma). This replaces the old '--naxis1', '--naxis2' and '--xshift' and '--yshift' options. MakeProfiles: The new '--ccol' option can take the center coordinate columns of the catalog (in multiple calls) and the new '--mode' option is used to identify what standard to interpret them in (image or WCS). Together, these replace the old '--xcol', '--ycol', '--racol' and '--deccol'. MakeProfiles: The new '--crpix', '--crval' and '--cdelt' options now accept multiple values separated by a comma. So they replace the old '--crpix1', '--crpix2', '--crval1', '--crval2' and '--resolution' options. 'gal_data_free_contents': when the input 'gal_data_t' is a tile, its 'array' element will not be freed. This enables safe usage of this function (and thus 'gal_data_free') on tiles without worrying about the memory block associated with the tile. 'gal_box_bound_ellipse' is the new name for the old 'gal_box_ellipse_in_box' (to be more clear and avoid repetition of the term 'box'). The input position angle is now also in degrees, not radians. 'gal_box_overlap' now works on data of any dimensionality and thus also needs the number of dimensions (elements in each input array). 'gal_box_border_from_center' now accepts an array of coordinates as one argument and the number of dimensions as another. This allows it to work on any dimensionality. 'gal_fits_img_info' now also returns the name and units of the dataset (if they aren't NULL). So it takes two extra arguments. 'gal_wcs_pixel_scale' now replaces the old 'gal_wcs_pixel_scale_deg', since it doesn't only apply to degrees. The pixel scale units are defined by the units of the WCS. 'GAL_TILE_PARSE_OPERATE' (only when 'OTHER' is given) can now parse and operate on different datasets independent of the size of allocated block of memory (the tile sizes of 'IN' and 'OTHER' have to be identical, but not their allocated blocks of memory). Until now, it was necessary for the two blocks to have the same size and this is no longer the case. ** Bugs fixed MakeProfiles long options on 32bit big endian systems (bug #51341). Pure rotation around pixel coordinate (0,0) (bug #51353). NoiseChisel segfault when no usable region for sky clumps (bug #51372). Pixel scale measurement when dimension scale isn't equal or doesn't decrease (bug #51385). Improper types for function code in MakeProfiles (bug #51467). Crashes on 32-bit and big-endian systems (bug #51476). Warp's align matrix when second dimension must be reversed (bug #51536). Reading BZERO for unsigned 64-bit integers (bug #51555). Arithmetic with one file and no operators (bug #51559). NoiseChisel segfault when detection contains no clumps (bug #51906). Correct size checking when allocating gal_data_t (bug #52544). * Noteworthy changes in release 0.3 (library 1.0.0) (2017-06-01) [stable] This is a full re-write of Gnuastro. Most importantly, Gnuastro now has a new generic data container ('gal_data_t'). This new container can now deal natively with all standard numeric data types, work in RAM or HDD/SSD, keep data in any dimensions and has enabled many other very useful features. Some of the most prominent of the new features are discussed below. It is strongly recommended to review the respective section of the Gnuastro manual/book for a better feeling of all the new features. As discussed below, some program names have changed, if you have a previous version of Gnuastro installed from source, it is recommended to uninstall it first (with 'make uninstall' using the corresponding tarball), then install this new version. Building Gnuastro can be slow, so please build in parallel with Make's '-j8' option (to build on 8 threads). ** New programs or library features Library functions that deal with datasets now use this generic data container for inputs and outputs, significantly simplifying their API. Nearly all library functions have been re-written with much more clear names, argument lists and individual purpose. Some example library functions are shown below, also see the "Library demos" section of the book for some complete working example: -- 'gal_table_read' and 'gal_table_write' will read and write data to plain text, FITS ASCII and FITS Binary formats. -- 'gal_fits_img_read' and 'gal_fits_img_write' can read a FITS image to memory or write a FITS image from memory. Gnuastro now defines a simple comment line format to keep basic information in a plain text table, see the "Gnuastro text table format" section of the book. In short for every column, a comment line like below can be used to give a name, units, comments, or a type to a column. This allows a FITS binary table for example to be written to plain text and converted back to binary without loosing any information (except for very small floating point errors if not enough decimals are printed). # Column N: NAME [UNIT, TYPE, BLANK] COMMENT The new Fits program replaces the old Header program. But besides reading/checking FITS header keywords, it can now also work on FITS extensions/HDUs. For example with no options, it will list all the HDUs in a FITS file along with basic information. It can copy a whole HDU to another file, or delete a HDU from a FITS file. To get the previous behavior of listing all the keywords in a FITS HDU, you can run it with the '-p' option. All programs now write data into the second HDU of a FITS file to allow a clean first HDU. Note that following CFITSIO, HDU counting still starts from zero, so FITS images and tables written by Gnuastro in a new file can always be accessed with the '--hdu=1' option (which is now the default). If any program is run within a Git version controlled directory, a 'COMMIT' header keyword will be added to the created FITS files, see the "Output headers" section of the book for a discussion on the usefulness of this new feature. BuildProgram: a new program to easily compile, link and run a C program you have written with Gnuastro's libraries without having to worry about which libraries (Gnuastro dependencies) your program needs. Debugging ('-g'), optimizations ('-O'), warnings ('-W'), include search path ('-I'), link search path ('-L'), and linked libraries ('-l') compiler options are also supported. BuildProgram will greatly facilitate the easy usage of Gnuastro's libraries. ** Removed programs or library features The following program names have been renamed: ImageCrop --> Crop ImageWarp --> Warp ImageStatistics --> Statistics Header --> Fits SubtractSky has been removed. The Statistics program now has tools to estimate the Sky value and it can be subtracted with Arithmetic. ** New features All Gnuastro programs that read and write tables can now do so in plain text table format or in FITS ASCII or FITS Binary tables. Depending on the filename or with the new '--tableformat' common option to all programs. The option management system in all Gnuastro programs has been completely re-written with many new features, some of the most important ones are listed below. For developers, you will notice that there is no more usage of macros and adding new options has become much more easier. -- All programs will now also look for a 'gnuastro.conf' configuration file to keep common options for all programs in every directory. -- The '--lastconfig' option can be used on the command-line or in any configuration file to stop parsing any further configuration files. -- The '--config' option can now be used to identify any arbitrary file to be parsed as a configuration file. Any file that is given to this option is parsed immediately. -- The '--printparams' option now also prints the short documentation of each option (same description in '--help') after its value. It is now possible to choose columns in tables based on column names as well as column numbers. It is also possible to search for columns based on searching in their units or comments. When using column numbers, counting now starts from 1 (one), not 0 (as before). See the new "Selecting table columns" section for more on these new features. Where relevant, all programs now accept a '--type' option that you can use to specify the numerical datatype of the output. With the new common option '--minmapsize', you can specify a minimum size of an array (in bytes) to store data in SSD/HDD and not in RAM. This can be instrumental when you are dealing with large datasets, or even smaller ones, but when your RAM is getting full. Making a log file is now optional and users have to explicitly ask for it with the '--log' option. Slower building of Gnuastro: Binary operators (e.g., plus or multiply) are now done in the native type of the input dataset. Doing so for all the different combinations of types, greatly slows down the initial compilation of Gnuastro (after running 'make'). So for every type there is now a '--enable-bin-op-*' configure time option. When the dataset's type isn't compiled (only for the binary operators), it will be converted to a compiled type and then converted back in the end. Arithmetic: all operations are done in the native data type of the dataset. Until now, it would convert the data internally to double precision floating point, do the requested operation and write the data back in the proper type. So this new implementation is much more efficient. Arithmetic: two new classes of operators: type conversion operators to all standard types and integer-only operators (all C bit-wise operators and the modulo operator). ConvertType: can also print the input dataset to the command-line ('stdout'). To use this feature set the output filename to 'stdout'. Convolve now has the '--minsharpspec' option to specify the minimum spectrum value to use in deconvolution (matching PSFs). Crop: when in WCS mode it can still only work on aligned images. However, very small floating point errors in writing the WCS (for example 10e-13 degrees) are now acceptable. Until now these would cause Crop to complain and abort. Crop: the name of the crop can be pre-determined based on the values in a given in a table column. This allows your object's IDs to be directly used as the crop's file name for example. MakeCatalog now also reports the surface brightness in mag/arcsec^2. Until now, MakeCatalog would only print the dataset's surface brightness in units of magnitudes/pixel. But that is not nicely comparable to other datasets. Hence, using a simple calculation (from the pixel projected size, fully derived in the book), it now also reports the surface brightness in magnitudes/arcsec^2 also. MakeProfiles: Profile codes now start from '1' (until now they started from '0'). MakeProfiles: now accepts the radial function of profiles as human-readable strings instead of a code for each profile (which was very cryptic, although codes are also still acceptable). For example in the profile column you can now write 'sersic' instead of the code '1'. NoiseChisel: the new '--cleandilated' option will remove dilated objects that have a low S/N (it is mainly useful on very clean or mock images). For non-clean noise, it will result in a decrease of completeness. With this new option, NoiseChisel will also print detection S/N values when run with the '--checkdetsn' option. Statistics: now reads table columns as well as images and does basic operations on them. It can also only work on a certain range of the data (instead of the whole set). Alternatively, you can define the range on another reference column, but use values of the main column. Statistics: all its single-valued measurements can now be done on a tessellation (tile grid) over the input dataset. Statistics: can now estimate the Sky value on the input dataset using the mode's quantile similar to what NoiseChisel does to find its initial threshold. Statistics: has several new single valued calculations: '--quantile', '--quantfunc' (quantile function), '--mode', '--modequant', '--modesym', and '--modesymvalue'. Warp: align the image with the celestial coordinates using the '--align' option. Warp: standard modular warpings can now be requested without an input matrix, using the following options: '--shear', '--flip', '--project', '--rotate', '--scale', '--translate'. Any number of these transformations (along with the '--align' option) can be called on the command-line and they will be applied in the same order to create one warping matrix. By default the WCS will also be corrected. ** Changes in behavior Mask image options have been removed from all programs. Instead, all programs can work directly on data with blank values. So when some pixels must be masked, the Arithmetic program's 'where' operator can be used to select special pixels and set them to blank. In particular bit-wise operations are now available in Arithmetic to use bit-mask images. Managing all these different choices in every program would only confuse the user (with too many options). Arithmetic: the 'x' letter is now used to represent the multiplication operator. Previously it was '*' which needed quotation and was thus very inconvenient. Convolve: the old '--frequency' and '--spatial' options have been removed and are replaced by '--domain' which accepts values of 'frequency' and 'spatial'. Convolve: the old '--viewfreqsteps' was changed to '--checkfreqsteps' to fit with the general style of such check images in all Gnuastro's programs. Crop: '--section' syntax is now inclusive in both bounds. Crop: only checks if the center of a crop is filled when the crop was defined by its center (for example with '--ra' and '--dec'). The verbose outputs of Crop are also not cryptic 0s or 1s. The are human readable text. Crop: doesn't have separate '--imgmode' and '--wcsmode' options any more. There is now a single '--mode' option which accepts values of 'img' or 'wcs'. MakeProfiles: the old '--inputascanvas' is now called '--clearcanvas'. MakeProfiles: until now, it would abort with an error when the input columns had blank values. But for masking, it might happen that you set a blank magnitude. So this check has now been removed when reading the magnitude column. NoiseChisel: default value of the '--minskyfrac' option (new name for the old '--minbfrac') is now 0.7 as opposed to 0.5. This will allow much better estimation of noise properties (by default). It may be slightly too high for a crowded field, but the users can change it on the command-line (or in a configuration file) for such datasets. NoiseChisel: when it is run with any of the '--check' options, it will abort after all the check images have been created. This is very useful for checking your parameters until each step and not be distracted (or have to wait) for later steps to finish. Statistics: will not make a histogram and cumulative frequency files, or calculate sigma-clipped results by default (with no options). It will just print some basic information. Table: Previously, if a column was requested, the '-i' option would be ignored. But it often happens that the users forget a column name after already typing several of their desired columns. So the opposite behavior is preferred. Because when more than a couple of columns are needed, you will probably forget the column identifiers of the last few and having to retype everything is very frustrating. Something like how '--help' takes precedence over all other options. Table: to select column(s) by regular expression searching, the name now has to be put in '/ /' (similar to AWK). If a value isn't in '/ /', the programs will only select a column with the exact match. Warp: when a 2 by 2 matrix is given, the FITS pixel positions (which define the center of a pixel as an integer) are automatically implemented internally, see "Invoking Warp" in the manual for more. Warp: the old '--nofitscorrect' option has been changed to '--centeroncorner' to be more clear. The new option is now more general than before and also works on warping with a matrix, not just on modular warpings. Warp: the old '--nowcscorrection' option has been given a more clear name of '--keepwcs'. With this option, Warp will not apply the warp the input's WCS structure. Warp: the old '--maxblankfrac' option has been changed to '--coveredfrac'. Until now, Warp would only look for the fraction of input blank/NaN pixel area over the output pixel. But this would be useless on the edges of the image. So the new '--coveredfrac' option takes the acceptable fraction of output pixel area that must be covered by input pixels in order to give that output pixel a value. You can use this to set edge pixels that are not fully covered in the new grid to blank and have a flat warped image. ** Bugs fixed Using '%zu' to print 'size_t' variables for clean build on 32-bit systems. Crash in Table for some operating systems due to memory is now fixed (bug #49347). Table's man-page is now created and installed (bug #49418). Fixes in the documentation (sr #109170, bug #49419). Check for malloc returning valid pointer (bug #49459). Segfault in mesh interpolation corrected (bug #49588). Corrected bad status usage in calls to wcsp2s and wcss2p (bug #49752). Stricter checking in Crop's polygon point list (bug #48978). Correction in alignment and getting pixel scale (bug #50072). Decomposing PCi_j and CDELTi matrices in output WCS (bug #50073). Using image naxes[n] when checkcenter is larger (bug #50099). Memory leak in MakeCatalog corrected (bug #51118). Fix copy-paste error in MakeCatalog flag arrays (bug #51130). * Noteworthy changes in release 0.2 (library 0.0.0) (2016-10-03) [stable] ** Bugs fixed Linker errors on some operating systems have been fixed (bug #48076). Several memory allocation, checks or redundancies have been fixed: bugs #48453, #48516, #48603, #48611, #48571, #48650, #48657, #48692, #48770, #47866, #48899, #49049, #49007. ImageCrop no longer crashes with very long output file names (bugs #46241 and #45380). ** New programs or headers Table: a new utility to read and write FITS binary and ASCII tables. It can also print the column information or select columns using regular expressions (task #13579). Shared libraries and headers are now installed. The libraries can be used in C and C++ programs. This release includes the following headers: 'gnuastro.h', 'array.h', 'box.h', 'fits.h', 'linkedlist.h', 'mesh.h', 'polygon.h', 'qsort.h', 'spatialconvolve.h', 'statistics.h', 'threads.h', 'wcs.h', 'txtarray.h' (task #13765). Gnuastro now comes with a script in its top source directory ('tmpfs-config-make') to configure and build it in the tmpfs (on the RAM), for those systems that have it. See the new "Configure and build in RAM" section in the book for more (task #14100). ** New features MakeProfiles also accepts WCS positions (task #13566). Flat profiles in MakeProfiles can be given a profile specific value. The new '--mforflatpix' option MakeProfile will use the value in the magnitude column as a fixed value for each pixel. This can be very useful in defining a mask, or creating segmentation maps or labeled images (task #14115). MakeProfiles can now use input image as canvas. Instead of specifying the WSC and image size parameters manually. With the new '--inputascanvas' option, MakeProfiles will get this information (along with blank pixels) from an already existing image (task #14116). Type of output in MakeProfiles and Arithmetic can be specified with the '--type' option. Magnitude error column in MakeCatalog with the '--magnitudeerr' option. Arithmetic now has new conditional (task #13870) and logical operators (task #14153) along with an operator for actions only when conditions are true: 'where'. The new 'isblank' operator will also enable you to select blank, or masked, pixels (task #14146). The '--noerodequant' in NoiseChisel enables it to detect small and sharper profiles by disabling erosion on pixels above a certain quantile (task #14139). MakeCatalog can also calculate the upper limit magnitude for each given object in the image by randomly positioning the object's footprint over undetected regions of the image (task #14167). The source tarball is now also distributed with Lzip for a much better compression ratio and more robust archival file format. ** Changes in behavior The two MakeProfiles options '--setconsttonan', '--setconsttomin' have been removed (see '--mforflatpix' above for their alternative). MakeCatalog makes clump catalog only when asked (when the 'WCLUMPS' header exists in the objects HDU). This can be very useful in cases like aperture photometry, when the object labels are not generated by NoiseChisel and so a clump image doesn't exist (task #14122). Default cosmological parameters in CosmicCalculator set to Plank 2015 results: A&A (2016), 594, A13 (arXiv 1502.01589). The '--envseed' option (to read random number generator type and seed from the environment) to MakeProfiles and MakeNoise can also be given in the configuration files. * Noteworthy changes in release 0.1 (2016-05-30) [stable] ** Bugs fixed MakeCatalog's problem in checking the sizes of all input images is now fixed. NoiseChisel's problem with reading the '--kernel' option is now corrected (bug #46750). lib/mesh.c's problem in correctly calculating the mesh sizes was corrected (bug #47611). 'make check' will not look into system utility configuration files. In the previous release, if Gnuastro was already installed, the configuration files already present on the system would also be read. Now only configuration files in the tested package are used (bug #47833). Ghostscript's version is now checked at configure time after its existence. ConvertType uses the '-dPDFFitPage' option to Ghostscript which was introduced in version 9.10, so older versions would pass configure but at 'make check' time, the PDF test would fail. Now this test is skipped (bug #47868). Most tests would fail when 'make check -jN' was run (to do the checks on N threads). A dependency structure has now been defined to fix this problem and greatly speed up the testing process (bug #47957). ** New utilities Arithmetic: For arithmetic operations on the pixels of input images. With this utility, it is now possible to add multiple images with each other, or easily calculate a median image. It as a large set of other arithmetic operations and some functions which can be done on the input image(s). An unlimited number of input images can be given, the images will only be loaded when necessary and will be freed as soon as they are no longer necessary. CosmicCalculator: For doing cosmological calculations at a given redshift. ** New features All the utilities that would produce a log file now have a '--nolog' option to avoid printing a log file. The tiled image compression convention (.fits.fz, created with 'fpack') files can now be used as input in the utilities. ImageCrop can now also crop a polygon from the input image. The polygon vertices can be given in the world or image coordinates. The simple '--polygon' option will keep the insides of the polygon while the '--outpolygon' will keep the outside of the polygon. ImageCrop and ImageWarp can now read the WCS information of a FITS header from a specific region with the '--hstartwcs' and '--hendwcs'. In some older FITS images, when the WCS distortions were not as standardized as now, there were cases which would confuse WCSLIB. NoiseChisel can now save the grown clumps image instead of the original clumps image in the output with the '--grownclumps' option. Convolve can now do deconvolution with the '--makekernel' option. MakeProfiles now has a '--setconsttonan' option which will fill the constant profiles with a NaN (blank) value, not a number, allowing the creations of elliptical masked regions for example. Header can now import a keyword directly from a string with the '--asis' option. MakeCatalog can now output the geometric (average position independent of pixel flux value) positions of the objects too. MakeCatalog can now produce the object's elliptical parameters (for example semi-major axis, semi-minor axis, and position angle). This can also be done both in standard flux weighted and geometric methods too. MakeCatalog now has a '--threshold' function to only use pixels above a given threshold in each object or clump. This is useful to avoid diffuse regions in calculations. MakeCatalog now has a '--noriverbrightness' option. With this option it is possible to calculate the clump flux without subtracting the river pixels on its circumference. The number of CPU threads is no longer a configuration option, it is now determined at run-time for each program. Therefore it is now easily possible to built Gnuastro on one system to use on another (commonly done in the GNU/Linux package managers). Therefore ./configure no longer has a '--with-numthreads' option. Every commit in Gnuastro's history (after implementing this feature) can now be given a unique version number. Since the version number is printed in possible outputs, this feature can help reproducibility, even when the an official/stable release isn't used. The AUTHORS file is now automatically generated from the version controlled history. Also all the authors that have contributed to Gnuastro are included in the second (copyright) page of the PDF book. All the bootstrapped directories are now moved with a new 'bootstrapped' directory in the top source directory. This significantly cleans up this directory, allowing users to more easily find the hand-written Gnuastro source files they like. A 'bug-gnuastro' Info page was created so users can easily go to that page for information on how to submit bug reports. It is accessible on the command line with the command 'info bug-gnuastro'. ** Changes in behavior The separate utilities no longer have a separate version number. With the introduction of unofficial version numbers generated from each commit in Gnuastro's history, the utility version numbers would cause confusion. MakeProfiles will now add a suffix to the individual images and put them in the output directory if specified. NoiseChisel and MakeCatalog now use the median mesh standard deviation to define the over-all depth of the image. Previously they used the maximum value (or the least depth). NoiseChisel no longer outputs a sky subtracted image. This job can now be done with the new Arithmetic utility. NoiseChisel's '--segsnhistnbins' option was renamed to '--clumpsnhistnbins'. ** Improvements NoiseChisel's default quantiles were changed to 0.95 from 0.99. Since the old value was too severe. NoiseChisel's S/N thresholds are now found from the distribution of pseudo-detections and clumps from the full image, not within each large mesh. This was done to increase the accuracy of the S/N threshold. There were commonly not enough points in large mesh sizes and this would add scatter. With ImageCrop's polygon capabilities, it is now easily possible to cut out the region that has uniform noise properties (depth and correlated noise). Therefore the old '--checkdetectionsn' and 'checkclumpsn' options are no longer present. When building from the version controlled source, the whole bootstrapping process is done with one script. In the previous version, all the separate operations should have been done by hand (as instructed in the old manual). All the build steps now report what was done and suggest the next step. This feature can be disabled with the '--disable-guide-message' at configure time. * Copyright notice Copyright (C) 2015-2024 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts.