GNU Astronomy Utilities NEWS -*- outline -*- * Noteworthy changes in release 0.6 (library 4.0.0) (2018-06-04) [stable] ** New features Building: - New optional dependency: The TIFF library (libtiff). All programs: - Input image dataset(s) can be in any of the formats recognized by Gnuastro (e.g., FITS, TIFF, JPEG), provided that their libraries (optional dependencies) were present at installation time. New program: - Segment: new program in charge of segmentation over detections. This operation was previously done by NoiseChisel. NoiseChisel is now ONLY in charge of detection. Arithmetic: - erode: Erode the foreground of a binary dataset. - dilate: Dilate the foreground of a binary dataset. - filter-sigclip-mean: sigma-clipped, mean filter operator. - filter-sigclip-median: sigma-clipped, median filter operator. - connected-components: label the connected elements of the input. - invert: subtract the maximum of unsigned types (absorption to emission). - interpolate-medianngb: Interpolate (only blanks) with nearest neighbors. ConvertType: - TIFF images can also be used as input. MakeCatalog: MakeCatalog will only read the datasets necessary for the requested columns. Until now, it would read all the possible datasets and all the intermediate measurements. This is thus major improvement in memory and CPU usage. As a result, the input argument is no longer assumed to be the values file, but the object labels file. Please see the "MakeCatalog inputs and basic settings" section of the book for more. Here is the summary of the new options: --insky: Sky value as a single value or file (dataset). --instd: Sky standard deviation as a single value or file (dataset). --valuesfile: filename containing the values dataset. --valueshdu: HDU/extension containing the values dataset. --clumpscat: Make a clumps catalog also. --noclumpsort: Don't sort the clumps catalog by host object ID. --subtractsky: Subtract the given Sky from the values dataset. --variance: input standard deviation image is actually variance. --checkupperlimit: make a table for random positions and measurements. --geoarea: the number of labeled pixels (irrespective of value). --brightnesserr: error in estimating the brightness. --mean: calculate the mean pixel value within an object or clump. --median: calculate the median pixel value within an object or clump. --upperlimitsigma: position in random distribution (in units of sigma). --upperlimitquantile: position in random distribution (quantile). --upperlimitonesigma: 1sigma value of the random distribution. --upperlimitskew: (mean-median)/sigma or skewness of random distribution. NoiseChisel: - New tutorial on detecting large and extended targets. --rawoutput: only output the detection labels and Sky and its STD. --ignoreblankinsky: don't set the pixels that are blank in the input to blank in the Sky and Sky standard deviation outputs (when `--oneelempertile' is not called). --label: label the connected detections. Until now this was the default behavior. However, from this release, NoiseChisel is only in charge of detection. Segmentation is done by a new program (Segment). Since detection is ultimately just a binary operator, the default output now has an 8-bit unsigned integer type with values of 0 or 1. With this option, you can ask for it to label/count the connected detections instead of the default binary output. Statistics: --manualbinrange: histogram or CFP range can be outside of distribution. --ignoreblankinsky: similar to same option in NoiseChisel. Libraries: gal_array_read: read array from any of known formats (FITS, TIFF, JPEG,...). gal_array_read_to_type: similar to `gal_array_read', but to given type. gal_array_read_one_ch: Read a dataset, abort if it has multiple channels. gal_array_read_one_ch_to_type: Make sure input is in one channel and type. gal_binary_label_holes: label the holes within the foreground. gal_blank_is: check to see if argument is blank in its type or not. gal_eps_name_is_eps: Returns 1 if given filename is EPS. gal_eps_suffix_is_eps: Returns 1 if given suffix is EPS. gal_eps_to_pt: Converts dataset size to PostScript points. gal_eps_write: Writes a dataset into an EPS file. gal_interpolate_1d_blank: Fill blank elements through interpolation. gal_interpolate_1d_make_gsl_spline: Allocate and initalize `gsl_spline'. gal_jpeg_name_is_jpeg: Returns 1 if given filename is JPEG. gal_jpeg_suffix_is_jpeg: Returns 1 if given suffix is JPEG. gal_jpeg_read: Reads input JPEG image into `gal_data_t'. gal_jpeg_write: Writes a `gal_data_t' into a JPEG file. gal_label_grow_indexs: grow known indexs into desired areas. gal_label_watershed: apply watershed algorithm on desired region. gal_label_clump_significance: measure significance of all clumps in region. gal_pdf_name_is_pdf: Returns 1 if given filename is PDF. gal_pdf_suffix_is_pdf: Returns 1 if given suffix is PDF. gal_pdf_write: Writes a dataset into an PDF file. gal_pointer_allocate_mmap: Allocate space in a file, not in RAM. gal_qsort_index_single_d: Sort indexs of single array in decreasing order. gal_qsort_index_single_i: Sort indexs of single array in decreasing order. gal_qsort_index_multi_d: Sort indexs in multiple arrays (different threads). gal_qsort_index_multi_i: Sort indexs in multiple arrays (different threads). gal_tiff_name_is_tiff: check if name contains a TIFF suffix. gal_tiff_suffix_is_tiff: check if suffix is a TIFF suffix. gal_tiff_dir_string_read: convert a string to a TIFF directory number. gal_tiff_read: Read the contents of a TIFF "directory" to `gal_data_t'. ** Removed features NoiseChisel: - Segmentation (and thus the options below) moved to the new Segment program: --onlydetection, --segsnminarea, --checkclumpsn, --segquant, --keepmaxnearriver, --gthresh, --minriverlength, --objbordersn, --grownclumps, --checksegmentation. --skysubtracted: no longer necessary, included in noise measuremnts. MakeCatalog: --skysubtracted: no longer necessary, included in noise measuremnts. Library: - The macros `GAL_STATISTICS_SORTED_NOT', `GAL_STATISTICS_SORTED_INVALID', `GAL_STATISTICS_SORTED_INCREASING', `GAL_STATISTICS_SORTED_DECREASING': these macros are removed because we already have the `GAL_DATA_FLAG_SORT*'' bit-flags in `gal_data_t'. ** Changed features Arithmetic: - filter-mean: a blank value in the input can be non-blank in the output when non-blank elements present in filter. - filter-median: similar to filter-mean. Fits: --history: can be called/written multiple times in one run. --comment: can be called/written multiple times in one run. MakeCatalog: - The `WCLUMPS' keyword in the objects labeled image is no longer used to see if a clumps catalog should also be made. To build a clumps catalog, you can now use the `--clumpscat' option. - Estimation of noise-level is now done per-pixel over the whole label. Until now the average noise level was used. --objectsfile has been removed. The main input argument is now assumed to be the objects file. NoiseChisel: From this release, NoiseChisel is only in charge of detection and won't do segmentation any more. The new Segment program is in charge of segmentation. Many of the changes below are due to this new limited scope of NoiseChisel. --kernel: value `none' will disable convolution. - Renamed options: --convolvedhdu ==> --chdu --wkhdu ==> --whdu --detsnminarea ==> --snminarea --checkdetsn ==> --checksn --detquant ==> --snquant - By default the output detection map is a binary image (values of 0 or 1). - With no output name, the output has a `_detected.fits' suffix. Segment: - [Previously in NoiseChisel]: For finding true clumps, the difference in the peak of the clump and the highest valued river pixel, divided by the noise standard deviation are used. Until now, the total signal-to-noise ratio was used as a criteria. In initial tests, this algorithm was much more promising in detecting clumps over strong gradients and also on flatter gradients. Table: --column: multiple columns (comma separated) can be used in one instance of this option (multiple instances of this option are still acceptable also). Libraries: gal_binary_holes_fill: new name for `gal_binary_fill_holes'. gal_dimension_is_different: new name for `gal_data_dsize_is_different'. gal_fits_img_read: now only reads the data not the WCS, therefore it no longer needs the last two arguments. A subsequent call to `gal_wcs_read' can be used to read the WCS information in the file. gal_pointer_increment: new name for `gal_data_ptr_increment'. gal_pointer_num_between: new name for `gal_data_ptr_dist'. gal_pointer_allocate: replaces `gal_data_malloc_array' and `gal_data_calloc_array', through an argument you can ask for the allocated memory to be cleared or not. gal_qsort_TYPE_i: new name for gal_qsort_TYPE_increasing. gal_qsort_TYPE_d: new name for gal_qsort_TYPE_decreasing. gal_statistics_is_sorted: can now also update the bit-flags regarding the sorted nature of the input (to optimize future calls to the function). gal_statistics_quantile_function: returns `inf' or `-inf' if the given value is smaller than the minimum or larger than the maximum of the input dataset's range. Until now, it would return blank in such cases. gal_statistics_number: the output dataset now has a `size_t' type. Until now it was `uint64_t'. ** Bug fixes bug #50957: --version output not possible on Mac OS X bug #52979: Many unused result warnings for asprintf in some compilers. bug #53122: Configure time CPPFLAGS and LDFLAGS don't pass to BuildProgram. bug #53142: Crash when printing values with the `--onlyversion' option. bug #53147: NULL value of onlyversion option causing a crash. bug #53226: Match output directory ignored when making multiple files. bug #53230: Statistics program bad results on integer columns with limits. bug #53268: NoiseChisel crash when no growth is possible. bug #53295: MakeCatalog parses area larger than clump. bug #53304: NoiseChisel crash when there is no detection. bug #53312: Fits crash on keyword editing (except --delete). bug #53407: Instrumental noise in MakeNoise should be squared. bug #53424: Sigma-clipping seg-faults when there are no more elements. bug #53580: Warp crash when no WCS present. bug #53825: NoiseChisel not accounting for fully zero-valued tiles. * Noteworthy changes in release 0.5 (library 3.0.0) (2017-12-22) [stable] ** New features Manual/Book: An extended tutorial is added showing some general applications of almost all the programs. This may be a good place to get a feeling of how Gnuastro is intended to be used and some of the programs. New Program and library: Match is a new program that will match two given inputs (currently catalogs). Its output is the re-arranged inputs with the same number of rows/records such that all the rows match. The main work is also done with the new low-level `gal_match_catalog' library function which can also be used in more generic contexts. All programs: a value of `0' to the `--numthreads' option will use the number of threads available to the system at run time. Arithmetic: The new operators `filter-median' and `filter-mean' can be used to filter (smooth) the input. The size of the filter can be set as the other operands to these operators. BuildProgram: The new `--la' option allows the identification of a different Libtool `.la' file for Libtool linking information. BuildProgram: The new `--deletecompiled' option will delete the compiled program after running it. CosmicCalculator: all the various cosmological calculations can now be requested individually in one line with a specific option added for each calculation (for example `--age' or `--luminositydist' for the age of the universe at a given redshift or the luminosity distance). Therefore the old `--onlyvolume' and `--onlyabsmagconv' options are now removed. To effectively use these new features, please review the "Invoking CosmicCalculator" section of the book. Fits: when an extension/HDU is identified on the command-line with the `--hdu' option and no operation is requested, the full list of header keywords in that HDU will be printed (as if only `--printallkeys' was called). MakeCatalog: physical nature agnostic WCS column names. Previously the first WCS axis was always assumed to be RA and the second DEC. So for example even if you had a spectrum (with X and wavelength as the two WCS dimensions), you would have to ask for `--ra' and `--dec'. The new `--w1' and `--w2' options are now generic and don't assume any particular type only their order in the FITS header. MakeCatalog now also uses the CTYPE and CUNIT keywords to set the names and units of its output columns. The `--ra' and `--dec' options are now just internal aliases for `--w1' or `--w2' which will be determined based on the input's CTYPE keyword. Also the new `--geow1', `--geow2', `--clumpsw1', `--clumpsw2', `--clumpsgeow1', `--clumpsgeow2' options replace the old options `--geora', `--geodec', `--clumpsra', `--clumpsdec', `--clumpsgeora', `--clumpsgeodec'. No alias is currently defined for the latter group. MakeCatalog: the new `--uprange' option allows you to specify a range for the random values around each object. This is useful when the noise properties of the dataset vary gradually and sampling from the whole dataset might produce biased results. NoiseChisel: with the new `--convolved' and `--convolvedhdu' options, NoiseChisel will not convolve the input any more and use the given dataset instead. In many cases, as the inputs get larger, convolution is the most time consuming step of NoiseChisel. With this option, you can greatly speed up your tests (to find the best parameters by varying them, for a given analysis). See the book for more information and examples. NoiseChisel: with the new `--widekernel' option it is now possible to use a wider kernel to identify which tiles contain signal. The rest of the steps (identifying the quantile threshold on the selected tiles and etc) are done on the dataset convolved with `--kernel' as they were before. Since it is time consuming, this is an optional feature. NoiseChisel: with the new `--qthreshtilequant' option, it is now possible to discard high-valued (outlier) tiles before estimating qthresh over the whole image. This can be useful in detecting very large diffuse/flat regions that would otherwise be detected as background (and effectively removed). NoiseChisel: the finally selected true detections are now grown based on signal contiguity, not by blind dilation. The growth process is the same as the growing of clumps to define objects. Only for true detections, the growth occurs in the noise. You can configure this growth with the `--detgrowquant' and `--detgrowmaxholesize'. With this new feature it is now possible to detect signal out to much lower surface brightness limits and the detections don't look boxy any more. Cosmology library: A new set of cosmology functions are now included in the library (declared in `gnuastro/cosmology.h'). These functions are also used in the CosmicCalculator program. `gal_table_read' can now return the number of columns matched with each input column (for example with regular expressions), a new argument has been added to allow this feature. `gal_fits_key_img_blank': returns the value that must be used in the BLANK keyword for the given type as defined by the FITS standard. `gal_txt_write' and `gal_fits_tab_write' now accept an extension name as argument to allow a name for the FITS extension they write. `gal_box_bound_ellipse_extent' will return the maximum extent of an ellipse along each axis from the ellipse center in floating point. ** Removed features Installation: The `--enable-bin-op-*' configuration options that were introduced in Gnuastro 0.3 have been removed. By managing the arithmetic functions in a better manner (a separate source file for each operator), compilation for all types (when done in parallel) takes about the same time as it took with the default (only four) types until now. MakeCatalog: `--zeropoint' option doesn't have a short option name any more. Previously it was `-z' which was confusing because `-x' and `-y' were used to refer to image coordinate positions. NoiseChisel: The `--dilate' and `--dilatengb' options have been removed. Growing of true detections is no longer done through dilation but through the `--detgrowquant' and `--detgrowmaxholesize' options (see above). ** Changed features CosmicCalculator: The redshift is no longer mandatory. When no redshift is given, it will only print the input parameters (cosmology) and abort. MakeCatalog: when the output is a FITS file, the two object and clumps catalogs will be stored as multiple extensions of a single file. Until now, two separate FITS files would be created. Plain text outputs are the same as before (two files will be created). `gal_binary_fill_holes' now accepts a `connectivity' and `maxsize' argument to specify the connectivity of the holes and the maximum size of acceptable holes to fill. `gal_fits_img_read' and `gal_fits_img_read_to_type' now also read the WCS structure of the extension/HDU in a FITS file and have two extra arguments: `hstartwcs' and `hendwcs'. With these options it is possible to limit the range of header keywords to read the WCS, similar to how they are used in `gal_wcs_read'. `gal_txt_write', `gal_table_write_log', `gal_fits_tab_write' and `gal_txt_write' don't have the `dontdelete' argument any more. The action they take if the file already exists depends on the file: for FITS, a new extension will be added and for text, they will abort with an error. `gal_tile_block_write_const_value' and `gal_tile_full_values_write' now accept a new `withblank' option to set all pixels that are blank in the tile's block to be blank in the check image. `gal_wcs_pixel_area_arcsec2' will return NaN (instead of aborting) when input is unreasonable (not two dimensions or not in units of degrees). `gal_wcs_world_to_img' and `gal_wcs_img_to_world': Until now, these two WCS conversion functions would explicitly assume RA and Dec and work based on input arrays (so for example it was also necessary to give the number of elements and etc). They now accept `gal_data_t' as input for the input coordinates, thus their API has been greatly simplified and their functionality increased. ** Bug fixes ConvertType crash when changing values (bug #52010). Arithmetic not accounting for integer blank pixels in binary operators (bug #52014). NoiseChisel segfault when memory mapping to a file (bug #52043). CFITSIO 3.42 and libcurl crash at Gnuastro configure time (bug #52152). MakeCatalog crash in upper-limit with full size label (bug #52281). NoiseChisel leaving unlabeled regions after clump growth (bug #52327). Arithmetic crash with no input tokens (bug #52422). Libtool checks only in non-current directory (bug #52427). * Noteworthy changes in release 0.4 (library 2.0.0) (2017-09-13) [stable] ** New features All programs: `.fit' is now a recognized FITS file suffix. All programs: ASCII text files (tables) created with CRLF line terminators (for example text files created in MS Windows) are now also readable as input when necessary. Arithmetic: now has a new `--globalhdu' (`-g') option which can be used once for all the input images. MakeNoise: with the new `--sigma' (`-s') option, it is now possible to directly request the noise sigma or standard deviation. When this option is called, the `--background', `--zeropoint' and other option values will be ignored. MakeProfiles: the new `--kernel' option can make a kernel image without the need to define a catalog. With this option, a catalog (or accompanying background image) must not be given. MakeProfiles: the new `--pc', `--cunit' and `--ctype' options can be used to specify the PC matrix, CUNIT and CTYPE world coordinate system keywords of the output FITS file. MakeProfiles: the new `distance' profile will save the radial distance of each pixel. This may be used to define your own profiles that are not currently supported in MakeProfiles. MakeProfiles: with the new `--mcolisbrightness' ("mcol-is-brightness") option, the `--mcol' values of the catalog will be interpretted as total brightness (sum of pixel values), not magnitude. NoiseChisel: with the new `--dilatengb' option, it is now possible to identify the connectivity of the final dilation. Library: Functions that read data from an ASCII text file (`gal_txt_table_info', `gal_txt_table_read', `gal_txt_image_read') now also operate on files with CRLF line terminators. ** Removed features ** Changed features Crop: The new `--center' option is now used to define the center of a single crop. Hence the old `--ra', `--dec', `--xc', `--yc' have been removed. This new option can take multiple values (one value for each dimension). Fractions are also acceptable. Crop: The new `--width' option is now used to define the width of a single crop. Hence the old `--iwidth', `--wwidth' were removed. The units to interpret the value to the option are specified by the `--mode' option. With the new `--width' option it is also possible to define a non-square crop (different widths along each dimension). In WCS mode, its units are no longer arcseconds but are the same units of the WCS (degrees for angles). `--width' can also accept fractions. So to set a width of 5 arcseconds, you can give it a value of `5/3600' for the angular dimensions. Crop: The new `--coordcol' option is now used to determine the catalog columns that define coordinates. Hence the old `--racol', `--deccol', `--xcol', and `--ycol' have been removed. This new option can be called multiple times and the order of its calling will be used for the column containing the center in the respective dimension (in FITS format). MakeNoise: the old `--stdadd' (`-s') option has been renamed to `--instrumental' (`-i') to be more clear. MakeProfiles: The new `--naxis' and `--shift' options can take multiple values for each dimension (separated by a comma). This replaces the old `--naxis1', `--naxis2' and `--xshift' and `--yshift' options. MakeProfiles: The new `--ccol' option can take the center coordinate columns of the catalog (in multiple calls) and the new `--mode' option is used to identify what standard to interpret them in (image or WCS). Together, these replace the old `--xcol', `--ycol', `--racol' and `--deccol'. MakeProfiles: The new `--crpix', `--crval' and `--cdelt' options now accept multiple values separated by a comma. So they replace the old `--crpix1', `--crpix2', `--crval1', `--crval2' and `--resolution' options. `gal_data_free_contents': when the input `gal_data_t' is a tile, its `array' element will not be freed. This enables safe usage of this function (and thus `gal_data_free') on tiles without worrying about the memory block associated with the tile. `gal_box_bound_ellipse' is the new name for the old `gal_box_ellipse_in_box' (to be more clear and avoid repetition of the term `box'). The input position angle is now also in degrees, not radians. `gal_box_overlap' now works on data of any dimensionality and thus also needs the number of dimensions (elements in each input array). `gal_box_border_from_center' now accepts an array of coordinates as one argument and the number of dimensions as another. This allows it to work on any dimensionality. `gal_fits_img_info' now also returns the name and units of the dataset (if they aren't NULL). So it takes two extra arguments. `gal_wcs_pixel_scale' now replaces the old `gal_wcs_pixel_scale_deg', since it doesn't only apply to degrees. The pixel scale units are defined by the units of the WCS. `GAL_TILE_PARSE_OPERATE' (only when `OTHER' is given) can now parse and operate on different datasets independent of the size of allocated block of memory (the tile sizes of `IN' and `OTHER' have to be identical, but not their allocated blocks of memory). Until now, it was necessary for the two blocks to have the same size and this is no longer the case. ** Bug fixes MakeProfiles long options on 32bit big endian systems (bug #51341). Pure rotation around pixel coordinate (0,0) (bug #51353). NoiseChisel segfault when no usable region for sky clumps (bug #51372). Pixel scale measurement when dimension scale isn't equal or doesn't decrease (bug #51385). Improper types for function code in MakeProfiles (bug #51467). Crashes on 32-bit and big-endian systems (bug #51476). Warp's align matrix when second dimension must be reversed (bug #51536). Reading BZERO for unsigned 64-bit integers (bug #51555). Arithmetic with one file and no operators (bug #51559). NoiseChisel segfault when detection contains no clumps (bug #51906). Correct size checking when allocating gal_data_t (bug #52544). * Noteworthy changes in release 0.3 (library 1.0.0) (2017-06-01) [stable] This is a full re-write of Gnuastro. Most importantly, Gnuastro now has a new generic data container (`gal_data_t'). This new container can now deal natively with all standard numeric data types, work in RAM or HDD/SSD, keep data in any dimensions and has enabled many other very useful features. Some of the most prominent of the new features are discussed below. It is strongly recommended to review the respective section of the Gnuastro manual/book for a better feeling of all the new features. As discussed below, some program names have changed, if you have a previous version of Gnuastro installed from source, it is recommended to uninstall it first (with `make uninstall' using the corresponding tarball), then install this new version. Building Gnuastro can be slow, so please build in parallel with Make's `-j8' option (to build on 8 threads). ** New programs or library features Library functions that deal with datasets now use this generic data container for inputs and outputs, significantly simplifying their API. Nearly all library functions have been re-written with much more clear names, argument lists and individual purpose. Some example library functions are shown below, also see the "Library demos" section of the book for some complete working example: -- `gal_table_read' and `gal_table_write' will read and write data to plain text, FITS ASCII and FITS Binary formats. -- `gal_fits_img_read' and `gal_fits_img_write' can read a FITS image to memory or write a FITS image from memory. Gnuastro now defines a simple comment line format to keep basic information in a plain text table, see the "Gnuastro text table format" section of the book. In short for every column, a comment line like below can be used to give a name, units, comments, or a type to a column. This allows a FITS binary table for example to be written to plain text and converted back to binary without loosing any information (except for very small floating point errors if not enough decimals are printed). # Column N: NAME [UNIT, TYPE, BLANK] COMMENT The new Fits program replaces the old Header program. But besides reading/checking FITS header keywords, it can now also work on FITS extensions/HDUs. For example with no options, it will list all the HDUs in a FITS file along with basic information. It can copy a whole HDU to another file, or delete a HDU from a FITS file. To get the previous behavior of listing all the keywords in a FITS HDU, you can run it with the `-p' option. All programs now write data into the second HDU of a FITS file to allow a clean first HDU. Note that following CFITSIO, HDU counting still starts from zero, so FITS images and tables written by Gnuastro in a new file can always be accessed with the `--hdu=1' option (which is now the default). If any program is run within a Git version controlled directory, a `COMMIT' header keyword will be added to the created FITS files, see the "Output headers" section of the book for a discussion on the usefulness of this new feature. BuildProgram: a new program to easily compile, link and run a C program you have written with Gnuastro's libraries without having to worry about which libraries (Gnuastro dependencies) your program needs. Debugging (`-g'), optimizations (`-O'), warnings (`-W'), include search path (`-I'), link search path (`-L'), and linked libraries (`-l') compiler options are also supported. BuildProgram will greatly facilitate the easy usage of Gnuastro's libraries. ** Removed programs or library features The following program names have been renamed: ImageCrop --> Crop ImageWarp --> Warp ImageStatistics --> Statistics Header --> Fits SubtractSky has been removed. The Statistics program now has tools to estimate the Sky value and it can be subtracted with Arithmetic. ** New features All Gnuastro programs that read and write tables can now do so in plain text table format or in FITS ASCII or FITS Binary tables. Depending on the filename or with the new `--tableformat' common option to all programs. The option management system in all Gnuastro programs has been completely re-written with many new features, some of the most important ones are listed below. For developers, you will notice that there is no more usage of macros and adding new options has become much more easier. -- All programs will now also look for a `gnuastro.conf' configuration file to keep common options for all programs in every directory. -- The `--lastconfig' option can be used on the command-line or in any configuration file to stop parsing any further configuration files. -- The `--config' option can now be used to identify any arbitrary file to be parsed as a configuration file. Any file that is given to this option is parsed immediately. -- The `--printparams' option now also prints the short documentation of each option (same description in `--help') after its value. It is now possible to choose columns in tables based on column names as well as column numbers. It is also possible to search for columns based on searching in their units or comments. When using column numbers, counting now starts from 1 (one), not 0 (as before). See the new "Selecting table columns" section for more on these new features. Where relevant, all programs now accept a `--type' option that you can use to specify the numerical datatype of the output. With the new common option `--minmapsize', you can specify a minimum size of an array (in bytes) to store data in SSD/HDD and not in RAM. This can be instrumental when you are dealing with large datasets, or even smaller ones, but when your RAM is getting full. Making a log file is now optional and users have to explicitly ask for it with the `--log' option. Slower building of Gnuastro: Binary operators (e.g., plus or multiply) are now done in the native type of the input dataset. Doing so for all the different combinations of types, greatly slows down the initial compilation of Gnuastro (after running `make'). So for every type there is now a `--enable-bin-op-*' configure time option. When the dataset's type isn't compiled (only for the binary operators), it will be converted to a compiled type and then converted back in the end. Arithmetic: all operations are done in the native data type of the dataset. Until now, it would convert the data internally to double precision floating point, do the requested operation and write the data back in the proper type. So this new implementation is much more efficient. Arithmetic: two new classes of operators: type conversion operators to all standard types and integer-only operators (all C bit-wise operators and the modulo operator). ConvertType: can also print the input dataset to the command-line (`stdout'). To use this feature set the output filename to `stdout'. Convolve now has the `--minsharpspec' option to specify the minimum spectrum value to use in deconvolution (matching PSFs). Crop: when in WCS mode it can still only work on aligned images. However, very small floating point errors in writing the WCS (for example 10e-13 degrees) are now acceptable. Until now these would cause Crop to complain and abort. Crop: the name of the crop can be pre-determined based on the values in a given in a table column. This allows your object's IDs to be directly used as the crop's file name for example. MakeCatalog now also reports the surface brightness in mag/arcsec^2. Until now, MakeCatalog would only print the dataset's surface brightness in units of magnitudes/pixel. But that is not nicely comparable to other datasets. Hence, using a simple calculation (from the pixel projected size, fully derived in the book), it now also reports the surface brightness in magnitudes/arcsec^2 also. MakeProfiles: Profile codes now start from `1' (until now they started from `0'). MakeProfiles: now accepts the radial function of profiles as human-readable strings instead of a code for each profile (which was very cryptic, although codes are also still acceptable). For example in the profile column you can now write `sersic' instead of the code `1'. NoiseChisel: the new `--cleandilated' option will remove dilated objects that have a low S/N (it is mainly useful on very clean or mock images). For non-clean noise, it will result in a decrease of completeness. With this new option, NoiseChisel will also print detection S/N values when run with the `--checkdetsn' option. Statistics: now reads table columns as well as images and does basic operations on them. It can also only work on a certain range of the data (instead of the whole set). Alternatively, you can define the range on another reference column, but use values of the main column. Statistics: all its single-valued measurements can now be done on a tessellation (tile grid) over the input dataset. Statistics: can now estimate the Sky value on the input dataset using the mode's quantile similar to what NoiseChisel does to find its initial threshold. Statistics: has several new single valued calculations: `--quantile', `--quantfunc' (quantile function), `--mode', `--modequant', `--modesym', and `--modesymvalue'. Warp: align the image with the celestial coordinates using the `--align' option. Warp: standard modular warpings can now be requested without an input matrix, using the following options: `--shear', `--flip', `--project', `--rotate', `--scale', `--translate'. Any number of these transformations (along with the `--align' option) can be called on the command-line and they will be applied in the same order to create one warping matrix. By default the WCS will also be corrected. ** Changes in behavior Mask image options have been removed from all programs. Instead, all programs can work directly on data with blank values. So when some pixels must be masked, the Arithmetic program's `where' operator can be used to select special pixels and set them to blank. In particular bit-wise operations are now available in Arithmetic to use bit-mask images. Managing all these different choices in every program would only confuse the user (with too many options). Arithmetic: the `x' letter is now used to represent the multiplication operator. Previously it was `*' which needed quotation and was thus very inconvenient. Convolve: the old `--frequency' and `--spatial' options have been removed and are replaced by `--domain' which accepts values of `frequency' and `spatial'. Convolve: the old `--viewfreqsteps' was changed to `--checkfreqsteps' to fit with the general style of such check images in all Gnuastro's programs. Crop: `--section' syntax is now inclusive in both bounds. Crop: only checks if the center of a crop is filled when the crop was defined by its center (for example with `--ra' and `--dec'). The verbose outputs of Crop are also not cryptic 0s or 1s. The are human readable text. Crop: doesn't have separate `--imgmode' and `--wcsmode' options any more. There is now a single `--mode' option which accepts values of `img' or `wcs'. MakeProfiles: the old `--inputascanvas' is now called `--clearcanvas'. MakeProfiles: until now, it would abort with an error when the input columns had blank values. But for masking, it might happen that you set a blank magnitude. So this check has now been removed when reading the magnitude column. NoiseChisel: default value of the `--minskyfrac' option (new name for the old `--minbfrac') is now 0.7 as opposed to 0.5. This will allow much better estimation of noise properties (by default). It may be slightly too high for a crowded field, but the users can change it on the command-line (or in a configuration file) for such datasets. NoiseChisel: when it is run with any of the `--check' options, it will abort after all the check images have been created. This is very useful for checking your parameters until each step and not be distracted (or have to wait) for later steps to finish. Statistics: will not make a histogram and cumulative frequency files, or calculate sigma-clipped results by default (with no options). It will just print some basic information. Table: Previously, if a column was requested, the `-i' option would be ignored. But it often happens that the users forget a column name after already typing several of their desired columns. So the opposite behavior is preferred. Because when more than a couple of columns are needed, you will probably forget the column identifiers of the last few and having to retype everything is very frustrating. Something like how `--help' takes precedence over all other options. Table: to select column(s) by regular expression searching, the name now has to be put in `/ /' (similar to AWK). If a value isn't in `/ /', the programs will only select a column with the exact match. Warp: when a 2 by 2 matrix is given, the FITS pixel positions (which define the center of a pixel as an integer) are automatically implemented internally, see "Invoking Warp" in the manual for more. Warp: the old `--nofitscorrect' option has been changed to `--centeroncorner' to be more clear. The new option is now more general than before and also works on warping with a matrix, not just on modular warpings. Warp: the old `--nowcscorrection' option has been given a more clear name of `--keepwcs'. With this option, Warp will not apply the warp the input's WCS structure. Warp: the old `--maxblankfrac' option has been changed to `--coveredfrac'. Until now, Warp would only look for the fraction of input blank/NaN pixel area over the output pixel. But this would be useless on the edges of the image. So the new `--coveredfrac' option takes the acceptable fraction of output pixel area that must be covered by input pixels in order to give that output pixel a value. You can use this to set edge pixels that are not fully covered in the new grid to blank and have a flat warped image. ** Bug fixes Using `%zu' to print `size_t' variables for clean build on 32-bit systems. Crash in Table for some operating systems due to memory is now fixed (bug #49347). Table's man-page is now created and installed (bug #49418). Fixes in the documentation (sr #109170, bug #49419). Check for malloc returning valid pointer (bug #49459). Segfault in mesh interpolation corrected (bug #49588). Corrected bad status usage in calls to wcsp2s and wcss2p (bug #49752). Stricter checking in Crop's polygon point list (bug #48978). Correction in alignment and getting pixel scale (bug #50072). Decomposing PCi_j and CDELTi matrices in output WCS (bug #50073). Using image naxes[n] when checkcenter is larger (bug #50099). Memory leak in MakeCatalog corrected (bug #51118). Fix copy-paste error in MakeCatalog flag arrays (bug #51130). * Noteworthy changes in release 0.2 (library 0.0.0) (2016-10-03) [stable] ** Bug fixes Linker errors on some operating systems have been fixed (bug #48076). Several memory allocation, checks or redundancies have been fixed: bugs #48453, #48516, #48603, #48611, #48571, #48650, #48657, #48692, #48770, #47866, #48899, #49049, #49007. ImageCrop no longer crashes with very long output file names (bugs #46241 and #45380). ** New programs or headers Table: a new utility to read and write FITS binary and ASCII tables. It can also print the column information or select columns using regular expressions (task #13579). Shared libraries and headers are now installed. The libraries can be used in C and C++ programs. This release includes the following headers: `gnuastro.h', `array.h', `box.h', `fits.h', `linkedlist.h', `mesh.h', `polygon.h', `qsort.h', `spatialconvolve.h', `statistics.h', `threads.h', `wcs.h', `txtarray.h' (task #13765). Gnuastro now comes with a script in its top source directory (`tmpfs-config-make') to configure and build it in the tmpfs (on the RAM), for those systems that have it. See the new "Configure and build in RAM" section in the book for more (task #14100). ** New features MakeProfiles also accepts WCS positions (task #13566). Flat profiles in MakeProfiles can be given a profile specific value. The new `--mforflatpix' option MakeProfile will use the value in the magnitude column as a fixed value for each pixel. This can be very useful in defining a mask, or creating segmentation maps or labeled images (task #14115). MakeProfiles can now use input image as canvas. Instead of specifying the WSC and image size parameters manually. With the new `--inputascanvas' option, MakeProfiles will get this information (along with blank pixels) from an already existing image (task #14116). Type of output in MakeProfiles and Arithmetic can be specified with the `--type' option. Magnitude error column in MakeCatalog with the `--magnitudeerr' option. Arithmetic now has new conditional (task #13870) and logical operators (task #14153) along with an operator for actions only when conditions are true: `where'. The new `isblank' operator will also enable you to select blank, or masked, pixels (task #14146). The `--noerodequant' in NoiseChisel enables it to detect small and sharper profiles by disabling erosion on pixels above a certain quantile (task #14139). MakeCatalog can also calculate the upper limit magnitude for each given object in the image by randomly positioning the object's footprint over undetected regions of the image (task #14167). The source tarball is now also distributed with Lzip for a much better compression ratio and more robust archival file format. ** Changes in behavior The two MakeProfiles options `--setconsttonan', `--setconsttomin' have been removed (see `--mforflatpix' above for their alternative). MakeCatalog makes clump catalog only when asked (when the `WCLUMPS' header exists in the objects HDU). This can be very useful in cases like aperture photometry, when the object labels are not generated by NoiseChisel and so a clump image doesn't exist (task #14122). Default cosmological parameters in CosmicCalculator set to Plank 2015 results: A&A (2016), 594, A13 (arXiv 1502.01589). The `--envseed' option (to read random number generator type and seed from the environment) to MakeProfiles and MakeNoise can also be given in the configuration files. * Noteworthy changes in release 0.1 (2016-05-30) [stable] ** Bug fixes MakeCatalog's problem in checking the sizes of all input images is now fixed. NoiseChisel's problem with reading the `--kernel' option is now corrected (bug #46750). lib/mesh.c's problem in correctly calculating the mesh sizes was corrected (bug #47611). `make check' will not look into system utility configuration files. In the previous release, if Gnuastro was already installed, the configuration files already present on the system would also be read. Now only configuration files in the tested package are used (bug #47833). Ghostscript's version is now checked at configure time after its existence. ConvertType uses the `-dPDFFitPage' option to Ghostscript which was introduced in version 9.10, so older versions would pass configure but at `make check' time, the PDF test would fail. Now this test is skipped (bug #47868). Most tests would fail when `make check -jN' was run (to do the checks on N threads). A dependency structure has now been defined to fix this problem and greatly speed up the testing process (bug #47957). ** New utilities Arithmetic: For arithmetic operations on the pixels of input images. With this utility, it is now possible to add multiple images with each other, or easily calculate a median image. It as a large set of other arithmetic operations and some functions which can be done on the input image(s). An unlimited number of input images can be given, the images will only be loaded when necessary and will be freed as soon as they are no longer necessary. CosmicCalculator: For doing cosmological calculations at a given redshift. ** New features All the utilities that would produce a log file now have a `--nolog' option to avoid printing a log file. The tiled image compression convention (.fits.fz, created with `fpack') files can now be used as input in the utilities. ImageCrop can now also crop a polygon from the input image. The polygon vertices can be given in the world or image coordinates. The simple `--polygon' option will keep the insides of the polygon while the `--outpolygon' will keep the outside of the polygon. ImageCrop and ImageWarp can now read the WCS information of a FITS header from a specific region with the `--hstartwcs' and `--hendwcs'. In some older FITS images, when the WCS distortions were not as standardized as now, there were cases which would confuse WCSLIB. NoiseChisel can now save the grown clumps image instead of the original clumps image in the output with the `--grownclumps' option. Convolve can now do deconvolution with the `--makekernel' option. MakeProfiles now has a `--setconsttonan' option which will fill the constant profiles with a NaN (blank) value, not a number, allowing the creations of elliptical masked regions for example. Header can now import a keyword directly from a string with the `--asis' option. MakeCatalog can now output the geometric (average position independent of pixel flux value) positions of the objects too. MakeCatalog can now produce the object's elliptical parameters (for example semi-major axis, semi-minor axis, and position angle). This can also be done both in standard flux weighted and geometric methods too. MakeCatalog now has a `--threshold' function to only use pixels above a given threshold in each object or clump. This is useful to avoid diffuse regions in calculations. MakeCatalog now has a `--noriverbrightness' option. With this option it is possible to calculate the clump flux without subtracting the river pixels on its circumference. The number of CPU threads is no longer a configuration option, it is now determined at run-time for each program. Therefore it is now easily possible to built Gnuastro on one system to use on another (commonly done in the GNU/Linux package managers). Therefore ./configure no longer has a `--with-numthreads' option. Every commit in Gnuastro's history (after implementing this feature) can now be given a unique version number. Since the version number is printed in possible outputs, this feature can help reproducibility, even when the an official/stable release isn't used. The AUTHORS file is now automatically generated from the version controlled history. Also all the authors that have contributed to Gnuastro are included in the second (copyright) page of the PDF book. All the bootstrapped directories are now moved with a new `bootstrapped' directory in the top source directory. This significantly cleans up this directory, allowing users to more easily find the hand-written Gnuastro source files they like. A `bug-gnuastro' Info page was created so users can easily go to that page for information on how to submit bug reports. It is accessible on the command line with the command `info bug-gnuastro'. ** Changes in behavior The separate utilities no longer have a separate version number. With the introduction of unofficial version numbers generated from each commit in Gnuastro's history, the utility version numbers would cause confusion. MakeProfiles will now add a suffix to the individual images and put them in the output directory if specified. NoiseChisel and MakeCatalog now use the median mesh standard deviation to define the over-all depth of the image. Previously they used the maximum value (or the least depth). NoiseChisel no longer outputs a sky subtracted image. This job can now be done with the new Arithmetic utility. NoiseChisel's `--segsnhistnbins' option was renamed to `--clumpsnhistnbins'. ** Improvements NoiseChisel's default quantiles were changed to 0.95 from 0.99. Since the old value was too severe. NoiseChisel's S/N thresholds are now found from the distribution of pseudo-detections and clumps from the full image, not within each large mesh. This was done to increase the accuracy of the S/N threshold. There were commonly not enough points in large mesh sizes and this would add scatter. With ImageCrop's polygon capabilities, it is now easily possible to cut out the region that has uniform noise properties (depth and correlated noise). Therefore the old `--checkdetectionsn' and `checkclumpsn' options are no longer present. When building from the version controlled source, the whole bootstrapping process is done with one script. In the previous version, all the separate operations should have been done by hand (as instructed in the old manual). All the build steps now report what was done and suggest the next step. This feature can be disabled with the `--disable-guide-message' at configure time.