GNU Astronomy Utilities NEWS -*- outline -*- Copyright (C) 2015-2019 Free Software Foundation, Inc. See the end of the file for license conditions. * Noteworthy changes in release 0.10 (library 8.0.0) (2019-08-03) [stable] ** New features Installation: - With the the following options at configure time, its possible to build Gnuastro without the optional libraries (even if they are present on the host system): `--without-libjpeg', `--without-libtiff', `--without-libgit2'. All programs: - When an array is memory-mapped to non-volatile space (like the HDD/SSD), a warning/message is printed that shows the file name and its size. Later, when its deleted, a warning/message is also printed, informing you that it has been deleted. These warnings can be very useful when you actually have enough RAM, but forget to increase the `--minmapsize' value (therefore significantly slowing down the program). When you don't have enough RAM, but don't want to be annoyed with the warnings, you can use the new `--quietmmap' option to disable them. Arithmetic: - `unique' operator removes all duplicate (and blank) elements from the dataset and returns a single-dimension output, containing only the unique values in the dataset. Crop: - Can also crop 3D datasets (data cubes). A 3D crop has the same syntax as the old 2D mode, only when the dataset is 3D, three coordinates (values, ranges or catalog-columns) should be given to the relevant option. Just note that `--polygon' crops are still not supported in 3D. CosmicCalculator: --obsline: alternative way to set the used redshift. With this option instead of explicitly giving the redshift, you can give a rest-frame and observed wavelength and the redshift to use will be calculated internally. For many lines, it is possible to give the line name instead of its rest-frame wavelength. For example `--obsline=lyalpha,6000' will use the redshift where the Lyman-alpha line has been shifted to 6000 Angstroms. --usedredshift: Print the used redshift as a "Specific calculation" (in line with other single-valued calculations). Fits: --primaryimghdu: Copy/cut the given image HDU to the zero-th/first HDU of the output file that doesn't yet exist. Statistics: --sigclip-number, --sigclip-median, --sigclip-mean, --sigclip-std: Do sigma-clipping and only print the desired value as a single-value measurement. Until now sigma-clipping results included a lot of visually useful information, which also made automatic usage of results hard. These options fix this issue. Please see the example in the book under `--sigclip-median' for a nice use case. Table: - Column arithmetic. It is now possible to apply many operations on the input table columns before printing them in the output. Similar to Arithmetic, but on table columns. The operators and notation is just like the Arithmetic program. See the "Column Arithmetic" section of the book for a detailed discussion and several examples. - WCS to Image coordinate conversion with `wcstoimg' and `imgtowcs'. For example if the input catalog has at least an `ID' column and two `RA' and `DEC' columns, the set of options below will produce 5 columns where the last two columns are the image coordinates for each row based on the WCS in `a.fits': `-cID,RA,DEC -c"arith RA DEC wcstoimg" --wcsfile=a.fits' --head: Only output the given number of rows from the top of columns. --tail: Only output the given number of rows from the bottom of columns. Library: - New `speclines.h' library functions and macros related to spectral lines. It has many macros with line wavelengths, and several functions for using them in combination with their names. - list.h: Functions to return the last element in linked lists. For example `gal_list_sizet_last' or `gal_list_data_last'. - gal_arithmetic_operator_string: Return operator string from code. - gal_arithmetic_set_operator: Return operator code from string. - gal_blank_initialize_array: Initialize an array with blank values. - gal_dimension_remove_extra: Remove extra (length 1) dimensions. - gal_list_data_to_array_ptr: Make an array of pointers from the list. - gal_fits_img_info_dim: Only return the size information of a dataset. - GAL_TYPE_INT: Corresponding to respective width based on system. - GAL_TYPE_UINT: Corresponding to respective width based on system. - GAL_BLANK_INT: Blank value for `int' (can be 16-bit or 32-bit). - GAL_BLANK_UINT: Blank value for unsigned `int' (can be 16-bit or 32-bit). ** Removed features ** Changed features Installation: - Better `./configure' tests (using Gnulib's `AC_LIB_HAVE_LINKFLAGS') to avoid some crashes during `make' when the host had multiple conflicting versions of some dependencies (GSL in particular). Arithmetic: - The output of co-adding operators is no longer the same type as the input in general. The output of the `min' and `max' operators are still the same type as the input. However the `number' and `sigclip-number' operators will output an unsigned 32-bit integer type and the rest (`sum', `mean', `std', `median', `sigclip-median', `sigclip-mean' and `sigclip-std') return 32-bit floating point datasets MakeCatalog: - When a clumps catalog is requested, MakeCatalog will automatically deduce the total number of clumps (at a small cost in performance). Until now, it was mandatory for the clumps label dataset to contain the total number of clumps in the `NUMLABS' keyword. Library: - gal_statistics_outlier_flat_cfp: Improved implementation with new API. - New `quietmmap' argument added to the following functions (as the argument following `minmapsize'). For more, see the description above of the new similarly named option to all programs: `gal_array_read' `gal_array_read_to_type', `gal_array_read_one_ch', `gal_array_read_one_ch_to_type', `gal_data_alloc', `gal_data_initialize', `gal_fits_img_read', `gal_fits_img_read_to_type', `gal_fits_img_read_kernel', `gal_fits_tab_read', `gal_jpeg_read', `gal_label_indexs', `gal_list_data_add_alloc', `gal_match_coordinates', `gal_pointer_allocate_mmap', `gal_table_read', `gal_tiff_read' and `gal_txt_image_read' Book: - The two larger tutorials ("General program usage tutorial", and "Detecting large extended targets") have been broken into subsections for easier readability. - The "Hubble visually checks and classifies his catalog" tutorial has been removed because it didn't come with a dataset, so it was hard for people to use. Also, all its operations were already present in the general tutorial. ** Bugs fixed bug #56195: astscript-sort-by-night crashing because of AWK. bug #56246: Single-valued measurement must abort with no value in Statistics. bug #56256: Segmentation fault when reading plain text array/image. bug #56257: ConvertType: Values not preserved when converting text to FITS. bug #56299: CosmicCalculator fails at z=0. bug #56324: Column metadata not usable when input is from pipe/stdin. bug #56424: Warp crashes with empty string given to options. bug #56480: Segfault in statistics library's histogram function. bug #56641: MakeProfile's center position changes based on precision. bug #56635: Update tutorial 3 with bug-fixed NoiseChisel. bug #56662: Converting -R to -Wl,-R causes a crash in configure on macOS. bug #56671: Bad sorting with asttable if nan is present. bug #56709: Segment crash when input has blanks, but labels don't. bug #56710: NoiseChisel sometimes not including blank values in output. * Noteworthy changes in release 0.9 (library 7.0.0) (2019-04-17) [stable] ** New features All programs: --checkconfig: print the names and values given to options as they are parsed on the command-line or in various configuration files (the configuration file name is also printed). This can be very useful in debugging (finding which configuration file is responsible for a given option's value). Arithmetic: - The new `tofile-' and `tofilefree-' operators will save the top operand into a file. They can be very handy in debugging/understanding an Arithmetic command (especially as it gets complicated), or to produce multiple files/extensions with a single call to Arithmetic. - Four new operators have been added to allow coadding multiple datasets into one using sigma-clipping: `sigclip-number', `sigclip-mean', `sigclip-median', and `sigclip-std'. These are very useful when several inputs have strong outliers that affect the median, or the mean is required. - Multithreaded operation for the following operators that combine/co-add multiple inputs into one output with same size: `min', `max', `number', `sum', `mean', `std', `median', `sigclip-number', `sigclip-median', `sigclip-mean', `sigclip-std'. --wcsfile and --wcshdu: these two options can be used to specify a different file for reading the WCS of the output. This is useful when the default (the WCS of the first dataset that is read) is not the required one. --interpmetric: new option that is necessary with the `interpolate-medianngb' operator. For more, see the description of this option in NoiseChisel. Fits: - Add "title" to group FITS keywords with `--write=/,"title name". This "title" is composed of two keyword records/lines: a blank one (all whitespace), followed by another starting with `/' and ending in any string given to this option. This visually separates the keywords and acts as a title. Classifying the keywords into contextually similar groups greatly helps in visual inspection and is encouraged. - Calculate and write `CHECKSUM' and `DATASUM' integrity keywords into the specified header using `--write=checksum' (for both) or `--write=datasum' (only for `DATASUM'). --datetosec: Convert the FITS date format (old or new) to number of seconds since since the Unix epoch time (1970-01-01,00:00:00). The FITS date format (for example `YYYY-MM-DDThh:mm:ss') is hard to use for automatic processing (requires calendar peculiarities like number of days in each month, or leap years and etc). With this option a single integer is returned that can be used for example to sort FITS files by date keywords without worrying about calendar peculiarities. --verify: confirm if the `DATASUM' and `CHECKSUM' keyword values agree with the specified HDU's content and/or data. --copykeys: Copy several keyword records (in a given range) from one FITS HDU/extension into another (possibly in another file). --outhdu: The name/number of the output HDU (for `--copykeys'). Match: - All the columns from one of the input catalogs can now be merged with any of the columns of the second using the special `_all' name of `--outcols'. For example the output of `--outcols=a_all,b5' will contain all the columns from the first input and the 5th column of the second input. This greatly simplifies the merging of different table columns into one. --coord: manually specify coordinates to match on the command-line. Until now, if you only wanted to make check a specific coordinate's matching with a catalog. It was necessary to make a single-row catalog as a file and feed that into Match. With this option, you can now specify the coordinates to match against another catalog with the command-line. NoiseChisel: --interpmetric: Set the metric to use to identify the nearest neighbors for tile interpolation (quantile threshold, initial Sky, and final Sky). Until now only the manhattan/taxicab metric was available, which is fast, but could cause 45-degree lines in the interpolation. From this version, with this option, its also possible to use the radial distance (which is now the default). Just note that if there are many tiles over the image, the radial distance will be slower. --snthresh: Manually set the signal-to-noise ratio of true pseudo-detections. With this option, NoiseChisel will not attempt to find pseudo-detections over the noisy regions of the dataset, but will directly go onto applying the manually input value. - Several new options have been added to remove NoiseChisel's dependence on values that were hard-coded in its source and thus not modifiable at run-time by the user. To allow full configurability, these steps can also be configured by the user. --pseudoconcomp: allows setting the connectivity (4 or 8, in a 2D image) to define separate pseudo-detections. If its stronger, pseudo-detections that are touching on the corner will be identified as one. --dopening: The number of openings to do after applying `--dthresh'. --dopeningngb: The connectivity (4 or 8) to use for `--dopening'. Statistics: --interpmetric: Similar to NoiseChisel. Table: --range: Limit the output rows to those with a value within the given numeric range with this format: `--range=COL,low,high'. This is very useful when only certain rows of the input are required not the output. The advantage over piping to AWK is that you can save the output directly to FITS (preserving the metadata). See the book for more. --sort: Sort the output rows based on the value of the given column in ascending order. --descending: When called with `--sort', will arrange the output rows in descending order. Installed scripts: With this release, Gnuastro also installs Bash scripts for common higher-level usage of (possibly multiple) programs. These scripts have a `astscript-*' name, to easily show up on the command-line as Gnuastro executables with the other Gnuastro programs, but are identifiable from them. They support options just like the programs (which can be listed with `--help'). Please see the new "Installed scripts" chapter of the book for more. - astscript-sort-by-night: New Gnuastro executable, using Gnuastro's Fits program to identify files with dates in the same night (possibly spanning two calendar dates). Library: GAL_BLANK_LONG: new macro for the `long' type blank value. GAL_BLANK_ULONG: new macro for the `unsigned long' type blank value. gal_blank_number: Return the number of blank elements in a dataset. gal_dimension_dist_radial: Radial distance between two coordinates. gal_fits_key_date_to_struct_tm: FITS date format to C broken-down time. gal_fits_key_date_to_seconds: FITS date format to Unix epoch time. gal_qsort_index_single_TYPE_i: Set of functions to sort indexs ascending. gal_qsort_index_single_TYPE_d: Set of functions to sort indexs descending. gal_statistics_outlier_cumulative: Uses flatness of the cumulative distribution to find outliers. gal_table_list_of_indexs: returns the list of indexs matching columns. gal_type_is_int: to see if we have an integer (any width, any sign). ** Removed features ** Changed features Arithmetic: - `num' operator is renamed to `number'. - `numvalue' operator is renamed to `numbervalue'. - `--dontdelete' will append the output to any existing file. Note that this change is only in Arithmetic, other programs will still just complain and abort. ConvertType: --forcemin & --forcemax: until now, `--flminbyte' and `--flmaxbyte' were used to force the range of conversion to color channels (even if the range is beyond the limits of the dataset). With the introduction of color maps in Gnuastro 0.8, it is also necessary to force a range on non-byte datasets. It is thus necessary to use a more generic name. MakeCatalog: --std: Until now, this option would measure the mean standard deviation under the label. But this is not a statistically meaningful measure for the Sky standard deviation and could be incorrectly used. From now on, this option measures the square root of the mean variance, or root mean square (the correct definition of the Sky standard deviation). NoiseChisel: --ignoreblankintiles: Until now `--ignoreblankinsky', would specify if blank values should also be written into the tiled Sky and Sky standard deviation outputs. But NoiseChisel can optionally produce many more tiled outputs (for example with `--checkqthresh'). So the option was renamed to `--ignoreblankintiles' to highlight that the status of blank elements can be set in all tiled outputs. Statistics: --ignoreblankintiles: similar to same option in NoiseChisel. Table: --colinfoinstdout: now corresponds to the `-O' short option. Until this version, the `-s' short option was used for it. But with the new `--sort' option, `-s' may cause confusion. Library gal_arithmetic: new argument: number of threads to use (when relevant). gal_eps_write: new argument: optional bit-optimization with `dontoptimize'. gal_pdf_write: new argument: optional bit-optimization with `dontoptimize'. ** Bugs fixed bug #55313: Fits program writing --write values in reverse order bug #55333: Fits program crash when --write or --update have no value. bug #55439: Arithmetic segmentation fault when reusing dataset name. bug #55478: Memory mapping crashes when .gnuastro is not writable. bug #55491: NoiseChisel crash when no tiles good for quantile thresholding. bug #55544: Arithmetic's output WCS with where operator is not as expected. bug #55740: Diamond shapes in nearst-ngb interpolation affecting NoiseChisel. bug #55763: Crop not keeping Blank pixels on unsigned types. bug #55844: WCS library (and thus all programs) can't deal with CROTAi values. bug #55845: Crash when necessary column not found from standard input. bug #55988: MakeProfiles segfault when input catalog has no rows. bug #56001: Bad tile sizes with only one tile and small remainder. bug #56048: Crash when WCS cannot be written to FITS file. * Noteworthy changes in release 0.8 (library 6.0.0) (2018-12-28) [stable] ** New features All programs: - Input files and parameters written as keywords in the first extension of output when it is FITS. This is only relevant for some programs, (for example not the Fits or Table programs). - Standard input (for example from pipes) is now available to feed input to all programs that accept plain text input (ConvertType, Convolve, Match, MakeProfiles, Statistics, Table). - Updated acknowledgment statement (output of `--cite' option). Arithmetic: --onedasimage: write output as an image if it has one dimension, not table. ConvertType: --colormap: color map to display a single-channel dataset (for example FITS image) with a range of colors in formats that support color (like JPEG, or PDF). Until now, the only available mapping was grayscale. Now "Hue, Saturation, Value" (HSV) and "SLS" (from SAO DS9) colormaps are also supported. --rgbtohsv: Convert the RGB input channels to HSV (when the output is in FITS format). Convolve: - Convolves 1D arrays (table columns, for example spectra) also. Therefore two new options have been added to it: `--column' (`-c', similar to other programs that read table columns), and `--kernelcolumn' (to specify the column of the kernel in its own file/extension). Fits: --numhdus: prints the number of HDUs in the given FITS file. NoiseChisel: - New outlier identification algorithm for quantile thresholds. This is very useful when there are extended and bright sources in the dataset: the tiles containing very faint signal that pass the general pixel-value distribution test due to the flatness of the extended profiles, can be identified and removed as outliers in comparison with the other passed tiles. The outlier finding algorithm (`gal_statistics_outlier_positive': a new library function) uses the distribution of distances between the sorted elements and is configured with these options. --outliersclip: Sigma-clipping parameters for the process. --outliersigma: Multiple of sigma to define an outlier. --blankasforeground: Treat blank elements as foreground (regions above the threshold) in the binary processing steps: initial erosion and opening as well as the filling holes and opening of pseudo-detections. From this version, by default, blank elements in the dataset are considered to be background, so if a foreground pixel is touching it, it will be eroded. This option is irrelevant if the datasets contains no blank elements, but can help remove false positives that are touching blank elements. --holengb: allows defining the connectivity of the holes that are filled when defining pseudo-detections. Until now, this was hard-wired into the code (=8) and not modifiable at run-time. --skyfracnoblank: Ignore blank pixels when estimating the fraction of undetected pixels for Sky estimation. NoiseChisel only measures the Sky over the tiles that have a sufficiently large fraction of undetected pixels. This is done to decrease the bias caused by faint un-detected wings of bright galaxies or stars, see description of `--minskyfrac' for more. Until now the reference for this fraction was the whole tile size (irrespective of how many blank elements it contains). With this option, it is now possible to ask for ignoring blank pixels when calculating the fraction. This is useful when blank/masked pixels are distributed across the image. Statistics: - If an input table has only one column, Statistics won't complain and abort when no `--column' (`-c') is given: there is only one column to use anyway, so it will be used. In the absence of which column to use, it will still complain and abort if the input has more than one column. - Sky estimation: new outlier estimation algorithm similar to NoiseChisel. - Input can be given using the standard input (for example a pipe). Library: - gal_blank_flag_apply: Set all flagged/masked elements to blank. - gal_fits_key_list_reverse: Reverse the given list of FITS keywords. - gal_fits_key_write_title_in_ptr: Write a two-line title FITS keyword. - gal_fits_key_write_in_ptr: New name of gal_fits_key_write. - gal_fits_key_write_version_in_ptr: old gal_fits_key_write_version. - gal_fits_key_write_config: write key list and version as config header. - gal_statistics_outlier_positive: Find the first positive outlier. - gal_txt_stdin_read: Read lines in standard input into a list of strings. ** Removed features NoiseChisel: --mirrordist: mean quantile is now used (not mode), see changes below. --qthreshtilequant: removed due to new outlier rejection algorithm. Statistics: --mirrordist: mean quantile is now used (not mode), see changes below. ** Changed features Arithmetic: - If the output has one dimension, it will be written as a table, not a FITS image/array. This can be changed with the new `--onedasimage' option. Convolve: - The short option for `--numchannels' is now `-n'. Until now, it was `-c', but that would conflict with the short option used for `--column' in all the other programs that also read from a table. MakeProfiles: --mergedsize: new name for the old `--naxis' option. Since the option names and values are now written into the FITS header of the output, this option's name would get confused with the mandatory FITS keyword `NAXIS'. NoiseChisel: - Until now, the mode's quantile was used to identify tiles with no significant signal. But from this version, the mean's quantile in each tile is used instead. The reason is that the mean is more sensitive to outliers (signal). Therefore the old `--modmedqdiff' is now called `--meanmedqdiff' . Statistics: --meanmedqdiff: new name for `--modmedqdiff'. Similar to NoiseChisel. Library: - gal_array_read: list of strings (from standard input) acceptable. - gal_array_read_to_type: list of strings (from stin) acceptable. - gal_array_read_one_ch: list of strings (from stdin) acceptable. - gal_array_read_one_ch_to_type: list of strings (from stdin) acceptable. - gal_data_copy_to_allocated: Also copies string metadata (e.g., name). - gal_fits_key_write: filename and hdu instead of FITS pointer. - gal_fits_key_write_version: filename and hdu instead of FITS pointer. - gal_fits_key_write_filename: write at the top or end of the input list. - gal_statistics_outlier_positive: Window-size is now adjustable (new argument). - gal_table_info: list of strings (from stdin) acceptable. - gal_table_read: list of strings (from stdin) acceptable. - gal_txt_table_info: list of strings (from stdin) acceptable. - gal_txt_image_info: list of strings (from stdin) acceptable. - gal_txt_table_read: list of strings (from stdin) acceptable. - gal_txt_image_read: list of strings (from stdin) acceptable. ** Bugs fixed bug #54493: Warp crashes when type isn't set. bug #54526: Invalid r, q and truncation of point profiles in MakeProfiles. bug #54579: NoiseChisel pseudo-detection failure when dataset is negative. bug #54782: Segment's check image not removing sky clumps some tiles. bug #54810: Arithmetic crash when previously named operand renamed. bug #55025: MakeCatalog's `--prepforconv' option being ignored. bug #55079: Blank EPS or PDF page when width options not given. bug #55157: No sanity check on values given to Crop's --section. bug #55295: Crash when more than one collapse operator called. bug #55298: Arithmetic reading 255 on command-line as blank. * Noteworthy changes in release 0.7 (library 5.0.0) (2018-08-08) [stable] ** New features Installation: --enable-debug: debugging flags, no optimization, no shared libraries. --enable-check-with-valgrind: Run `make check' tests within Valgrind. Arithmetic: - `set-A': Set a name (`A' in this case) for the popped dataset. This allows only reading the dataset it into memory once and possibly using it many times. - `fill-holes': Flip background (0) pixels surrounded by foreground (1). - `collapse-sum': collapse/remove a dimension by summing over it. - `collapse-min': collapse/remove a dimension by using minimum value. - `collapse-max': collapse/remove a dimension by using maximum value. - `collapse-mean': collapse/remove a dimension by averaging over it. - `collapse-number': Number of elements included in the collapse. CosmicCalculator: - Default cosmology set to Plank 2018 results (Paper VI). MakeCatalog: --minx: minimum position along first FITS axis. --maxx: maximum position along first FITS axis. --miny: minimum position along second FITS axis. --maxy: maximum position along second FITS axis. Table: --colinfoinstdout: column information when writing to standard output. Library: - gal_array_name_recognized_multiext: If format contains multiple datasets. - gal_dimension_collapse_sum: collapse/remove a dimension by summing. - gal_dimension_collapse_mean: collapse/remove a dimension by averaging. - gal_dimension_collapse_number: collapse/remove a dimension by number. - gal_dimension_collapse_minmax: collapse/remove a dimension by extremum. - gal_wcs_remove_dimension: Remove a dimension in the given WCS structure. ** Removed features ** Changed features Crop: --checkcenter: the units of value depend on mode (image or WCS). MakeCatalog: - `--checkuplim': new name for `--checkupperlimit'. - `--brightnessnoriver': new name for `--noriverbrightness'. Library: - gal_txt_write: new `colinfoinstdout' argument. - gal_table_write: new `colinfoinstdout' argument. ** Bugs fixed bug #54057: Building failure due to not finding gsl_interp_steffen. bug #54063: Match tests in make check fail randomly. bug #54186: MakeCatalog's --checkupperlimit not keeping output's name. bug #54188: MakeCatalog's Upperlimit not being sigma-clipped properly. bug #54284: Crop segfault when catalog contains no data. bug #54285: make check fails if g++ not present. bug #54286: BuildProgram's configuration file, not built by default. bug #54297: No Match output when --notmatched called and no match. bug #54298: Table not writing array when there are no rows. bug #54312: Crash when CFITSIO doesn't have fits_is_reentrant function. bug #54346: Non '-I' or non '-L' strings in CPPFLAGS or LDFLAGS cause crash. bug #54358: Arithmetic's where, not ignoring blank values in condition. bug #54406: Insufficient sanity checks in mode symmetricity calculation. bug #54430: BuildProgram fails with older Libtool versions on dash. * Noteworthy changes in release 0.6 (library 4.0.0) (2018-06-04) [stable] ** New features Building: - New optional dependency: The TIFF library (libtiff). All programs: - Input image dataset(s) can be in any of the formats recognized by Gnuastro (e.g., FITS, TIFF, JPEG), provided that their libraries (optional dependencies) were present at installation time. New program: - Segment: new program in charge of segmentation over detections. This operation was previously done by NoiseChisel. NoiseChisel is now ONLY in charge of detection. Arithmetic: - erode: Erode the foreground of a binary dataset. - dilate: Dilate the foreground of a binary dataset. - filter-sigclip-mean: sigma-clipped, mean filter operator. - filter-sigclip-median: sigma-clipped, median filter operator. - connected-components: label the connected elements of the input. - invert: subtract the maximum of unsigned types (absorption to emission). - interpolate-medianngb: Interpolate (only blanks) with nearest neighbors. ConvertType: - TIFF images can also be used as input. MakeCatalog: MakeCatalog will only read the datasets necessary for the requested columns. Until now, it would read all the possible datasets and all the intermediate measurements. This is thus major improvement in memory and CPU usage. As a result, the input argument is no longer assumed to be the values file, but the object labels file. Please see the "MakeCatalog inputs and basic settings" section of the book for more. Here is the summary of the new options: --insky: Sky value as a single value or file (dataset). --instd: Sky standard deviation as a single value or file (dataset). --valuesfile: filename containing the values dataset. --valueshdu: HDU/extension containing the values dataset. --clumpscat: Make a clumps catalog also. --noclumpsort: Don't sort the clumps catalog by host object ID. --subtractsky: Subtract the given Sky from the values dataset. --variance: input standard deviation image is actually variance. --checkupperlimit: make a table for random positions and measurements. --geoarea: the number of labeled pixels (irrespective of value). --brightnesserr: error in estimating the brightness. --mean: calculate the mean pixel value within an object or clump. --median: calculate the median pixel value within an object or clump. --upperlimitsigma: position in random distribution (in units of sigma). --upperlimitquantile: position in random distribution (quantile). --upperlimitonesigma: 1sigma value of the random distribution. --upperlimitskew: (mean-median)/sigma or skewness of random distribution. NoiseChisel: - New tutorial on detecting large and extended targets. --rawoutput: only output the detection labels and Sky and its STD. --ignoreblankinsky: don't set the pixels that are blank in the input to blank in the Sky and Sky standard deviation outputs (when `--oneelempertile' is not called). --label: label the connected detections. Until now this was the default behavior. However, from this release, NoiseChisel is only in charge of detection. Segmentation is done by a new program (Segment). Since detection is ultimately just a binary operator, the default output now has an 8-bit unsigned integer type with values of 0 or 1. With this option, you can ask for it to label/count the connected detections instead of the default binary output. Statistics: --manualbinrange: histogram or CFP range can be outside of distribution. --ignoreblankinsky: similar to same option in NoiseChisel. Libraries: gal_array_read: read array from any of known formats (FITS, TIFF, JPEG,...). gal_array_read_to_type: similar to `gal_array_read', but to given type. gal_array_read_one_ch: Read a dataset, abort if it has multiple channels. gal_array_read_one_ch_to_type: Make sure input is in one channel and type. gal_binary_label_holes: label the holes within the foreground. gal_blank_is: check to see if argument is blank in its type or not. gal_eps_name_is_eps: Returns 1 if given filename is EPS. gal_eps_suffix_is_eps: Returns 1 if given suffix is EPS. gal_eps_to_pt: Converts dataset size to PostScript points. gal_eps_write: Writes a dataset into an EPS file. gal_interpolate_1d_blank: Fill blank elements through interpolation. gal_interpolate_1d_make_gsl_spline: Allocate and initalize `gsl_spline'. gal_jpeg_name_is_jpeg: Returns 1 if given filename is JPEG. gal_jpeg_suffix_is_jpeg: Returns 1 if given suffix is JPEG. gal_jpeg_read: Reads input JPEG image into `gal_data_t'. gal_jpeg_write: Writes a `gal_data_t' into a JPEG file. gal_label_grow_indexs: grow known indexs into desired areas. gal_label_watershed: apply watershed algorithm on desired region. gal_label_clump_significance: measure significance of all clumps in region. gal_pdf_name_is_pdf: Returns 1 if given filename is PDF. gal_pdf_suffix_is_pdf: Returns 1 if given suffix is PDF. gal_pdf_write: Writes a dataset into an PDF file. gal_pointer_allocate_mmap: Allocate space in a file, not in RAM. gal_qsort_index_single_d: Sort indexs of single array in decreasing order. gal_qsort_index_single_i: Sort indexs of single array in decreasing order. gal_qsort_index_multi_d: Sort indexs in multiple arrays (different threads). gal_qsort_index_multi_i: Sort indexs in multiple arrays (different threads). gal_tiff_name_is_tiff: check if name contains a TIFF suffix. gal_tiff_suffix_is_tiff: check if suffix is a TIFF suffix. gal_tiff_dir_string_read: convert a string to a TIFF directory number. gal_tiff_read: Read the contents of a TIFF "directory" to `gal_data_t'. ** Removed features NoiseChisel: - Segmentation (and thus the options below) moved to the new Segment program: --onlydetection, --segsnminarea, --checkclumpsn, --segquant, --keepmaxnearriver, --gthresh, --minriverlength, --objbordersn, --grownclumps, --checksegmentation. --skysubtracted: no longer necessary, included in noise measuremnts. MakeCatalog: --skysubtracted: no longer necessary, included in noise measuremnts. Library: - The macros `GAL_STATISTICS_SORTED_NOT', `GAL_STATISTICS_SORTED_INVALID', `GAL_STATISTICS_SORTED_INCREASING', `GAL_STATISTICS_SORTED_DECREASING': these macros are removed because we already have the `GAL_DATA_FLAG_SORT*'' bit-flags in `gal_data_t'. ** Changed features Arithmetic: - filter-mean: a blank value in the input can be non-blank in the output when non-blank elements present in filter. - filter-median: similar to filter-mean. Fits: --history: can be called/written multiple times in one run. --comment: can be called/written multiple times in one run. MakeCatalog: - The `WCLUMPS' keyword in the objects labeled image is no longer used to see if a clumps catalog should also be made. To build a clumps catalog, you can now use the `--clumpscat' option. - Estimation of noise-level is now done per-pixel over the whole label. Until now the average noise level was used. --objectsfile has been removed. The main input argument is now assumed to be the objects file. NoiseChisel: From this release, NoiseChisel is only in charge of detection and won't do segmentation any more. The new Segment program is in charge of segmentation. Many of the changes below are due to this new limited scope of NoiseChisel. --kernel: value `none' will disable convolution. - Renamed options: --convolvedhdu ==> --chdu --wkhdu ==> --whdu --detsnminarea ==> --snminarea --checkdetsn ==> --checksn --detquant ==> --snquant - By default the output detection map is a binary image (values of 0 or 1). - With no output name, the output has a `_detected.fits' suffix. Segment: - [Previously in NoiseChisel]: For finding true clumps, the difference in the peak of the clump and the highest valued river pixel, divided by the noise standard deviation are used. Until now, the total signal-to-noise ratio was used as a criteria. In initial tests, this algorithm was much more promising in detecting clumps over strong gradients and also on flatter gradients. Table: --column: multiple columns (comma separated) can be used in one instance of this option (multiple instances of this option are still acceptable also). Libraries: gal_binary_holes_fill: new name for `gal_binary_fill_holes'. gal_dimension_is_different: new name for `gal_data_dsize_is_different'. gal_fits_img_read: now only reads the data not the WCS, therefore it no longer needs the last two arguments. A subsequent call to `gal_wcs_read' can be used to read the WCS information in the file. gal_pointer_increment: new name for `gal_data_ptr_increment'. gal_pointer_num_between: new name for `gal_data_ptr_dist'. gal_pointer_allocate: replaces `gal_data_malloc_array' and `gal_data_calloc_array', through an argument you can ask for the allocated memory to be cleared or not. gal_qsort_TYPE_i: new name for gal_qsort_TYPE_increasing. gal_qsort_TYPE_d: new name for gal_qsort_TYPE_decreasing. gal_statistics_is_sorted: can now also update the bit-flags regarding the sorted nature of the input (to optimize future calls to the function). gal_statistics_quantile_function: returns `inf' or `-inf' if the given value is smaller than the minimum or larger than the maximum of the input dataset's range. Until now, it would return blank in such cases. gal_statistics_number: the output dataset now has a `size_t' type. Until now it was `uint64_t'. ** Bugs fixed bug #50957: --version output not possible on Mac OS X bug #52979: Many unused result warnings for asprintf in some compilers. bug #53122: Configure time CPPFLAGS and LDFLAGS don't pass to BuildProgram. bug #53142: Crash when printing values with the `--onlyversion' option. bug #53147: NULL value of onlyversion option causing a crash. bug #53226: Match output directory ignored when making multiple files. bug #53230: Statistics program bad results on integer columns with limits. bug #53268: NoiseChisel crash when no growth is possible. bug #53295: MakeCatalog parses area larger than clump. bug #53304: NoiseChisel crash when there is no detection. bug #53312: Fits crash on keyword editing (except --delete). bug #53407: Instrumental noise in MakeNoise should be squared. bug #53424: Sigma-clipping seg-faults when there are no more elements. bug #53580: Warp crash when no WCS present. bug #53825: NoiseChisel not accounting for fully zero-valued tiles. * Noteworthy changes in release 0.5 (library 3.0.0) (2017-12-22) [stable] ** New features Manual/Book: An extended tutorial is added showing some general applications of almost all the programs. This may be a good place to get a feeling of how Gnuastro is intended to be used and some of the programs. New Program and library: Match is a new program that will match two given inputs (currently catalogs). Its output is the re-arranged inputs with the same number of rows/records such that all the rows match. The main work is also done with the new low-level `gal_match_catalog' library function which can also be used in more generic contexts. All programs: a value of `0' to the `--numthreads' option will use the number of threads available to the system at run time. Arithmetic: The new operators `filter-median' and `filter-mean' can be used to filter (smooth) the input. The size of the filter can be set as the other operands to these operators. BuildProgram: The new `--la' option allows the identification of a different Libtool `.la' file for Libtool linking information. BuildProgram: The new `--deletecompiled' option will delete the compiled program after running it. CosmicCalculator: all the various cosmological calculations can now be requested individually in one line with a specific option added for each calculation (for example `--age' or `--luminositydist' for the age of the universe at a given redshift or the luminosity distance). Therefore the old `--onlyvolume' and `--onlyabsmagconv' options are now removed. To effectively use these new features, please review the "Invoking CosmicCalculator" section of the book. Fits: when an extension/HDU is identified on the command-line with the `--hdu' option and no operation is requested, the full list of header keywords in that HDU will be printed (as if only `--printallkeys' was called). MakeCatalog: physical nature agnostic WCS column names. Previously the first WCS axis was always assumed to be RA and the second DEC. So for example even if you had a spectrum (with X and wavelength as the two WCS dimensions), you would have to ask for `--ra' and `--dec'. The new `--w1' and `--w2' options are now generic and don't assume any particular type only their order in the FITS header. MakeCatalog now also uses the CTYPE and CUNIT keywords to set the names and units of its output columns. The `--ra' and `--dec' options are now just internal aliases for `--w1' or `--w2' which will be determined based on the input's CTYPE keyword. Also the new `--geow1', `--geow2', `--clumpsw1', `--clumpsw2', `--clumpsgeow1', `--clumpsgeow2' options replace the old options `--geora', `--geodec', `--clumpsra', `--clumpsdec', `--clumpsgeora', `--clumpsgeodec'. No alias is currently defined for the latter group. MakeCatalog: the new `--uprange' option allows you to specify a range for the random values around each object. This is useful when the noise properties of the dataset vary gradually and sampling from the whole dataset might produce biased results. NoiseChisel: with the new `--convolved' and `--convolvedhdu' options, NoiseChisel will not convolve the input any more and use the given dataset instead. In many cases, as the inputs get larger, convolution is the most time consuming step of NoiseChisel. With this option, you can greatly speed up your tests (to find the best parameters by varying them, for a given analysis). See the book for more information and examples. NoiseChisel: with the new `--widekernel' option it is now possible to use a wider kernel to identify which tiles contain signal. The rest of the steps (identifying the quantile threshold on the selected tiles and etc) are done on the dataset convolved with `--kernel' as they were before. Since it is time consuming, this is an optional feature. NoiseChisel: with the new `--qthreshtilequant' option, it is now possible to discard high-valued (outlier) tiles before estimating qthresh over the whole image. This can be useful in detecting very large diffuse/flat regions that would otherwise be detected as background (and effectively removed). NoiseChisel: the finally selected true detections are now grown based on signal contiguity, not by blind dilation. The growth process is the same as the growing of clumps to define objects. Only for true detections, the growth occurs in the noise. You can configure this growth with the `--detgrowquant' and `--detgrowmaxholesize'. With this new feature it is now possible to detect signal out to much lower surface brightness limits and the detections don't look boxy any more. Cosmology library: A new set of cosmology functions are now included in the library (declared in `gnuastro/cosmology.h'). These functions are also used in the CosmicCalculator program. `gal_table_read' can now return the number of columns matched with each input column (for example with regular expressions), a new argument has been added to allow this feature. `gal_fits_key_img_blank': returns the value that must be used in the BLANK keyword for the given type as defined by the FITS standard. `gal_txt_write' and `gal_fits_tab_write' now accept an extension name as argument to allow a name for the FITS extension they write. `gal_box_bound_ellipse_extent' will return the maximum extent of an ellipse along each axis from the ellipse center in floating point. ** Removed features Installation: The `--enable-bin-op-*' configuration options that were introduced in Gnuastro 0.3 have been removed. By managing the arithmetic functions in a better manner (a separate source file for each operator), compilation for all types (when done in parallel) takes about the same time as it took with the default (only four) types until now. MakeCatalog: `--zeropoint' option doesn't have a short option name any more. Previously it was `-z' which was confusing because `-x' and `-y' were used to refer to image coordinate positions. NoiseChisel: The `--dilate' and `--dilatengb' options have been removed. Growing of true detections is no longer done through dilation but through the `--detgrowquant' and `--detgrowmaxholesize' options (see above). ** Changed features CosmicCalculator: The redshift is no longer mandatory. When no redshift is given, it will only print the input parameters (cosmology) and abort. MakeCatalog: when the output is a FITS file, the two object and clumps catalogs will be stored as multiple extensions of a single file. Until now, two separate FITS files would be created. Plain text outputs are the same as before (two files will be created). `gal_binary_fill_holes' now accepts a `connectivity' and `maxsize' argument to specify the connectivity of the holes and the maximum size of acceptable holes to fill. `gal_fits_img_read' and `gal_fits_img_read_to_type' now also read the WCS structure of the extension/HDU in a FITS file and have two extra arguments: `hstartwcs' and `hendwcs'. With these options it is possible to limit the range of header keywords to read the WCS, similar to how they are used in `gal_wcs_read'. `gal_txt_write', `gal_table_write_log', `gal_fits_tab_write' and `gal_txt_write' don't have the `dontdelete' argument any more. The action they take if the file already exists depends on the file: for FITS, a new extension will be added and for text, they will abort with an error. `gal_tile_block_write_const_value' and `gal_tile_full_values_write' now accept a new `withblank' option to set all pixels that are blank in the tile's block to be blank in the check image. `gal_wcs_pixel_area_arcsec2' will return NaN (instead of aborting) when input is unreasonable (not two dimensions or not in units of degrees). `gal_wcs_world_to_img' and `gal_wcs_img_to_world': Until now, these two WCS conversion functions would explicitly assume RA and Dec and work based on input arrays (so for example it was also necessary to give the number of elements and etc). They now accept `gal_data_t' as input for the input coordinates, thus their API has been greatly simplified and their functionality increased. ** Bugs fixed ConvertType crash when changing values (bug #52010). Arithmetic not accounting for integer blank pixels in binary operators (bug #52014). NoiseChisel segfault when memory mapping to a file (bug #52043). CFITSIO 3.42 and libcurl crash at Gnuastro configure time (bug #52152). MakeCatalog crash in upper-limit with full size label (bug #52281). NoiseChisel leaving unlabeled regions after clump growth (bug #52327). Arithmetic crash with no input tokens (bug #52422). Libtool checks only in non-current directory (bug #52427). * Noteworthy changes in release 0.4 (library 2.0.0) (2017-09-13) [stable] ** New features All programs: `.fit' is now a recognized FITS file suffix. All programs: ASCII text files (tables) created with CRLF line terminators (for example text files created in MS Windows) are now also readable as input when necessary. Arithmetic: now has a new `--globalhdu' (`-g') option which can be used once for all the input images. MakeNoise: with the new `--sigma' (`-s') option, it is now possible to directly request the noise sigma or standard deviation. When this option is called, the `--background', `--zeropoint' and other option values will be ignored. MakeProfiles: the new `--kernel' option can make a kernel image without the need to define a catalog. With this option, a catalog (or accompanying background image) must not be given. MakeProfiles: the new `--pc', `--cunit' and `--ctype' options can be used to specify the PC matrix, CUNIT and CTYPE world coordinate system keywords of the output FITS file. MakeProfiles: the new `distance' profile will save the radial distance of each pixel. This may be used to define your own profiles that are not currently supported in MakeProfiles. MakeProfiles: with the new `--mcolisbrightness' ("mcol-is-brightness") option, the `--mcol' values of the catalog will be interpretted as total brightness (sum of pixel values), not magnitude. NoiseChisel: with the new `--dilatengb' option, it is now possible to identify the connectivity of the final dilation. Library: Functions that read data from an ASCII text file (`gal_txt_table_info', `gal_txt_table_read', `gal_txt_image_read') now also operate on files with CRLF line terminators. ** Removed features ** Changed features Crop: The new `--center' option is now used to define the center of a single crop. Hence the old `--ra', `--dec', `--xc', `--yc' have been removed. This new option can take multiple values (one value for each dimension). Fractions are also acceptable. Crop: The new `--width' option is now used to define the width of a single crop. Hence the old `--iwidth', `--wwidth' were removed. The units to interpret the value to the option are specified by the `--mode' option. With the new `--width' option it is also possible to define a non-square crop (different widths along each dimension). In WCS mode, its units are no longer arcseconds but are the same units of the WCS (degrees for angles). `--width' can also accept fractions. So to set a width of 5 arcseconds, you can give it a value of `5/3600' for the angular dimensions. Crop: The new `--coordcol' option is now used to determine the catalog columns that define coordinates. Hence the old `--racol', `--deccol', `--xcol', and `--ycol' have been removed. This new option can be called multiple times and the order of its calling will be used for the column containing the center in the respective dimension (in FITS format). MakeNoise: the old `--stdadd' (`-s') option has been renamed to `--instrumental' (`-i') to be more clear. MakeProfiles: The new `--naxis' and `--shift' options can take multiple values for each dimension (separated by a comma). This replaces the old `--naxis1', `--naxis2' and `--xshift' and `--yshift' options. MakeProfiles: The new `--ccol' option can take the center coordinate columns of the catalog (in multiple calls) and the new `--mode' option is used to identify what standard to interpret them in (image or WCS). Together, these replace the old `--xcol', `--ycol', `--racol' and `--deccol'. MakeProfiles: The new `--crpix', `--crval' and `--cdelt' options now accept multiple values separated by a comma. So they replace the old `--crpix1', `--crpix2', `--crval1', `--crval2' and `--resolution' options. `gal_data_free_contents': when the input `gal_data_t' is a tile, its `array' element will not be freed. This enables safe usage of this function (and thus `gal_data_free') on tiles without worrying about the memory block associated with the tile. `gal_box_bound_ellipse' is the new name for the old `gal_box_ellipse_in_box' (to be more clear and avoid repetition of the term `box'). The input position angle is now also in degrees, not radians. `gal_box_overlap' now works on data of any dimensionality and thus also needs the number of dimensions (elements in each input array). `gal_box_border_from_center' now accepts an array of coordinates as one argument and the number of dimensions as another. This allows it to work on any dimensionality. `gal_fits_img_info' now also returns the name and units of the dataset (if they aren't NULL). So it takes two extra arguments. `gal_wcs_pixel_scale' now replaces the old `gal_wcs_pixel_scale_deg', since it doesn't only apply to degrees. The pixel scale units are defined by the units of the WCS. `GAL_TILE_PARSE_OPERATE' (only when `OTHER' is given) can now parse and operate on different datasets independent of the size of allocated block of memory (the tile sizes of `IN' and `OTHER' have to be identical, but not their allocated blocks of memory). Until now, it was necessary for the two blocks to have the same size and this is no longer the case. ** Bugs fixed MakeProfiles long options on 32bit big endian systems (bug #51341). Pure rotation around pixel coordinate (0,0) (bug #51353). NoiseChisel segfault when no usable region for sky clumps (bug #51372). Pixel scale measurement when dimension scale isn't equal or doesn't decrease (bug #51385). Improper types for function code in MakeProfiles (bug #51467). Crashes on 32-bit and big-endian systems (bug #51476). Warp's align matrix when second dimension must be reversed (bug #51536). Reading BZERO for unsigned 64-bit integers (bug #51555). Arithmetic with one file and no operators (bug #51559). NoiseChisel segfault when detection contains no clumps (bug #51906). Correct size checking when allocating gal_data_t (bug #52544). * Noteworthy changes in release 0.3 (library 1.0.0) (2017-06-01) [stable] This is a full re-write of Gnuastro. Most importantly, Gnuastro now has a new generic data container (`gal_data_t'). This new container can now deal natively with all standard numeric data types, work in RAM or HDD/SSD, keep data in any dimensions and has enabled many other very useful features. Some of the most prominent of the new features are discussed below. It is strongly recommended to review the respective section of the Gnuastro manual/book for a better feeling of all the new features. As discussed below, some program names have changed, if you have a previous version of Gnuastro installed from source, it is recommended to uninstall it first (with `make uninstall' using the corresponding tarball), then install this new version. Building Gnuastro can be slow, so please build in parallel with Make's `-j8' option (to build on 8 threads). ** New programs or library features Library functions that deal with datasets now use this generic data container for inputs and outputs, significantly simplifying their API. Nearly all library functions have been re-written with much more clear names, argument lists and individual purpose. Some example library functions are shown below, also see the "Library demos" section of the book for some complete working example: -- `gal_table_read' and `gal_table_write' will read and write data to plain text, FITS ASCII and FITS Binary formats. -- `gal_fits_img_read' and `gal_fits_img_write' can read a FITS image to memory or write a FITS image from memory. Gnuastro now defines a simple comment line format to keep basic information in a plain text table, see the "Gnuastro text table format" section of the book. In short for every column, a comment line like below can be used to give a name, units, comments, or a type to a column. This allows a FITS binary table for example to be written to plain text and converted back to binary without loosing any information (except for very small floating point errors if not enough decimals are printed). # Column N: NAME [UNIT, TYPE, BLANK] COMMENT The new Fits program replaces the old Header program. But besides reading/checking FITS header keywords, it can now also work on FITS extensions/HDUs. For example with no options, it will list all the HDUs in a FITS file along with basic information. It can copy a whole HDU to another file, or delete a HDU from a FITS file. To get the previous behavior of listing all the keywords in a FITS HDU, you can run it with the `-p' option. All programs now write data into the second HDU of a FITS file to allow a clean first HDU. Note that following CFITSIO, HDU counting still starts from zero, so FITS images and tables written by Gnuastro in a new file can always be accessed with the `--hdu=1' option (which is now the default). If any program is run within a Git version controlled directory, a `COMMIT' header keyword will be added to the created FITS files, see the "Output headers" section of the book for a discussion on the usefulness of this new feature. BuildProgram: a new program to easily compile, link and run a C program you have written with Gnuastro's libraries without having to worry about which libraries (Gnuastro dependencies) your program needs. Debugging (`-g'), optimizations (`-O'), warnings (`-W'), include search path (`-I'), link search path (`-L'), and linked libraries (`-l') compiler options are also supported. BuildProgram will greatly facilitate the easy usage of Gnuastro's libraries. ** Removed programs or library features The following program names have been renamed: ImageCrop --> Crop ImageWarp --> Warp ImageStatistics --> Statistics Header --> Fits SubtractSky has been removed. The Statistics program now has tools to estimate the Sky value and it can be subtracted with Arithmetic. ** New features All Gnuastro programs that read and write tables can now do so in plain text table format or in FITS ASCII or FITS Binary tables. Depending on the filename or with the new `--tableformat' common option to all programs. The option management system in all Gnuastro programs has been completely re-written with many new features, some of the most important ones are listed below. For developers, you will notice that there is no more usage of macros and adding new options has become much more easier. -- All programs will now also look for a `gnuastro.conf' configuration file to keep common options for all programs in every directory. -- The `--lastconfig' option can be used on the command-line or in any configuration file to stop parsing any further configuration files. -- The `--config' option can now be used to identify any arbitrary file to be parsed as a configuration file. Any file that is given to this option is parsed immediately. -- The `--printparams' option now also prints the short documentation of each option (same description in `--help') after its value. It is now possible to choose columns in tables based on column names as well as column numbers. It is also possible to search for columns based on searching in their units or comments. When using column numbers, counting now starts from 1 (one), not 0 (as before). See the new "Selecting table columns" section for more on these new features. Where relevant, all programs now accept a `--type' option that you can use to specify the numerical datatype of the output. With the new common option `--minmapsize', you can specify a minimum size of an array (in bytes) to store data in SSD/HDD and not in RAM. This can be instrumental when you are dealing with large datasets, or even smaller ones, but when your RAM is getting full. Making a log file is now optional and users have to explicitly ask for it with the `--log' option. Slower building of Gnuastro: Binary operators (e.g., plus or multiply) are now done in the native type of the input dataset. Doing so for all the different combinations of types, greatly slows down the initial compilation of Gnuastro (after running `make'). So for every type there is now a `--enable-bin-op-*' configure time option. When the dataset's type isn't compiled (only for the binary operators), it will be converted to a compiled type and then converted back in the end. Arithmetic: all operations are done in the native data type of the dataset. Until now, it would convert the data internally to double precision floating point, do the requested operation and write the data back in the proper type. So this new implementation is much more efficient. Arithmetic: two new classes of operators: type conversion operators to all standard types and integer-only operators (all C bit-wise operators and the modulo operator). ConvertType: can also print the input dataset to the command-line (`stdout'). To use this feature set the output filename to `stdout'. Convolve now has the `--minsharpspec' option to specify the minimum spectrum value to use in deconvolution (matching PSFs). Crop: when in WCS mode it can still only work on aligned images. However, very small floating point errors in writing the WCS (for example 10e-13 degrees) are now acceptable. Until now these would cause Crop to complain and abort. Crop: the name of the crop can be pre-determined based on the values in a given in a table column. This allows your object's IDs to be directly used as the crop's file name for example. MakeCatalog now also reports the surface brightness in mag/arcsec^2. Until now, MakeCatalog would only print the dataset's surface brightness in units of magnitudes/pixel. But that is not nicely comparable to other datasets. Hence, using a simple calculation (from the pixel projected size, fully derived in the book), it now also reports the surface brightness in magnitudes/arcsec^2 also. MakeProfiles: Profile codes now start from `1' (until now they started from `0'). MakeProfiles: now accepts the radial function of profiles as human-readable strings instead of a code for each profile (which was very cryptic, although codes are also still acceptable). For example in the profile column you can now write `sersic' instead of the code `1'. NoiseChisel: the new `--cleandilated' option will remove dilated objects that have a low S/N (it is mainly useful on very clean or mock images). For non-clean noise, it will result in a decrease of completeness. With this new option, NoiseChisel will also print detection S/N values when run with the `--checkdetsn' option. Statistics: now reads table columns as well as images and does basic operations on them. It can also only work on a certain range of the data (instead of the whole set). Alternatively, you can define the range on another reference column, but use values of the main column. Statistics: all its single-valued measurements can now be done on a tessellation (tile grid) over the input dataset. Statistics: can now estimate the Sky value on the input dataset using the mode's quantile similar to what NoiseChisel does to find its initial threshold. Statistics: has several new single valued calculations: `--quantile', `--quantfunc' (quantile function), `--mode', `--modequant', `--modesym', and `--modesymvalue'. Warp: align the image with the celestial coordinates using the `--align' option. Warp: standard modular warpings can now be requested without an input matrix, using the following options: `--shear', `--flip', `--project', `--rotate', `--scale', `--translate'. Any number of these transformations (along with the `--align' option) can be called on the command-line and they will be applied in the same order to create one warping matrix. By default the WCS will also be corrected. ** Changes in behavior Mask image options have been removed from all programs. Instead, all programs can work directly on data with blank values. So when some pixels must be masked, the Arithmetic program's `where' operator can be used to select special pixels and set them to blank. In particular bit-wise operations are now available in Arithmetic to use bit-mask images. Managing all these different choices in every program would only confuse the user (with too many options). Arithmetic: the `x' letter is now used to represent the multiplication operator. Previously it was `*' which needed quotation and was thus very inconvenient. Convolve: the old `--frequency' and `--spatial' options have been removed and are replaced by `--domain' which accepts values of `frequency' and `spatial'. Convolve: the old `--viewfreqsteps' was changed to `--checkfreqsteps' to fit with the general style of such check images in all Gnuastro's programs. Crop: `--section' syntax is now inclusive in both bounds. Crop: only checks if the center of a crop is filled when the crop was defined by its center (for example with `--ra' and `--dec'). The verbose outputs of Crop are also not cryptic 0s or 1s. The are human readable text. Crop: doesn't have separate `--imgmode' and `--wcsmode' options any more. There is now a single `--mode' option which accepts values of `img' or `wcs'. MakeProfiles: the old `--inputascanvas' is now called `--clearcanvas'. MakeProfiles: until now, it would abort with an error when the input columns had blank values. But for masking, it might happen that you set a blank magnitude. So this check has now been removed when reading the magnitude column. NoiseChisel: default value of the `--minskyfrac' option (new name for the old `--minbfrac') is now 0.7 as opposed to 0.5. This will allow much better estimation of noise properties (by default). It may be slightly too high for a crowded field, but the users can change it on the command-line (or in a configuration file) for such datasets. NoiseChisel: when it is run with any of the `--check' options, it will abort after all the check images have been created. This is very useful for checking your parameters until each step and not be distracted (or have to wait) for later steps to finish. Statistics: will not make a histogram and cumulative frequency files, or calculate sigma-clipped results by default (with no options). It will just print some basic information. Table: Previously, if a column was requested, the `-i' option would be ignored. But it often happens that the users forget a column name after already typing several of their desired columns. So the opposite behavior is preferred. Because when more than a couple of columns are needed, you will probably forget the column identifiers of the last few and having to retype everything is very frustrating. Something like how `--help' takes precedence over all other options. Table: to select column(s) by regular expression searching, the name now has to be put in `/ /' (similar to AWK). If a value isn't in `/ /', the programs will only select a column with the exact match. Warp: when a 2 by 2 matrix is given, the FITS pixel positions (which define the center of a pixel as an integer) are automatically implemented internally, see "Invoking Warp" in the manual for more. Warp: the old `--nofitscorrect' option has been changed to `--centeroncorner' to be more clear. The new option is now more general than before and also works on warping with a matrix, not just on modular warpings. Warp: the old `--nowcscorrection' option has been given a more clear name of `--keepwcs'. With this option, Warp will not apply the warp the input's WCS structure. Warp: the old `--maxblankfrac' option has been changed to `--coveredfrac'. Until now, Warp would only look for the fraction of input blank/NaN pixel area over the output pixel. But this would be useless on the edges of the image. So the new `--coveredfrac' option takes the acceptable fraction of output pixel area that must be covered by input pixels in order to give that output pixel a value. You can use this to set edge pixels that are not fully covered in the new grid to blank and have a flat warped image. ** Bugs fixed Using `%zu' to print `size_t' variables for clean build on 32-bit systems. Crash in Table for some operating systems due to memory is now fixed (bug #49347). Table's man-page is now created and installed (bug #49418). Fixes in the documentation (sr #109170, bug #49419). Check for malloc returning valid pointer (bug #49459). Segfault in mesh interpolation corrected (bug #49588). Corrected bad status usage in calls to wcsp2s and wcss2p (bug #49752). Stricter checking in Crop's polygon point list (bug #48978). Correction in alignment and getting pixel scale (bug #50072). Decomposing PCi_j and CDELTi matrices in output WCS (bug #50073). Using image naxes[n] when checkcenter is larger (bug #50099). Memory leak in MakeCatalog corrected (bug #51118). Fix copy-paste error in MakeCatalog flag arrays (bug #51130). * Noteworthy changes in release 0.2 (library 0.0.0) (2016-10-03) [stable] ** Bugs fixed Linker errors on some operating systems have been fixed (bug #48076). Several memory allocation, checks or redundancies have been fixed: bugs #48453, #48516, #48603, #48611, #48571, #48650, #48657, #48692, #48770, #47866, #48899, #49049, #49007. ImageCrop no longer crashes with very long output file names (bugs #46241 and #45380). ** New programs or headers Table: a new utility to read and write FITS binary and ASCII tables. It can also print the column information or select columns using regular expressions (task #13579). Shared libraries and headers are now installed. The libraries can be used in C and C++ programs. This release includes the following headers: `gnuastro.h', `array.h', `box.h', `fits.h', `linkedlist.h', `mesh.h', `polygon.h', `qsort.h', `spatialconvolve.h', `statistics.h', `threads.h', `wcs.h', `txtarray.h' (task #13765). Gnuastro now comes with a script in its top source directory (`tmpfs-config-make') to configure and build it in the tmpfs (on the RAM), for those systems that have it. See the new "Configure and build in RAM" section in the book for more (task #14100). ** New features MakeProfiles also accepts WCS positions (task #13566). Flat profiles in MakeProfiles can be given a profile specific value. The new `--mforflatpix' option MakeProfile will use the value in the magnitude column as a fixed value for each pixel. This can be very useful in defining a mask, or creating segmentation maps or labeled images (task #14115). MakeProfiles can now use input image as canvas. Instead of specifying the WSC and image size parameters manually. With the new `--inputascanvas' option, MakeProfiles will get this information (along with blank pixels) from an already existing image (task #14116). Type of output in MakeProfiles and Arithmetic can be specified with the `--type' option. Magnitude error column in MakeCatalog with the `--magnitudeerr' option. Arithmetic now has new conditional (task #13870) and logical operators (task #14153) along with an operator for actions only when conditions are true: `where'. The new `isblank' operator will also enable you to select blank, or masked, pixels (task #14146). The `--noerodequant' in NoiseChisel enables it to detect small and sharper profiles by disabling erosion on pixels above a certain quantile (task #14139). MakeCatalog can also calculate the upper limit magnitude for each given object in the image by randomly positioning the object's footprint over undetected regions of the image (task #14167). The source tarball is now also distributed with Lzip for a much better compression ratio and more robust archival file format. ** Changes in behavior The two MakeProfiles options `--setconsttonan', `--setconsttomin' have been removed (see `--mforflatpix' above for their alternative). MakeCatalog makes clump catalog only when asked (when the `WCLUMPS' header exists in the objects HDU). This can be very useful in cases like aperture photometry, when the object labels are not generated by NoiseChisel and so a clump image doesn't exist (task #14122). Default cosmological parameters in CosmicCalculator set to Plank 2015 results: A&A (2016), 594, A13 (arXiv 1502.01589). The `--envseed' option (to read random number generator type and seed from the environment) to MakeProfiles and MakeNoise can also be given in the configuration files. * Noteworthy changes in release 0.1 (2016-05-30) [stable] ** Bugs fixed MakeCatalog's problem in checking the sizes of all input images is now fixed. NoiseChisel's problem with reading the `--kernel' option is now corrected (bug #46750). lib/mesh.c's problem in correctly calculating the mesh sizes was corrected (bug #47611). `make check' will not look into system utility configuration files. In the previous release, if Gnuastro was already installed, the configuration files already present on the system would also be read. Now only configuration files in the tested package are used (bug #47833). Ghostscript's version is now checked at configure time after its existence. ConvertType uses the `-dPDFFitPage' option to Ghostscript which was introduced in version 9.10, so older versions would pass configure but at `make check' time, the PDF test would fail. Now this test is skipped (bug #47868). Most tests would fail when `make check -jN' was run (to do the checks on N threads). A dependency structure has now been defined to fix this problem and greatly speed up the testing process (bug #47957). ** New utilities Arithmetic: For arithmetic operations on the pixels of input images. With this utility, it is now possible to add multiple images with each other, or easily calculate a median image. It as a large set of other arithmetic operations and some functions which can be done on the input image(s). An unlimited number of input images can be given, the images will only be loaded when necessary and will be freed as soon as they are no longer necessary. CosmicCalculator: For doing cosmological calculations at a given redshift. ** New features All the utilities that would produce a log file now have a `--nolog' option to avoid printing a log file. The tiled image compression convention (.fits.fz, created with `fpack') files can now be used as input in the utilities. ImageCrop can now also crop a polygon from the input image. The polygon vertices can be given in the world or image coordinates. The simple `--polygon' option will keep the insides of the polygon while the `--outpolygon' will keep the outside of the polygon. ImageCrop and ImageWarp can now read the WCS information of a FITS header from a specific region with the `--hstartwcs' and `--hendwcs'. In some older FITS images, when the WCS distortions were not as standardized as now, there were cases which would confuse WCSLIB. NoiseChisel can now save the grown clumps image instead of the original clumps image in the output with the `--grownclumps' option. Convolve can now do deconvolution with the `--makekernel' option. MakeProfiles now has a `--setconsttonan' option which will fill the constant profiles with a NaN (blank) value, not a number, allowing the creations of elliptical masked regions for example. Header can now import a keyword directly from a string with the `--asis' option. MakeCatalog can now output the geometric (average position independent of pixel flux value) positions of the objects too. MakeCatalog can now produce the object's elliptical parameters (for example semi-major axis, semi-minor axis, and position angle). This can also be done both in standard flux weighted and geometric methods too. MakeCatalog now has a `--threshold' function to only use pixels above a given threshold in each object or clump. This is useful to avoid diffuse regions in calculations. MakeCatalog now has a `--noriverbrightness' option. With this option it is possible to calculate the clump flux without subtracting the river pixels on its circumference. The number of CPU threads is no longer a configuration option, it is now determined at run-time for each program. Therefore it is now easily possible to built Gnuastro on one system to use on another (commonly done in the GNU/Linux package managers). Therefore ./configure no longer has a `--with-numthreads' option. Every commit in Gnuastro's history (after implementing this feature) can now be given a unique version number. Since the version number is printed in possible outputs, this feature can help reproducibility, even when the an official/stable release isn't used. The AUTHORS file is now automatically generated from the version controlled history. Also all the authors that have contributed to Gnuastro are included in the second (copyright) page of the PDF book. All the bootstrapped directories are now moved with a new `bootstrapped' directory in the top source directory. This significantly cleans up this directory, allowing users to more easily find the hand-written Gnuastro source files they like. A `bug-gnuastro' Info page was created so users can easily go to that page for information on how to submit bug reports. It is accessible on the command line with the command `info bug-gnuastro'. ** Changes in behavior The separate utilities no longer have a separate version number. With the introduction of unofficial version numbers generated from each commit in Gnuastro's history, the utility version numbers would cause confusion. MakeProfiles will now add a suffix to the individual images and put them in the output directory if specified. NoiseChisel and MakeCatalog now use the median mesh standard deviation to define the over-all depth of the image. Previously they used the maximum value (or the least depth). NoiseChisel no longer outputs a sky subtracted image. This job can now be done with the new Arithmetic utility. NoiseChisel's `--segsnhistnbins' option was renamed to `--clumpsnhistnbins'. ** Improvements NoiseChisel's default quantiles were changed to 0.95 from 0.99. Since the old value was too severe. NoiseChisel's S/N thresholds are now found from the distribution of pseudo-detections and clumps from the full image, not within each large mesh. This was done to increase the accuracy of the S/N threshold. There were commonly not enough points in large mesh sizes and this would add scatter. With ImageCrop's polygon capabilities, it is now easily possible to cut out the region that has uniform noise properties (depth and correlated noise). Therefore the old `--checkdetectionsn' and `checkclumpsn' options are no longer present. When building from the version controlled source, the whole bootstrapping process is done with one script. In the previous version, all the separate operations should have been done by hand (as instructed in the old manual). All the build steps now report what was done and suggest the next step. This feature can be disabled with the `--disable-guide-message' at configure time. * Copyright notice Copyright (C) 2015-2019 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts.