Tags

Moderner Python

Compatibility with Numpy 2

Main branch release

This release marks the official move of QuaLiKiz-pythontools from
"master" branch naming convention to "main" branch naming convention.
It also contains improvements to plotting and QLKNN run preparation.
A summary of the changes can be found below:
- Backward and forward compatibility fixes and improvements (for python3.6 and python3.10), notably:
   - Better entry points for the Pythontools CLI
   - Better versioning of the package; PyPA-approved and PyPI ready
   - Some comments to explain what is happening in setup.py
- Improvements to plotting !58:
  - Rewrites the plot fluxlike file to be more debuggable and modular
  - Adds clauses to not plot spectra if no spectra are found in the whole dataset
- Move to main !60

Merges in this release in chronological order:
   - !58
   - !59
   - !60

The "Marconi big-run multi-ion" version

Changes are numerous, notably the selection of "big run" improvements
and changes:
* !48 The "Bas rotation Airflow" merge
   * Updated files in scheduling/airflow directory to be compatible with
        version 1.3
   * Updated qualikizrun.py to use pathlib.Path wherever possible
   * Tested using qualikiz_tools==1.3, QuaLiKiz==2.8.1,
   * apache-airflow==1.10.14 in Marconi cluster
   * Note: Path.resolve() function fails when attempting to follow a broken
        symlink. This causes some tests to fail in comparison to the
        os.readlink() implementation when testing against dummy run folders
        with placeholder links. A similar Path.readlink() function will
        supposedly be available in Python 3.9, but currently uses a mix of
        the two.
* !50 The "Bas Macheral rotation" merge
   * Addition of the recalculation of rotation variables when assuming
     no poloidal rotation and no ion density and temperature curvature.
        * Added a function recalc_rot() to the QuaLiKizXpoint.
        * Added an option "recalc_rot_var" to the QuaLiKizXpoint.
        * Perform the recalculation in the QuaLiKizPlan if the
          "recalc_rot_var" is set to True.
        * Added three functions to conversion.py that calculate either
          gammaE, Machtor or Autor.
        * In the QuaLiKizPlan, there are remnants of a mixed generator, which
          scans gammaE and Machtor in parallel but the remaining variables
          rectangularly. This can be reworked in a more general form.
   * For more information, see README_BAS.txt
* !51 The "QLFlux Bas Pythontools extravaganza" merge
   * This MR contains mechanisms to interact with the new QuaLiKiz
     QLFLux scan mechanism, see
     https://gitlab.com/qualikiz-group/QuaLiKiz/-/tree/karel/QLflux.
      * Add the generate_ql subcommand to qualikiz_tools input
      * Implement QuaLiKizQLFluxScan to scan type; for scanning over
      * quasilinear fluxes
      * Tests for QL flux
      * Move to pathlib for paths

The "hobbying of Karel" changes:
* !49 The "Static Code Analysis" merge
   * This MR adds a developer method to create a new file
     qualikiz_tools/qualikiz_io/datamap.json which contains information
     on the sizes, types and "class" of QuaLiKiz in- and output
     variables. This file should not change often, so I'm just version
     controlling it in this repo so one does not need QuaLiKiz itself to
     use the QuaLiKiz-pythontools. This map can be ingested for a
     planned class-based interface to the QuaLiKiz IO, as well as have a
     human and machine readable description of QuaLiKiz IO.
* !52 The "Hotfix for CI" merge
   * Use name instead of stem for building link to qlk binary
   * Fix core detection tests for 10+ cores

The "Improved CI and packaging" changes:
* !53 The "Relative bound, remove binary equivalence" merge
* !54 !55 The "Improved documentation" merges
* !56 The "Python packageing and Heimdalification" merge
   * This MR makes major changes to the QuaLiKiz-pythontools CI, namely:
      * Removes the broken and hacky Heimdall-like install job (6e3f683b)
      * Bring CI in line with other "Karel Pythontools" (39983795)
      * Additionally run the documentation generation on branches named
      * (39983795, fd3ec654)
         * karel/*
         * feature/*
         * bugfix/*
         * hotfix/*
      * Use pip/python cache and venv as in gitlab.com examples. (8f106c8e,c1414148)
      * Use bash env variables for pip and python (beb4c5b5)
      * install_docker_heimdall_like -> install_docker_heimdall_like_user_install and install like it is on Heimdall (0ff745ef, fd833700)
      * Test the non-editable install in install_docker_heimdall_like_user_install_noneditable
      * Build manylinux Python packages (e07e5c9d, 29d20d70)
      * Use default stages build, run, test, and deploy; Removed doc stage. (d9a35126)
      * Remove logic from setup.py and move it to pyproject.toml and setup.cfg (c90cf5a9, fe2cc0ba)
      * Deploy packages to test.pypi (for now) (cb6f040f, 1df9354d, 649bb955, 1c13b256)

The "Improved CLI and plotting" changes:
* !51 The "Fix CLI plotting" merge
   * Fix the plot subcommand for the qualikiz_tools CLI.
      * Uses native Python argparse instead of the extra package docopt. The idea is to eventually move all CLIs to argparse.
      * Has --help and usage information
      * Adds the now-working sub-subcommand plot_fluxlike, which produces a plot of a 1D scan + relevant spectra
   * Replaces the Michele discrete slider by the now-working snapping slider from matplotlib https://matplotlib.org/devdocs/gallery/widgets/slider_snap_demo.html
   * Many fixes to the unholy monster of Frankenstein from @MicheleDiffer and @Karel-van-de-Plassche plot_fluxlike.py
      * Fix since-broken usage of xarray and pandas
      * Fix FutureWarnings from pandas
      * Fix multirun plotting
      * Add more axis labels
   * Add helper function to generate unique filenames (for figures)
   * Add/fix helper functions to generate "pretty names" for QuaLiKiz variables in pretty_names.py
   * Bump matplotlib version number
   * Always regenerate netCDFs on plot

Merges in this release in chronological order:
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/48
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/49
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/50
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/52
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/53
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/54
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/55
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/51
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/56
* https://gitlab.com/qualikiz-group/QuaLiKiz-pythontools/-/merge_requests/57

Modern netCDF pipelines version

* Add "working with netCDF" convenience functions in qualikiz-group/QuaLiKiz-pythontools!47
* Airflow DAG improvements in qualikiz-group/QuaLiKiz-pythontools!48
* Static analysis for QuaLiKiz source code qualikiz-group/QuaLiKiz-pythontools!49

QLKNN11D prep version

* Bump minimal version of xarray to `0.14.1` !46
* Implement new `rhomin` and `rhomax` flags in QLK IO !38
* QOL in packaging !35 !38
* Increased and easier debuggable CI !38 !39
  * Add tests for all supported Python versions !45
  * Add tests for netCDF !45
  * Add CI that simulates an Heimdall environment !31 !33 !36 !46
* Modern re-write of to_netcdf and qlk_from_dir interfaces !45 !56
  * Compress with zlib level 1 by default
  * Convert read-in ASCIIs to float32 by default
* Add Makefile for doing common things, like building docs and packages !40
* Reduces the spammability of pythontools !29
* Changed the default grid to the one in JETTO !29
* Use Black to reformat your code (optional): https://github.com/psf/black !41
* Make QuaLiKiz-pythontools more standard complaint and add tests
  measuring this !42
* Move pythontools documentation from Doxygen to Sphinx !30 !34 !37 !44 !45
  * Docstrings can still be written as before, but will be picked up by a different engine now
  * Use sphinx-apidoc to pull the Python docstrings into rst source
  * Use https://github.com/pypa/setuptools_scm as an explicit package
  * Google style for docstrings: https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings
  * The napoleon extension if you use Sphinx: https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html
* Add tools for "Manual processing pipeline" !35 !43 !46
  * Pure Python and Batch scripts to generate folder structures, generate input, launch jobs,
    collect output and process output
  * The `OrdinaryOrderedDictionaryStructure` based on OrderedDict with
    extra convenience functions
  * Improvements to slurm machine specific Run and Batch
  * QOL of input/output QuaLiKiz file handling
  * Reference run generation is now part of the main package
  * Multi-process Batch.to_netcdf implemented
  * Rename processes to n_processes in to_netcdf

QuaLiKizQuaLiTy version