Penny How

Data scientist based in Greenland and Denmark, interested in glacier dynamics, photogrammetry and programming.

PyPI version Anaconda-Server Badge Anaconda-Platform Badge Anaconda-Server Badge Dataverse DOI JOSS DOI Documentation Status

I am one of the developers of pypromice, which is designed for processing and handling PROMICE (Programme for Monitoring of the Greenland Ice Sheet) and GC-Net (Greenland Climate Network) automated weather station data.

It is envisioned for pypromice to be the go-to toolbox for handling and processing PROMICE and GC-Net data, along with other weather station datasets. New releases of pypromice are uploaded alongside PROMICE weather station data releases to our Dataverse portal for transparency purposes and to encourage collaboration on improving our data. Please visit the pypromice readthedocs for more information.

How pypromice came to be

pypromice first came to be in June 2022 as a merge of repositories that handled and processed data from automated weather stations. Lots of floating code suddenly had a home under a unifying source.

There were a couple of motivations to do this, the main one being that our department had recently assumed responsibility of a new set of weather stations called GC-Net, which are a different station design. Our current processing only accomodated for our current PROMICE weather stations, and therefore the workflow needed to be built out in order to process data from these new additions.

Another motivation was that the general users of the PROMICE weather station data were those from the scientific research community, mainly cryospheric sciences. Yet, weather observations are incredibly valuable datasets across many disciplines. We needed a better handle of our processing workflow and the outputted data formatting in order to reach other discipines and increase the usability of our data.

Finally, one of the big long-term goals with the PROMICE weather station network was for observations to be processed in near-real-time. In order to do so, the current processing needed to be repackaged into something that was easily deployable and computationally efficient.

What pypromice can do

The main thing that pypromice can do is perform all of the processing steps to transform, correct, flag and derive measurements from a weather station's raw, original data. We process in three steps, from the raw, original data (which we call Level 0) to a user-friendly and finalised data product (called Level 3).

pypromice processing workflow
The workflow for pypromice’s Level 0 to Level 3 weather station data processing

In addition, pypromice has automated quality control functionality which flags and fixes weather station data, and post-processing capabilities to format weather station data for input to international weather observation and forecasting databases. Up-to-date PROMICE and GC-Net weather station datasets can also be called through pypromice's data fetching routines. This ensures that a user has the most recent PROMICE and GC-Net weather observations.

Features of pypromice

  • Object oriented design for fast processing and handling different types of weather station (i.e. one-boom vs. two-boom, on ice vs. on bedrock
  • Powered by .toml file configurations to adequately describe a weather station and its processing procedures
  • Continuous integration with module unit testing and a "dress-rehearsal" of operational processing, which is triggered when any new changes are suggested through a pull request
  • Thorough documentation through the scripts, with auto-doc generation hosted at readthedocs
  • Distributed as a Python package through pypi and conda-forge
  • All pypromice releases backed up and citeable through the GEUS Dataverse

Citing pypromice

If you intend to use our automated weather station datasets and/or pypromice in your work, please cite these publications below, along with any other applicable PROMICE publications where possible:

Fausto, R.S., van As, D., Mankoff, K.D., Vandecrux, B., Citterio, M., Ahlstrøm, A.P., Andersen, S.B., Colgan, W., Karlsson, N.B., Kjeldsen, K.K., Korsgaard, N.J., Larsen, S.H., Nielsen, S., Pedersen, A.Ø., Shields, C.L., Solgaard, A.M., and Box, J.E. (2021) Programme for Monitoring of the Greenland Ice Sheet (PROMICE) automatic weather station data, Earth Syst. Sci. Data, 13, 3819–3845, https://doi.org/10.5194/essd-13-3819-2021

How, P., Wright, P.J., Mankoff, K., Vandecrux, B., Fausto, R.S. and Ahlstrøm, A.P. (2023) pypromice: A Python package for processing automated weather station data, Journal of Open Source Software, 8(86), 5298, https://doi.org/10.21105/joss.05298

How, P., Lund, M.C., Nielsen, R.B., Ahlstrøm, A.P., Fausto, R.S., Larsen, S.H., Mankoff, K.D., Vandecrux, B., Wright, P.J. (2023) pypromice, GEUS Dataverse, https://doi.org/10.22008/FK2/3TSBF0