{
"cells": [
{
"cell_type": "markdown",
"id": "heavy-pound",
"metadata": {},
"source": [
"# Depths and Snow Pit Data Package Contents\n",
"\n",
"(12 minutes)\n",
"\n",
"Learning Objectives:\n",
"- Tools to access data.\n",
"- Code snippets to extract and prep data."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "permanent-psychology",
"metadata": {},
"outputs": [],
"source": [
"# standard imports\n",
"import os\n",
"from pathlib import Path\n",
"import glob\n",
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"#plotting imports\n",
"import matplotlib as mpl\n",
"import matplotlib.pyplot as plt\n",
"plt.style.use(['seaborn-notebook'])"
]
},
{
"cell_type": "markdown",
"id": "improving-squad",
"metadata": {},
"source": [
"## Access tutorial data from Zenodo\n",
"\n",
"We've archived datasets used for 2021 Hackweek tutorials on [Zenodo](https://zenodo.org), to ensure that these tutorials can be run in the future. The following code pulls data from the Zenodo 'record' and unzips it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "meaningful-devices",
"metadata": {},
"outputs": [],
"source": [
"%%bash \n",
"\n",
"# Retrieve a copy of data files used in this tutorial from Zenodo.org:\n",
"# Re-running this cell will not re-download things if they already exist\n",
"\n",
"mkdir -p /tmp/tutorial-data\n",
"cd /tmp/tutorial-data\n",
"wget -q -nc -O data.zip https://zenodo.org/record/5504396/files/core-datasets.zip\n",
"unzip -q -n data.zip\n",
"rm data.zip"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "metropolitan-summer",
"metadata": {},
"outputs": [],
"source": [
"TUTORIAL_DATA = '/tmp/tutorial-data/core-datasets'"
]
},
{
"cell_type": "markdown",
"id": "surrounded-somalia",
"metadata": {},
"source": [
"## Download snow depth data from NSIDC\n",
"From the SnowEx20 [Depth Probe Landing Page](https://nsidc.org/data/SNEX20_SD/versions/1), you can download data and access the [User's Guide](https://nsidc.org/sites/nsidc.org/files/SNEX20_SD-V001-UserGuide_1.pdf)\n",
"\n",
"\n",
"The Community Snow Depth Probe data package is a single CSV with over 36,000 geolocated snow depths! Three different instrument types were used to measure depths and are recorded in the Measurement Tool column."
]
},
{
"cell_type": "markdown",
"id": "instructional-joining",
"metadata": {},
"source": [
"## Programmatically download snow depth data from NSIDC"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "impaired-gibraltar",
"metadata": {},
"outputs": [],
"source": [
"#os.chmod('/home/jovyan/.netrc', 0o600) #only necessary on snowex hackweek jupyterhub"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "reflected-engagement",
"metadata": {},
"outputs": [],
"source": [
"#%run './scripts/nsidc-download_SNEX20_SD.001.py' \n",
"#print('Grand Mesa 2020 Snow Depth data download complete') "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "entertaining-stanley",
"metadata": {},
"outputs": [],
"source": [
"#path = Path('./data/depths/')\n",
"\n",
"#for filename in path.glob('*.csv'):\n",
"# print(filename.name)"
]
},
{
"cell_type": "markdown",
"id": "individual-windsor",
"metadata": {},
"source": [
"### Read the Depth File"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "human-helicopter",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(f'{TUTORIAL_DATA}/depths/SnowEx2020_SnowDepths_COGM_alldepths_v01.csv', sep=',', header=0, parse_dates=[[2,3]]) #parse the date[2] and time[3] columns such that they are read in as datetime dtypes\n",
" \n",
"print('file has been read, and is ready to use.')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bigger-headset",
"metadata": {},
"outputs": [],
"source": [
"# check data types for each column\n",
"df.dtypes"
]
},
{
"cell_type": "markdown",
"id": "biological-captain",
"metadata": {},
"source": [
"### Prep for Data Analysis"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "beginning-reproduction",
"metadata": {},
"outputs": [],
"source": [
"# rename some columns for ease further down\n",
"df.rename(columns = {\n",
" 'Measurement Tool (MP = Magnaprobe; M2 = Mesa 2; PR = Pit Ruler)':'Measurement Tool', \n",
" 'Date (yyyymmdd)_Time (hh:mm, local, MST)': \"Datetime\"},\n",
" inplace = True)\n",
"\n",
"# set up filter for IOP date range\n",
"start = pd.to_datetime('1/28/2020') #first day of GM IOP campaign\n",
"end = pd.to_datetime('2/12/2020') #last day of GM IOP campaign\n",
"\n",
"# filter the IOP date range\n",
"df = df[(df['Datetime'] >= start) & (df['Datetime'] <= end)]\n",
"\n",
"print('DataFrame shape is: ', df.shape)\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"id": "dominican-plaza",
"metadata": {},
"source": [
"#### Use .groupby() to sort the data set"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "sweet-municipality",
"metadata": {},
"outputs": [],
"source": [
"# group data by the measurement tool \n",
"gb = df.groupby('Measurement Tool', as_index=False).mean().round(1)\n",
"\n",
"# show mean snow depth from each tool\n",
"gb[['Measurement Tool', 'Depth (cm)']]"
]
},
{
"cell_type": "markdown",
"id": "successful-folder",
"metadata": {},
"source": [
"#### ***Your turn***"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "molecular-globe",
"metadata": {},
"outputs": [],
"source": [
"# group data by the snow pit ID\n",
"\n",
"\n",
"# show mean snow depth around each snow pit\n",
"\n",
"\n",
"#(hint: what is the pit id column called? If you're not sure you can use df.columns to see a list of column names. Consider using .head() to display only the first 5 returns)"
]
},
{
"cell_type": "markdown",
"id": "authentic-robertson",
"metadata": {},
"source": [
"#### Find depths associated with a certain measurement tool"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "wicked-break",
"metadata": {},
"outputs": [],
"source": [
"print('List of Measurement Tools: ', df['Measurement Tool'].unique())"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "apparent-guide",
"metadata": {},
"outputs": [],
"source": [
"r = df.loc[df['Measurement Tool'] == 'PR']\n",
"print('DataFrame shape is: ', r.shape)\n",
"r.head()"
]
},
{
"cell_type": "markdown",
"id": "accessible-organ",
"metadata": {},
"source": [
"#### ***Your turn***"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "proved-video",
"metadata": {},
"outputs": [],
"source": [
"# find all depths recorded by the Mesa2\n",
"\n",
"\n",
"# (hint: copy/paste is your friend)"
]
},
{
"cell_type": "markdown",
"id": "periodic-estate",
"metadata": {},
"source": [
"Let's make sure we all have the same pd.DataFrame() again"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "searching-reducing",
"metadata": {},
"outputs": [],
"source": [
"# pit ruler snow depths from Grand Mesa IOP\n",
"r = df.loc[df['Measurement Tool'] == 'PR'] \n",
"print( 'DataFrame is back to only pit ruler depths')"
]
},
{
"cell_type": "markdown",
"id": "broke-canyon",
"metadata": {},
"source": [
"### Plotting"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aquatic-academy",
"metadata": {},
"outputs": [],
"source": [
"# plot pit ruler depths \n",
"ax = r.plot(x='Easting', y='Northing', c='Depth (cm)', kind='scatter', alpha=0.7, colorbar=True, colormap='PuBu', legend=True)\n",
"ax.set_title('Grand Mesa Pit Ruler Depths')\n",
"ax.set_xlabel('Easting [m]')\n",
"ax.set_ylabel('Northing [m]')\n",
"plt.show()\n",
"\n",
"print('Notice the point on the far right - that is the \"GML\" or Grand Mesa Lodge pit where all instruments were deployed for a comparison study. pitID=GML')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "russian-plenty",
"metadata": {},
"outputs": [],
"source": [
"# plot histogram of pit ruler depths\n",
"ax = r['Depth (cm)'].plot.hist(bins=25)\n",
"ax.grid()\n",
"ax.set_title('Grand Mesa Pit Ruler Depths')\n",
"ax.set_xlabel('Snow Depth (cm)')\n",
"ax.set_ylabel('Frequency')"
]
},
{
"cell_type": "markdown",
"id": "friendly-harassment",
"metadata": {},
"source": [
"## Download snow pit data from NSIDC\n",
"From the SnowEx20 [Snow Pit Landing Page](https://nsidc.org/data/SNEX20_GM_SP/versions/1), you can download data and access the [User's Guide](https://nsidc.org/data/SNEX20_GM_SP/versions/1). \n",
"\n",
""
]
},
{
"cell_type": "markdown",
"id": "prescribed-bristol",
"metadata": {},
"source": [
"## Programmatically download snow pit data from NSIDC"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "descending-tender",
"metadata": {},
"outputs": [],
"source": [
"# load snow pit data\n",
"#%run 'scripts/nsidc-download_SNEX20_GM_SP.001.py'\n",
"#print('Grand Mesa 2020 Snow Pit data download complete')"
]
},
{
"cell_type": "markdown",
"id": "removable-ballet",
"metadata": {},
"source": [
"### Don't want to work with all the files? Method to filter files"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "charitable-madison",
"metadata": {},
"outputs": [],
"source": [
"# what files would you like to find?\n",
"parameter = 'temperature'\n",
"pitID = '5N19'\n",
"date = '20200128'\n",
"\n",
"path = '{}/pits/SnowEx20_SnowPits_GMIOP_{}_{}_{}_v01.csv'.format(TUTORIAL_DATA, date, pitID, parameter)"
]
},
{
"cell_type": "markdown",
"id": "inside-speed",
"metadata": {},
"source": [
"### Read the Pit Parameter File"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dressed-train",
"metadata": {},
"outputs": [],
"source": [
"t = pd.read_csv(path, header=7)\n",
"t"
]
},
{
"cell_type": "markdown",
"id": "fatty-accounting",
"metadata": {},
"source": [
"### Plotting"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "crude-charter",
"metadata": {},
"outputs": [],
"source": [
"# plot temperature\n",
"ax = t.plot(x='Temperature (deg C)',y='# Height (cm)', legend=False)\n",
"ax.set_aspect(0.4)\n",
"ax.grid()\n",
"ax.set_title('pit {}: Temperature Profile'.format(pitID))\n",
"ax.set_xlabel('Temperature (deg C)')\n",
"ax.set_ylabel('Snow Depth (cm)')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "higher-middle",
"metadata": {},
"outputs": [],
"source": [
"# grab a different pit parameter file\n",
"parameter = 'density'\n",
"path = '/tmp/tutorial-data/core-datasets/pits/SnowEx20_SnowPits_GMIOP_{}_{}_{}_v01.csv'.format(date, pitID, parameter)\n",
"d = pd.read_csv(path, header=7)\n",
"d"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "quality-manitoba",
"metadata": {},
"outputs": [],
"source": [
"# get the average density \n",
"d['Avg Density (kg/m3)'] = d[['Density A (kg/m3)', 'Density B (kg/m3)', 'Density C (kg/m3)']].mean(axis=1, skipna=True)\n",
"d"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "relevant-going",
"metadata": {},
"outputs": [],
"source": [
"def plot_density(ax, dataframe):\n",
" \n",
" '''\n",
" This function helps you plot density profiles from snow pits. Use it to iterate through \n",
" DataFrame rows and plot the density for each top and bottom segment.\n",
" \n",
" '''\n",
" \n",
" for index, row in dataframe.iterrows():\n",
" # plot blue bars to represent 10cm density intervals\n",
" top = row['# Top (cm)']\n",
" bottom = row['Bottom (cm)']\n",
" dens = row[\"Avg Density (kg/m3)\"]\n",
" ax.plot([dens, dens],[bottom, top], color='blue', linewidth=4)\n",
"\n",
" # plot a red cross to show the spread between A and B samples\n",
" densA = row[\"Density A (kg/m3)\"]\n",
" densB = row[\"Density B (kg/m3)\"]\n",
" middle = bottom + 5.\n",
" ax.plot([densA, densB],[middle,middle], color='red')\n",
" \n",
" return ax\n",
"\n",
"fig, ax = plt.subplots()\n",
"ax = plot_density(ax, d)\n",
"ax.set_xlim(50, 400)\n",
"ax.set_ylim(0, 140)\n",
"ax.grid(color='lightgray', alpha=.5)\n",
"ax.set_aspect(2)\n",
"ax.set_title('pit {}: Density Profile'.format(pitID))\n",
"ax.set_xlabel('Density kg/m3')\n",
"ax.set_ylabel('Snow Depth (cm)') "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}