{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0e7dd000",
   "metadata": {},
   "source": [
    "# Decomposing a time series into seasonal cycle, trend, and residuals\n",
    "\n",
    "Many time series contain a cyclical component, long-term trend, and short-term fluctuations. These components are often additive\n",
    "$$y(t) = T(t) + S(t) + R(t)$$\n",
    "where $y(t)$ is the observed data, $T(t)$ is the trend component, $S(t)$ is the seasonal or cyclical component, and $R(t)$ are the residuals containing short-term fluctuations. The additive situation will be illustrated below, but the components could also be multiplicative\n",
    "$$y(t) = T(t) \\, S(t) \\,  R(t)$$\n",
    "There are numerous methods for decomposing time series into these components. Selecting an appropriate method requires knowledge of the underlying data and goals of the analysis.\n",
    "\n",
    "### Parametric and non-parametric methods \n",
    "\n",
    "*Parametric* methods make assumptions about the functional form of the seasonal cycle or trend. For example, the trend may be modeled as a linear function or polynomial and the seasonal cycle may be modeled as a sinusoid or sum of sinusoids.\n",
    "\n",
    "*Non-parametric* methods make no assumption about the functional form of the trend or seasonal cycle. Non-parametric methods for the seasonal or diel cycle include binning data by calendar month or hour of the day, then computing the mean or median of each bin. This may be called a climatology. The trend may be fitted with a smoothing function, such as LOWESS (locally weighted least squares). Frequency filtering can also be used to separate  \n",
    "\n",
    "### Detrending and deseasonalizing\n",
    "\n",
    "Detrending an deseasonalizing mean removing one of the components from the time series. \n",
    "The detrended time series is\n",
    "$$y(t) - T(t) = S(t) + R(t)$$\n",
    "The deseasonalized time series is\n",
    "$$y(t) - S(t) = T(t) + R(t)$$\n",
    "\n",
    "### Sample data\n",
    "\n",
    "This notebook demonstrates some methods to identify and separate the seasonal cycle and trend in a time series. CO2 measurements at Mauna Loa and C2H6 measurements at Utqiagvik (Barrow), Alaska are used as examples.\n",
    "\n",
    "The CO2 time series exhibits long-term trend, clear seasonal cycle, and other variability. The upward trend has not been constant and the seasonal cycle has increased over decades.\n",
    "\n",
    "The C2H6 time series has a large seasonal cycle little trend over the years shown here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d705af46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 80052 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "import acgc.stats as mystats\n",
    "from statsmodels.tsa.seasonal import STL, seasonal_decompose\n",
    "\n",
    "# Load CO2 data\n",
    "co2 = sm.datasets.co2.load().data.co2\n",
    "co2_monthly = co2.resample('MS').mean()\n",
    "co2_monthly_fill = co2_monthly.ffill()\n",
    "\n",
    "# Load C2H6 data\n",
    "c2h6 = pd.read_csv('demo/c2h6_brw_surface-flask_1_arl_event.txt', \n",
    "                 comment='#', sep=r'\\s+',header=None,\n",
    "                 names=['site_id','year','month','day','hour','minute','second',\n",
    "                        'sample_id','sample_method','parameter_formula','analysis_group_abbr',\n",
    "                        'value','uncertainty','analysis_flag',\n",
    "                        'analysis_instrument','analysis_year','analysis_month','analysis_day','analysis_hour','analysis_minute','analysis_seconds',\n",
    "                        'latitude','longitude','altitude','event_number'],\n",
    "                 na_values=-999.99)\n",
    "c2h6['time'] = pd.to_datetime(c2h6[['year','month','day','hour','minute','second']])\n",
    "\n",
    "fig, (ax0,ax1) = plt.subplots(2,1, figsize=(6,5))\n",
    "co2.plot(ax=ax0, label = 'weekly', title='Mauna Loa CO2', ylabel='ppm')\n",
    "co2_monthly_fill.plot(ax=ax0, label = 'monthly (filled)')\n",
    "ax0.legend()\n",
    "\n",
    "c2h6.plot.scatter(x='time',y='value', ax=ax1,title='Utqiagvik C2H6', ylabel='ppb')\n",
    "fig.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ae71e7",
   "metadata": {},
   "source": [
    "### Time series without trend; binned averages\n",
    "\n",
    "For a steady time series with little or no trend, $T(t)$ is a constant equal to the mean fo the time series. The simplest way to determine the seasonal component is with binned averages (e.g. using groupby() ). For example, the average of all Januaries, average of all Februaries, etc. This is often called a climatology.\n",
    "\n",
    "This approach is non-parameteric for the cycle. This method does not require evenly spaced data and missing data are allowed.\n",
    "\n",
    "An alternate approach would be to fit the time series with sinusoids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6fe50dd7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0, 0.5, 'residuals')]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 79108 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trend = c2h6['value'].mean()\n",
    "seasonal = (c2h6['value'] - trend).groupby(c2h6.time.dt.month).mean()\n",
    "residual = c2h6['value'] - trend - seasonal[c2h6.time.dt.month.values].values\n",
    "\n",
    "fig, (ax0,ax1) = plt.subplots(2,1, figsize=(6,5))\n",
    "ax0.scatter('time', 'value', data=c2h6, label='data')\n",
    "ax0.plot(c2h6.time,trend + seasonal[c2h6.time.dt.month.values], \n",
    "         label='seasonal + mean',\n",
    "         color='C1')\n",
    "ax0.legend()\n",
    "\n",
    "ax1.scatter(c2h6.time, residual, label='residual')\n",
    "ax1.axhline(y=0, color='k', ls='--')\n",
    "ax1.set(ylabel='residuals')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16c6bb6c",
   "metadata": {},
   "source": [
    "### `seasonal_decompose`\n",
    "\n",
    "This method in statsmodels is non-parametric for the trend and cycle. It requires evenly spaced data with no missing values. A boxcar filter with length equal to the period is used to separate the trend component. The cyclical component is determined by the grouped mean of detrended data. For example, the average of all Januaries. The result is a fixed seasonal cycle. \n",
    "\n",
    "More sophisticated methods may be preferable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3f89bcce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 82580 characters]",
      "text/plain": [
       "<Figure size 640x480 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from statsmodels.tsa.seasonal import seasonal_decompose\n",
    "\n",
    "result_sd = seasonal_decompose(co2_monthly_fill, model='additive', period=12)\n",
    "fig = result_sd.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07a32a2f",
   "metadata": {},
   "source": [
    "### Seasonal-Trend decomposisiont with LOESS `STL`\n",
    "\n",
    "This method is non-parametric for trend and cycle components. It requires evenly spaced data without gaps. The cyclic component is not fixed in STL, but is allowed to vary slowly in this decomposition.\n",
    "\n",
    "`STL` has many additional parameters described in the statsmodels documentation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8ce3240e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 92492 characters]",
      "text/plain": [
       "<Figure size 640x480 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from statsmodels.tsa.seasonal import STL\n",
    "\n",
    "result = STL(co2_monthly_fill, period=12).fit()\n",
    "fig = result.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b47f3017",
   "metadata": {},
   "source": [
    "### STL with smoother seasonal cycle\n",
    "\n",
    "Seasonal cycles are often (but not always) expected to vary smoothly if they are sample at a frequency much higher than the seasonal cycle. For example, if the annual cycle of temperature is estimated from daily data, we expect the smooth variations across sequential days in the average cycle. STL can produce irregular seasonal cycles under these conditions, because the cycle value on Jan 1 is constructed separately from the cycle value for Jan 2, etc. `acgc.stats` provides 2 functions to address this limitation. \n",
    "\n",
    "### `STL_smoother`\n",
    "\n",
    "`STL_smoother` applies additional smoothing to the seasonal cycle produced by STL.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7cb08eca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 94340 characters]",
      "text/plain": [
       "<Figure size 640x480 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import acgc.stats as mystats\n",
    "\n",
    "result = mystats.STL_smoother(co2_monthly_fill, period=12, smooth=4)\n",
    "fig = result.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "860862ff",
   "metadata": {},
   "source": [
    "### `STL_harmonic`\n",
    "\n",
    "`STL_harmonic` is a variation on the STL approach. The trend is fitted with LOWESS like STL, while the seasonal cycle is constructed from a sinusoid and its harmonics. This results in a smooth seasonal cycle, unless the number of harmonics is very hight. The resulting seasonal cycle is fixed, unlike STL. This method is non-parametric for the trend, but parametric for the cyclic component. Irregularly spaced data are allowed, but missing data are not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "460da50d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 86080 characters]",
      "text/plain": [
       "<Figure size 640x480 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import acgc.stats as mystats\n",
    "\n",
    "# Several ways to call STL_harmonic are shown. All produce the sane result.\n",
    "\n",
    "# Passing y and x explicitly.\n",
    "# Since x is a datetime index, the period should be a timedelta\n",
    "result = mystats.STL_harmonic(co2_monthly.dropna(), co2_monthly.dropna().index,\n",
    "                              period=pd.Timedelta(days=365.25))\n",
    "\n",
    "# Passing x implicitly; STL_harmonic uses the index of a pandas Series if x is not provided\n",
    "result = mystats.STL_harmonic(co2_monthly.dropna(), \n",
    "                              period=pd.Timedelta(days=365.25))\n",
    "\n",
    "# Resetting the index to simple integer range (values will be treated as equally spaced, slightly less accurate than above)\n",
    "# If x is not datetime-like, the period should be an integer or float\n",
    "result = mystats.STL_harmonic(co2_monthly.reset_index(drop=True).dropna(), \n",
    "                              period=12)\n",
    "\n",
    "\n",
    "fig = result.plot()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "main",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
