This page was generated from examples/od_prophet_weather.ipynb.

# Time-series outlier detection using Prophet on weather data¶

## Method¶

The Prophet outlier detector uses the Prophet time series forecasting package explained in this excellent paper. The underlying Prophet model is a decomposable univariate time series model combining trend, seasonality and holiday effects. The model forecast also includes an uncertainty interval around the estimated trend component using the MAP estimate of the extrapolated model. Alternatively, full Bayesian inference can be done at the expense of increased compute. The upper and lower values of the uncertainty interval can then be used as outlier thresholds for each point in time. First, the distance from the observed value to the nearest uncertainty boundary (upper or lower) is computed. If the observation is within the boundaries, the outlier score equals the negative distance. As a result, the outlier score is the lowest when the observation equals the model prediction. If the observation is outside of the boundaries, the score equals the distance measure and the observation is flagged as an outlier. One of the main drawbacks of the method however is that you need to refit the model as new data comes in. This is undesirable for applications with high throughput and real-time detection.

## Dataset¶

The example uses a weather time series dataset recorded by the Max-Planck-Institute for Biogeochemistry. The dataset contains 14 different features such as air temperature, atmospheric pressure, and humidity. These were collected every 10 minutes, beginning in 2003. Like the TensorFlow time-series tutorial, we only use data collected between 2009 and 2016.

[1]:

import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import tensorflow as tf

from alibi_detect.od import OutlierProphet
from alibi_detect.utils.fetching import fetch_detector
from alibi_detect.utils.saving import save_detector, load_detector


## Load dataset¶

[2]:

zip_path = tf.keras.utils.get_file(
origin='https://storage.googleapis.com/tensorflow/tf-keras-datasets/jena_climate_2009_2016.csv.zip',
fname='jena_climate_2009_2016.csv.zip',
extract=True
)
csv_path, _ = os.path.splitext(zip_path)
df = pd.read_csv(csv_path)
df['Date Time'] = pd.to_datetime(df['Date Time'], format='%d.%m.%Y %H:%M:%S')
print(df.shape)
df.head()

(420551, 15)

[2]:

Date Time p (mbar) T (degC) Tpot (K) Tdew (degC) rh (%) VPmax (mbar) VPact (mbar) VPdef (mbar) sh (g/kg) H2OC (mmol/mol) rho (g/m**3) wv (m/s) max. wv (m/s) wd (deg)
0 2009-01-01 00:10:00 996.52 -8.02 265.40 -8.90 93.3 3.33 3.11 0.22 1.94 3.12 1307.75 1.03 1.75 152.3
1 2009-01-01 00:20:00 996.57 -8.41 265.01 -9.28 93.4 3.23 3.02 0.21 1.89 3.03 1309.80 0.72 1.50 136.1
2 2009-01-01 00:30:00 996.53 -8.51 264.91 -9.31 93.9 3.21 3.01 0.20 1.88 3.02 1310.24 0.19 0.63 171.6
3 2009-01-01 00:40:00 996.51 -8.31 265.12 -9.07 94.2 3.26 3.07 0.19 1.92 3.08 1309.19 0.34 0.50 198.0
4 2009-01-01 00:50:00 996.51 -8.27 265.15 -9.04 94.1 3.27 3.08 0.19 1.92 3.09 1309.00 0.32 0.63 214.3

Select subset to test Prophet model on:

[3]:

n_prophet = 10000


Prophet model expects a DataFrame with 2 columns: one named ds with the timestamps and one named y with the time series to be evaluated. We will just look at the temperature data:

[4]:

d = {'ds': df['Date Time'][:n_prophet], 'y': df['T (degC)'][:n_prophet]}
df_T = pd.DataFrame(data=d)
print(df_T.shape)
df_T.head()

(10000, 2)

[4]:

ds y
0 2009-01-01 00:10:00 -8.02
1 2009-01-01 00:20:00 -8.41
2 2009-01-01 00:30:00 -8.51
3 2009-01-01 00:40:00 -8.31
4 2009-01-01 00:50:00 -8.27
[5]:

plt.plot(df_T['ds'], df_T['y'])
plt.title('T (in °C) over time')
plt.xlabel('Time')
plt.ylabel('T (in °C)')
plt.show()


## Load or define outlier detector¶

The pretrained outlier and adversarial detectors used in the example notebooks can be found here. You can use the built-in fetch_detector function which saves the pre-trained models in a local directory filepath and loads the detector. Alternatively, you can train a detector from scratch:

[6]:

load_outlier_detector = False

[7]:

filepath = 'my_path'  # change to directory where model is downloaded
if load_outlier_detector:  # load pretrained outlier detector
detector_type = 'outlier'
dataset = 'weather'
detector_name = 'OutlierProphet'
od = fetch_detector(filepath, detector_type, dataset, detector_name)
filepath = os.path.join(filepath, detector_name)
else:  # initialize, fit and save outlier detector
od = OutlierProphet(threshold=.9)
od.fit(df_T)
save_detector(od, filepath)

INFO:fbprophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.


Please check out the documentation as well as the original Prophet documentation on how to customize the Prophet-based outlier detector and add seasonalities, holidays, opt for a saturating logistic growth model or apply parameter regularization.

## Predict outliers on test data¶

Define the test data. It is important that the timestamps of the test data follow the training data. We check this below by comparing the first few rows of the test DataFrame with the last few of the training DataFrame:

[8]:

n_periods = 1000
d = {'ds': df['Date Time'][n_prophet:n_prophet+n_periods],
'y': df['T (degC)'][n_prophet:n_prophet+n_periods]}
df_T_test = pd.DataFrame(data=d)
df_T_test.head()

[8]:

ds y
10000 2009-03-11 10:50:00 4.12
10001 2009-03-11 11:00:00 4.62
10002 2009-03-11 11:10:00 4.29
10003 2009-03-11 11:20:00 3.95
10004 2009-03-11 11:30:00 3.96
[9]:

df_T.tail()

[9]:

ds y
9995 2009-03-11 10:00:00 2.69
9996 2009-03-11 10:10:00 2.98
9997 2009-03-11 10:20:00 3.66
9998 2009-03-11 10:30:00 4.21
9999 2009-03-11 10:40:00 4.19

Predict outliers on test data:

[10]:

od_preds = od.predict(
df_T_test,
return_instance_score=True,
return_forecast=True
)


## Visualize results¶

We can first visualize our predictions with Prophet’s built in plotting functionality. This also allows us to include historical predictions:

[11]:

future = od.model.make_future_dataframe(periods=n_periods, freq='10T', include_history=True)
forecast = od.model.predict(future)
fig = od.model.plot(forecast)


We can also plot the breakdown of the different components in the forecast. Since we did not do full Bayesian inference with mcmc_samples, the uncertaintly intervals of the forecast are determined by the MAP estimate of the extrapolated trend.

[12]:

fig = od.model.plot_components(forecast)


It is clear that the further we predict in the future, the wider the uncertainty intervals which determine the outlier threshold.

Let’s overlay the actual data with the upper and lower outlier thresholds predictions and check where we predicted outliers:

[13]:

forecast['y'] = df['T (degC)'][:n_prophet+n_periods]

[14]:

pd.plotting.register_matplotlib_converters()  # needed to plot timestamps
forecast[-n_periods:].plot(x='ds', y=['y', 'yhat', 'yhat_upper', 'yhat_lower'])
plt.title('Predicted T (in °C) over time')
plt.xlabel('Time')
plt.ylabel('T (in °C)')
plt.show()


Outlier scores and predictions:

[15]:

od_preds['data']['forecast']['threshold'] = np.zeros(n_periods)
od_preds['data']['forecast'][-n_periods:].plot(x='ds', y=['score', 'threshold'])
plt.title('Outlier score over time')
plt.xlabel('Time')
plt.ylabel('Outlier score')
plt.show()


The outlier scores naturally trend down as uncertainty increases when we predict further in the future.

Let’s look at some individual outliers:

[16]:

df_fcst = od_preds['data']['forecast']
df_outlier = df_fcst.loc[df_fcst['score'] > 0]

[17]:

print('Number of outliers: {}'.format(df_outlier.shape[0]))
df_outlier[['ds', 'yhat', 'yhat_lower', 'yhat_upper', 'y']]

Number of outliers: 42

[17]:

ds yhat yhat_lower yhat_upper y
166 2009-03-12 14:30:00 5.826341 2.359631 9.523064 2.32
279 2009-03-13 09:20:00 2.438507 -1.834116 6.408030 6.60
280 2009-03-13 09:30:00 2.546610 -1.484339 6.560092 7.22
281 2009-03-13 09:40:00 2.661307 -1.238431 6.712489 7.11
282 2009-03-13 09:50:00 2.782348 -1.284963 6.631783 7.22
283 2009-03-13 10:00:00 2.909426 -1.360733 7.213170 7.50
284 2009-03-13 10:10:00 3.042175 -0.938601 7.138561 7.71
285 2009-03-13 10:20:00 3.180168 -0.542519 7.159230 7.93
286 2009-03-13 10:30:00 3.322914 -0.749686 7.319972 7.98
287 2009-03-13 10:40:00 3.469862 -0.639899 7.248605 7.97
288 2009-03-13 10:50:00 3.620397 -0.536662 7.690310 8.11
289 2009-03-13 11:00:00 3.773845 -0.407588 7.735500 8.31
290 2009-03-13 11:10:00 3.929473 -0.242533 8.195561 8.22
291 2009-03-13 11:20:00 4.086493 0.070751 8.352415 8.47
292 2009-03-13 11:30:00 4.244068 -0.046052 8.216369 8.65
293 2009-03-13 11:40:00 4.401316 0.398563 8.058718 8.73
294 2009-03-13 11:50:00 4.557318 0.532095 8.671582 8.86
295 2009-03-13 12:00:00 4.711127 0.657084 8.844659 9.03
296 2009-03-13 12:10:00 4.861773 0.861597 8.995459 9.20
297 2009-03-13 12:20:00 5.008280 0.987469 9.190047 9.27
310 2009-03-13 14:30:00 6.146584 2.098111 10.394298 10.43
314 2009-03-13 15:10:00 6.108253 1.736977 10.220202 10.24
315 2009-03-13 15:20:00 6.070174 2.139051 10.002806 10.02
316 2009-03-13 15:30:00 6.021822 1.536603 10.264883 10.40
317 2009-03-13 15:40:00 5.963937 1.977076 9.995948 10.32
320 2009-03-13 16:10:00 5.741674 1.486856 10.023755 10.07
324 2009-03-13 16:50:00 5.368270 1.421167 9.341373 9.52
435 2009-03-14 11:20:00 4.284622 -0.291427 8.510237 8.77
437 2009-03-14 11:40:00 4.597406 -0.229902 9.058397 9.11
439 2009-03-14 12:00:00 4.904994 0.715068 9.270239 9.51
440 2009-03-14 12:10:00 5.054459 0.779145 9.515633 9.71
442 2009-03-14 12:30:00 5.339857 0.972692 9.709758 9.76
469 2009-03-14 17:00:00 5.406697 0.555661 9.653830 9.66
472 2009-03-14 17:30:00 5.096989 0.472095 9.414530 9.57
473 2009-03-14 17:40:00 4.996114 0.478616 9.479434 9.49
474 2009-03-14 17:50:00 4.897644 0.509235 9.347031 9.42
516 2009-03-15 00:50:00 2.658244 -2.097107 7.294615 7.56
517 2009-03-15 01:00:00 2.619797 -2.156021 7.082319 7.53
519 2009-03-15 01:20:00 2.547890 -1.977093 7.401913 7.42
521 2009-03-15 01:40:00 2.482139 -2.358331 7.093606 7.36
522 2009-03-15 01:50:00 2.451274 -2.391906 7.132587 7.35
523 2009-03-15 02:00:00 2.421542 -2.561413 7.114390 7.28