Anomaly Detection in Time Series Data Python: A Starter Guide

If you're curious about spotting outliers or unusual patterns in time series data using Python, you've come to the right place. This guide simplifies the complex task of anomaly detection into manageable steps, suitable even for beginners.

If this turns out too complex and time consuming for you- you might want to look into Eyer or alternative out-of-the-box solutions that might be a good fit instead of building on your own.

Here's a quick rundown of what we'll cover:

Why Anomaly Detection Matters: It's about finding data points that stand out because they could indicate significant events, like fraud or system failures.
Tools for the Job: We'll use Python libraries such as Pandas for data manipulation, Matplotlib for visualization, and Scikit-Learn for machine learning.
Methods Explained: From simple statistical techniques like the Z-score to advanced machine learning models like Isolation Forest and Local Outlier Factor.
Practical Example: A step-by-step walkthrough of applying these concepts to real-world data, including preparing your environment, choosing and implementing methods, and evaluating your model's performance.
Best Practices: Tips on choosing algorithms, tuning models, and reducing false positives and negatives to improve your anomaly detection efforts.

This comprehensive guide aims to equip you with the knowledge to start identifying anomalies in time series data using Python, enhancing your data analysis and prediction capabilities.

Understanding Anomaly Detection

Anomaly detection is all about spotting the odd ones out in your data. Think of it as finding the piece of a puzzle that doesn't quite fit. In time series data, which is data collected over time, anomalies can show up in a few ways:

Point anomalies - This is when one data point sticks out from the rest. Imagine you're tracking how many ice creams you sell every day, and suddenly, on one random Tuesday, you sell 100 times more than usual.

Contextual anomalies - These are odd data points that only seem strange when you consider the time they happened. For example, selling lots of ice creams in the middle of winter might be unusual.

Collective anomalies - Imagine a situation where, over a few weeks, your ice cream sales slowly go down. No single day is weird on its own, but the trend over time is unusual.

To find these odd data points, you can use different methods:

Statistical techniques like looking at how far away a point is from the average. This is good for spotting those one-off weird days (point anomalies).
Machine learning models like Isolation Forest and Local Outlier Factor can learn what normal looks like and then spot when something doesn't fit that pattern. This is great for catching oddities based on the context (contextual anomalies).
Deep learning models like autoencoders are a bit more complex. They try to summarize the data and then rebuild it. If they can't rebuild a piece of data well, it might be because it's an anomaly. This is useful for spotting when a bunch of data points together are strange (collective anomalies).

When you're checking how well you're doing at spotting these anomalies, you'll look at things like precision, recall, and F1-scores. It's also important to keep testing your methods on new data to make sure they really work and aren't just memorizing the data you trained them on. Here is a summary of Eyer’s last F1 test.

In short, anomaly detection is about finding the unusual patterns in your data over time. Python has lots of tools to help with this, from simple stats to fancy machine learning and deep learning models.

Prerequisites

Before you jump into spotting weird patterns in time data with Python, here's what you need to know:

Python Programming Basics

Know how to work with lists and dictionaries in Python
Be comfortable with if/else statements and for loops
Know how to add Python libraries to your project
Be able to create functions in Python

It's important to have these Python basics down because we'll be using them a lot. If you're a bit rusty, there are plenty of online tutorials to catch you up.

Key Python Libraries

We'll be using some special tools in Python that help us handle and analyze data:

Pandas: Great for organizing and looking at data
NumPy: Helps with doing math stuff
Matplotlib: Lets you make graphs to see what your data looks like
Scikit-Learn: Has tools for machine learning

Pandas is especially important for us because it lets us work with time data in a neat way.

Time Series Data

Having messed around with time series data before will help. This includes understanding:

How data can go up and down over time
The difference between data that changes predictably and data that doesn't
The basics of how time data is put together
How to check your work with time data

You don't have to be an expert, but knowing a bit about time series will make things easier.

Development Environment

Make sure you have Python 3 and a place to write and run your code, like Jupyter notebooks. It's also a smart move to use a virtual environment to keep your project organized.

With these basics in hand, you're ready to start finding the odd bits in time series data with Python! We'll go over any special stuff you need to know as we move along.

Setting Up the Environment

Before diving into finding weird patterns in your time series data with Python, you need to get your computer ready. Here's a simple guide on how to do it:

Install Python

First, if Python isn't on your computer yet, go to python.org and download the latest Python 3. Avoid Python 2 since it's old and not used anymore.

Set Up a Virtual Environment

It's smart to keep all the tools and libraries for your project in one neat package. This is called a virtual environment. Here's how to set it up:

Open a terminal and go to your project's folder with cd.
Create the virtual environment by typing python3 -m venv env.
Activate it with source env/bin/activate on Linux/macOS or env\Scripts\activate on Windows.

Install Python Libraries

With your environment ready, it's time to add the libraries we'll need:

pip install pandas numpy matplotlib scikit-learn

This command installs Pandas, NumPy, Matplotlib, and Scikit-Learn. These are the basic tools for data analysis, anomaly detection, and making graphs.

Get an IDE (Optional)

Writing Python is easier with an IDE, which is like a supercharged text editor. It has helpful features like spotting errors and auto-completing your code. Some good ones include:

Visual Studio Code
PyCharm
Spyder (comes with Anaconda)

Pick the one you like. Following these steps gets your computer ready for spotting the unusual in time series data with Python.

Exploring Time Series Data

Time series data is basically data points collected over time. To start understanding what's going on in your data, you need to get comfy with it first. Pandas and Matplotlib are two Python tools that are super helpful for this.

Loading Time Series Data with Pandas

Pandas is awesome for dealing with time series data. It puts your data in a neat table that makes it easy to work with. Here's how to get started:

Import Pandas and load your data into a DataFrame:

import pandas as pd

df = pd.read_csv('daily_icecream_sales.csv', parse_dates=['date'])

Take a quick look at your data:

print(df.head()) # Shows the first 5 rows
print(df.info()) # Gives a summary

Pick out specific dates or columns of data:

june_data = df['2020-06':'2020-06-30'] # Picks dates

revenue = df['revenue'] # Grabs a column

Pandas has a lot of cool features for time series data, so it's worth checking out their docs to learn more.

Visualizing Time Series Data

Graphs help you see what's happening in your data over time. Matplotlib lets you make all sorts of plots:

import matplotlib.pyplot as plt

plt.plot(df['date'], df['sales'])
plt.xlabel('Date')
plt.ylabel('Ice Cream Sales') 
plt.title('Daily Sales Over Time')
plt.show()

This code makes a simple line plot. You can also make other types of charts like scatter plots, histograms, and heatmaps to see your data in different ways.

Visuals can show you trends, seasonal changes, and help spot anomalies.

Preprocessing Data

Real-life data can be messy. You'll need to clean it up before analyzing it:

Handle missing values - Fill in gaps, drop missing parts, or guess missing values
Smooth out noise - Get rid of small, random changes to see the main trends
Change data scales - Adjust data so everything is on a similar scale
Add useful flags - Create new columns to mark special info (like holidays)

Cleaning your data helps your analysis be more accurate. There are lots of Python tools for this, including Pandas and NumPy.

By following these steps, you're ready to start looking for anomalies. You know how to pick up time series data, check it out, clean it, and prep it for deeper analysis. Now, let's dive into finding those anomalies!

Anomaly Detection Techniques

Finding odd bits in your time series data means spotting data points that don't quite match the rest. There are many ways to do this, using both simple stats and smarter machine learning methods. Let's look at some common ways people try to spot these outliers.

Statistical Methods

These methods use basic math to figure out which data points are way different from what's expected.

Mean Absolute Deviation (MAD)

MAD looks at how far away each piece of data is from the middle value (median). If a data point is way off from this middle value, it might be something unusual.

Z-Score

Z-score tells us how far a data point is from the average, measured in standard deviations. If a data point's z-score is more than 3, it's usually considered weird.

Advantages:

Quick and easy to do
Doesn't need a lot of computer power
Straightforward to understand

Disadvantages:

Can be thrown off by very unusual data points
Works best if data doesn't change much over time and is normally spread out

Machine Learning Models

Machine learning methods learn what normal looks like from your data and then spot the data points that don't fit this pattern.

Isolation Forest

Isolation forest finds outliers by splitting the data into smaller bits until it isolates the odd ones out. The weird data points are easier to isolate because they're not like the others.

Local Outlier Factor (LOF)

LOF finds odd data points by looking at how crowded an area is. If a data point is in a much less crowded area compared to its neighbors, it's likely an outlier.

Advantages:

Good at dealing with complicated patterns
Very accurate in finding groups of strange data points
Doesn't assume your data is spread out in a certain way

Disadvantages:

Can take a lot of computer power to run
Might fit too closely to the small details of your data

Picking the right method to find odd data points depends on what you're looking for, how your data behaves, and how easy you want the process to be.

Implementing Statistical Methods

Statistical methods like MAD and Z-score are pretty simple ways to find odd data points in your time series data using just basic math. These techniques help us spot those data points that just don't fit in. Let's go through how you can use these methods step by step:

Mean Absolute Deviation

MAD helps us find anomalies by checking how far away data points are from the median (the middle value). Here's what you do:

First, you need to load your time series data into a table using Pandas.
Find the median value of your data.
Subtract this median from each data point to see how far each one is from the median.
Calculate the MAD by finding the average of these distances.
Decide on a cut-off point for what you'll consider weird, like data points that are 3 times the MAD.
Look at each data point. If it's above your cut-off, it might be an anomaly.

Here's a quick example:

import pandas as pd
import numpy as np

series = pd.Series([1, 34, 5, 67, 88, 32, 109]) 

median = series.median()
absolute_deviations = np.abs(series - median)
mad = absolute_deviations.mean()
threshold = 3 * mad
outliers = series[series > threshold]

print(outliers)


This tells us that data points above 3 * MAD are considered anomalies.

### Z-score Method

Z-score shows us how far a data point is from the average, in terms of standard deviations. Here's how to do it:

1. Load your data into Pandas.
2. Calculate the average and standard deviation of your data.
3. Figure out the z-scores for each data point.
4. Pick a threshold, like 3 standard deviations.
5. Data points above this threshold are your anomalies.

And here's an example:

```python
import pandas as pd
import numpy as np

series = pd.Series(...)

mean = series.mean()  
std_dev = series.std()

z_scores = (series - mean)/std_dev  

threshold = 3
outliers = series[z_scores > threshold]  

print(outliers)

Data points with a z-score above 3 are considered outliers.

Starting with statistical methods like these is a good first step in anomaly detection. After getting the hang of these, you can move on to more complex machine learning techniques.

Implementing Machine Learning Methods

Isolation Forest

Isolation Forest is a smart way to find data that doesn't fit in, without needing to know what 'normal' looks like first. Let's walk through how to use it with Python:

Import libraries

from sklearn.ensemble import IsolationForest
import pandas as pd

Load dataset

df = pd.read_csv('data.csv')
X = df[['feature1', 'feature2']]

Define model

model = IsolationForest(n_estimators=100, contamination=0.1)

Fit model

model.fit(X)

Get anomalies

y_pred = model.predict(X)
X[y_pred == -1] # anomalies

Pros and cons of Isolation Forest:

Pros	Cons
Good for data with lots of features	Can be picky about settings
Doesn't use much memory	Might not do well with data that has natural groups
Quick	Works best when features don't depend on each other

Isolation Forest is great for spotting the odd ones out, especially if your data is pretty straightforward. Adjusting settings like how many 'trees' it uses (n_estimators) and how much weirdness you expect (contamination) can help it do a better job.

Local Outlier Factor

Local Outlier Factor (LOF) finds odd data by looking at how crowded an area is. Here's how to use LOF in Python:

Import libraries

from sklearn.neighbors import LocalOutlierFactor
import pandas as pd

Load data

df = pd.read_csv('data.csv')
X = df[['feature1', 'feature2']]

Define model

model = Local Outlier Factor()

Fit model

model.fit(X)

Get outliers

y_pred = model.fit_predict(X) 
X[y_pred == -1] # anomalies

Pros and cons of Local Outlier Factor:

Pros	Cons
No need to know data distribution upfront	Settings need careful tuning
Handles data with many features	Can miss anomalies if they're in groups
Good at finding dense spots	Struggles with overlapping groups

LOF is useful when your data forms natural clusters. Adjusting the settings like the number of neighbors (n_neighbors) and the size of the 'leaves' (leaf_size) can improve how well it works. Combining LOF with broader methods like Isolation Forest can help catch more anomalies.

Real-World Example

In this section, we're going to walk through a step-by-step example of finding odd or unusual patterns in a real-world time series dataset. We'll cover:

How to load and take a first look at the data
Getting the data ready for analysis
Choosing and applying a model to spot anomalies
Checking how well our model did
Making a chart to show where the anomalies are

The Dataset

We're going to use a dataset called artificialNoAnomaly from Amazon SageMaker, which is full of real-world time series data but doesn't have any anomalies marked. Let's load it up and see what we've got:

import pandas as pd

df = pd.read_csv('artificialNoAnomaly.csv', parse_dates=['timestamp'])
print(df.head())
print(df.shape)

This dataset has 60,000 rows, and each row is a measurement taken every minute. Since there are no anomalies pointed out, we'll add some fake ones later.

Let's draw a graph of the time series to understand the data better:

import matplotlib.pyplot as plt

plt.plot(df['timestamp'], df['value'])
plt.xlabel('Timestamp')
plt.ylabel('Value')
plt.title('Artificial Time Series')
plt.show()

The graph shows that the time series is pretty steady without any clear anomalies.

Preprocessing

Before we try to find anomalies, it's a good idea to get the data ready:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df['value'] = scaler.fit_transform(df['value'].values.reshape(-1,1))

Here, we made sure the data has a standard scale, which helps the model spot anomalies easier.

Introducing Anomalies

Since this dataset doesn't have any real anomalies, let's add some made-up ones:

anomaly_indices = [1000, 2500, 5000]
anomaly_values = [10, -10, 8]

for i, value in zip(anomaly_indices, anomaly_values):
    df.loc[i, 'value'] = value

We've added anomalies at different points in time.

Modeling

Now, let's use an isolation forest model to find anomalies:

from sklearn.ensemble import IsolationForest

model = IsolationForest(n_estimators=100, contamination=0.05)
model.fit(df[['value']])

The 'contamination' setting tells the model how many anomalies we're expecting.

Evaluation

To see how well our model did, let's check if it found the fake anomalies we added:

anomalies = model.predict(df[['value']])
df['anomaly'] = [True if x == -1 else False for x in anomalies]

for index in anomaly_indices:
    print(f\"Index {index} anomaly: {df.loc[index,'anomaly']}\")

It looks like the model did a great job and found all the anomalies we added!

Visualization

Lastly, let's make a graph to show where the anomalies are:

import seaborn as sns

plt.figure(figsize=(10, 6))
sns.lineplot(data=df, x='timestamp', y='value')
sns.scatterplot(data=df[df['anomaly']==True], x='timestamp', y='value', color='red')

plt.xlabel('Timestamp')
plt.ylabel('Value')
plt.title('Detected Anomalies')
plt.show()

This graph uses red dots to mark the anomalies, making them easy to spot.

And that's how you do it - from start to finish, we've just walked through an example of finding anomalies in a real-world time series dataset using Python. We've covered everything from loading the data, getting it ready, picking a model, checking its performance, to finally visualizing the anomalies. This can be a handy guide for your own projects.

Best Practices and Tips

Finding weird patterns in your time series data can be a bit of a puzzle. Here are some tips and tricks to help you pick the right tools and make your models work better:

Choosing the Best Algorithm

Begin with easy methods like z-scores and MAD before trying more complicated ones.
Use Isolation Forest for data with lots of different features.
Local Outlier Factor is good when your data forms groups.
Autoencoders are best for spotting unusual patterns that are hidden in complex data.

Tuning Hyperparameters

Adjusting your model's settings can take some experimenting:

For Isolation Forest, play around with n_estimators and contamination.
With Local Outlier Factor, try changing n_neighbors and leaf_size.
For autoencoders, experiment with the number of layers, nodes in each layer, and how the nodes are activated.

Reducing False Positives and Negatives

To lower the chances of wrongly labeling data as normal or weird:

Double-check your work with cross-validation.
Look at your model's decisions with graphs to spot mistakes.
Combine models to get better results.
Use the F1 score to find a good balance between precision and recall.

Handling Imbalanced Data

If finding anomalies is like looking for a needle in a haystack:

Try making the amount of normal and weird data more even.
Consider models that are okay with uneven data.
Combining different models can also help.

The main idea is to choose the right method for your data, adjust it as needed, and always double-check your findings. Following these steps can help you build a trustworthy system for spotting anomalies.

Eyer - an out-of-the-box alternative

Eyer’s AIOps / AI powered observability platform is essentially a plug-n-play anomaly detection platform for time series data. For some use case it might be easier to use Eyer or an alternative in the market if your use case fits with the requirements of Eyer - in summary:

You have data that you can stream to Eyer’s API
You can use open source metrics agents like Influx Telegraf to capture the data and feed to Eyer
You do not need to / and cannot tune any algorithms - its all out of the box. More about the algorithms which also includes a correlation engine
You can use Grafana or other dashboarding tools to visualize the data / anomalies and insight from Eyer

Conclusion

Finding unusual patterns in time series data is super important. It helps us catch problems or even spot good chances. This guide has shown that Python is a great tool for this job. It has many ways to help us, from simple math tricks to fancy learning methods.

Here are the main things to keep in mind:

It's key to first understand your time series data. You can do this by making graphs and using stats.
Simple methods like Z-scores and MAD can spot weird stuff for some data sets.
For trickier data, techniques like isolation forest, LOF, and autoencoders are better.
Making sure your models and checks are set up right helps avoid mistakes.
Dealing with data that doesn't match up evenly can make your models work better.
There are ready made tools in the market - like Eyer - that can be an easier solution for many use cases

The tips we've talked about are a good place to start for checking time series data for odd patterns. But every set of data is different. Trying these methods on new data will help you get better.

Using more than one method, like starting with simple checks and then using learning models, can make finding odd patterns more accurate. And for really important tasks, using a mix of models and having people check the results might be a good idea.

Like any careful work, checking data for odd patterns takes time and effort. But finding those key unusual points can be really valuable. It could help spot credit card fraud, predict when something might break, or find new chances for growth.

Additional Resources

If you're looking to dive deeper into finding unusual patterns in time series data using Python, here are some helpful places to start:

A Complete Guide to Time Series Anomaly Detection in Python - This resource breaks down different ways to spot anomalies using stats, machine learning, and deep learning, complete with code snippets to try out.
Understanding Anomaly Detection Algorithms for Time Series Data - A deep dive into the most used algorithms for spotting odd patterns in time data.
Anomaly Detection Made Easy - A great starting point for beginners, explaining the basics of spotting unusual data points in a way that's easy to grasp.
Collection of Anomaly Detection Tools for Time Series - A handy list of free tools and code for finding anomalies in time series data.
PyOD Library Guide - Offers a look at various models for anomaly detection available in the PyOD library.
Alibi-Detect Overview - Explains how Alibi-Detect can help find anomalies in your data.

These resources are packed with examples, tips, and detailed guides that can help you get better at spotting anomalies in time series data using Python.