Python Pandas - Interpolation of Missing Values
Interpolation is a powerful technique in Pandas that used for handling the missing values in a dataset. This technique estimates the missing values based on other data points of the dataset. Pandas provides the interpolate() method for both DataFrame and Series objects to fill in missing values using various interpolation methods.
In this tutorial, we will learn about the interpolate() methods in Pandas for filling the missing values in a time series data, numeric data, and more using the different interpolation methods.
Basic Interpolation
The Pandas interpolate() method of the both DataFrame and Series objects is used to fills the missing values using different Interpolation strategies. By default, Pandas automatically uses linear interpolation as the default method.
Example
Here is a basic example of calling the interpolate() method for filling the missing values.
import numpy as np
import pandas as pd
df = pd.DataFrame({"A": [1.1, np.nan, 3.5, np.nan, np.nan, np.nan, 6.2, 7.9],
"B": [0.25, np.nan, np.nan, 4.7, 10, 14.7, 1.3, 9.2],
})
print("Original DataFrame:")
print(df)
# Using the interpolate() method
result = df.interpolate()
print("\nResultant DataFrame after applying the interpolation:")
print(result)
Following is the output of the above code −
Original DataFrame:
| A | B | |
|---|---|---|
| 0 | 1.1 | 0.25 |
| 1 | NaN | NaN |
| 2 | 3.5 | NaN |
| 3 | NaN | 4.70 |
| 4 | NaN | 10.00 |
| 5 | NaN | 14.70 |
| 6 | 6.2 | 1.30 |
| 7 | 7.9 | 9.20 |
| A | B | |
|---|---|---|
| 0 | 1.100 | 0.250000 |
| 1 | 2.300 | 1.733333 |
| 2 | 3.500 | 3.216667 |
| 3 | 4.175 | 4.700000 |
| 4 | 4.850 | 10.000000 |
| 5 | 5.525 | 14.700000 |
| 6 | 6.200 | 1.300000 |
| 7 | 7.900 | 9.200000 |
Different Interpolating Methods
Pandas supports several interpolation methods, including linear, polynomial, pchip, akima, spline, and more. These methods provide flexibility for filling the missing values depending on the nature of your data.
Example
The following example demonstrates using the interpolate() method with the barycentric interpolation technique.
import numpy as np
import pandas as pd
df = pd.DataFrame({"A": [1.1, np.nan, 3.5, np.nan, np.nan, np.nan, 6.2, 7.9],
"B": [0.25, np.nan, np.nan, 4.7, 10, 14.7, 1.3, 9.2],
})
print("Original DataFrame:")
print(df)
# Applying the interpolate() with Barycentric method
result = df.interpolate(method='barycentric')
print("\nResultant DataFrame after applying the interpolation:")
print(result)
Following is the output of the above code −
Original DataFrame:
| i | A | B |
|---|---|---|
| 0 | 1.1 | 0.25 |
| 1 | NaN | NaN |
| 2 | 3.5 | NaN |
| 3 | NaN | 4.70 |
| 4 | NaN | 10.00 |
| 5 | NaN | 14.70 |
| 6 | 6.2 | 1.30 |
| 7 | 7.9 | 9.20 |
| A | B | |
|---|---|---|
| 0 | 1.100000 | 0.250000 |
| 1 | 2.596429 | 57.242857 |
| 2 | 3.500000 | 24.940476 |
| 3 | 4.061429 | 4.700000 |
| 4 | 4.531429 | 10.000000 |
| 5 | 5.160714 | 14.700000 |
| 6 | 6.200000 | 1.300000 |
| 7 | 7.900000 | 9.200000 |
Handling Limits in Interpolation
By default, Pandas interpolation fills all the missing values, but you can limit how many consecutive NaN values are filled using the limit parameter of the interpolate() method.
Example
The following example demonstrates filling the missing values of a Pandas DataFrame by limiting the consecutive fills using the limit parameter of the interpolate() method.
import numpy as np
import pandas as pd
df = pd.DataFrame({"A": [1.1, np.nan, 3.5, np.nan, np.nan, np.nan, 6.2, 7.9],
"B": [0.25, np.nan, np.nan, 4.7, 10, 14.7, 1.3, 9.2],
})
print("Original DataFrame:")
print(df)
# Applying the interpolate() with limit
result = df.interpolate(method='spline', order=2, limit=1)
print("\nResultant DataFrame after applying the interpolation:")
print(result)
Following is the output of the above code −
Original DataFrame:
| i | A | B |
|---|---|---|
| 0 | 1.1 | 0.25 |
| 1 | NaN | NaN |
| 2 | 3.5 | NaN |
| 3 | NaN | 4.70 |
| 4 | NaN | 10.00 |
| 5 | NaN | 14.70 |
| 6 | 6.2 | 1.30 |
| 7 | 7.9 | 9.20 |
| i | A | B |
|---|---|---|
| 0 | 1.100000 | 0.250000 |
| 1 | 2.231383 | -1.202052 |
| 2 | 3.500000 | NaN |
| 3 | 4.111529 | 4.700000 |
| 4 | NaN | 10.000000 |
| 5 | NaN | 14.700000 |
| 6 | 6.200000 | 1.300000 |
| 7 | 7.900000 | 9.200000 |
Interpolating Time Series Data
Interpolation can be applied to the Pandas time series data as well. It is useful when filling gaps in missing data points over time.
Example
Example statement −
import numpy as np
import pandas as pd
indx = pd.date_range("2024-01-01", periods=10, freq="D")
data = np.random.default_rng(2).integers(0, 10, 10).astype(np.float64)
s = pd.Series(data, index=indx)
s.iloc[[1, 2, 5, 6, 9]] = np.nan
print("Original Series:")
print(s)
result = s.interpolate(method="time")
print("\nResultant Time Series after applying the interpolation:")
print(result)
Following is the output of the above code −
Original Series:
| Date | Value |
|---|---|
| 2024-01-01 | 8.0 |
| 2024-01-02 | NaN |
| 2024-01-03 | NaN |
| 2024-01-04 | 2.0 |
| 2024-01-05 | 4.0 |
| 2024-01-06 | NaN |
| 2024-01-07 | NaN |
| 2024-01-08 | 0.0 |
| 2024-01-09 | 3.0 |
| 2024-01-10 | NaN |
| Date | Value |
|---|---|
| 2024-01-01 | 8.000000 |
| 2024-01-02 | 6.000000 |
| 2024-01-03 | 4.000000 |
| 2024-01-04 | 2.000000 |
| 2024-01-05 | 4.000000 |
| 2024-01-06 | 2.666667 |
| 2024-01-07 | 1.333333 |
| 2024-01-08 | 0.000000 |
| 2024-01-09 | 3.000000 |
| 2024-01-10 | 3.000000 |