vif

In this code, VIF will be used to remove features that can be represented via linear combination of other features. Importing libraries.

In [2]:

import numpy as np
import polars as pl
import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

Generating data, and creating a linear combination of features with extra noise.

In [4]:

n_samples = 100

x1 = np.random.normal(2, 1, n_samples)
x2 = np.random.normal(1, 1, n_samples)
x3 = np.random.normal(-1, 1, n_samples)
x4 = np.random.normal(-2, 1, n_samples)

x5 = 2 * x1 + 3 * x2 - x3 + x4 + np.random.normal(0, 0.01, n_samples)

df = pl.DataFrame({"x1": x1, "x2": x2, "x3": x3, "x4": x4, "x5": x5})
print(df)

shape: (100, 5)
┌──────────┬──────────┬───────────┬───────────┬───────────┐
│ x1       ┆ x2       ┆ x3        ┆ x4        ┆ x5        │
│ ---      ┆ ---      ┆ ---       ┆ ---       ┆ ---       │
│ f64      ┆ f64      ┆ f64       ┆ f64       ┆ f64       │
╞══════════╪══════════╪═══════════╪═══════════╪═══════════╡
│ 1.127813 ┆ 1.976019 ┆ -0.306735 ┆ -2.932537 ┆ 5.553077  │
│ 1.702715 ┆ 2.837258 ┆ 0.608874  ┆ -3.093993 ┆ 8.240383  │
│ 3.728344 ┆ 1.754147 ┆ -0.640377 ┆ -1.966818 ┆ 11.409613 │
│ 0.130101 ┆ 2.07627  ┆ -2.519377 ┆ -1.094331 ┆ 7.905766  │
│ 1.842719 ┆ 0.689719 ┆ -1.809076 ┆ -2.46972  ┆ 5.091085  │
│ …        ┆ …        ┆ …         ┆ …         ┆ …         │
│ 2.346223 ┆ 0.987793 ┆ 0.792093  ┆ -2.469477 ┆ 4.412669  │
│ 3.270795 ┆ 0.981068 ┆ -1.740227 ┆ -2.858559 ┆ 8.373868  │
│ 2.448963 ┆ 1.937447 ┆ -1.56691  ┆ -1.530378 ┆ 10.756647 │
│ 1.616927 ┆ 3.011794 ┆ -1.378437 ┆ -0.693779 ┆ 12.944466 │
│ 1.235787 ┆ 2.634723 ┆ -0.22323  ┆ -0.927663 ┆ 9.676474  │
└──────────┴──────────┴───────────┴───────────┴───────────┘

Creating a simple VIF function.

In [6]:

df_with_const = df.with_columns(pl.lit(1).alias("const"))
X_matrix = df_with_const.to_numpy()

for i, col in enumerate(df_with_const.columns):
    if col != "const":
        vif = variance_inflation_factor(X_matrix, i)
        print(f"{col}: {vif:.2f}")

x1: 49692.18
x2: 75753.11
x3: 10711.37
x4: 9551.55
x5: 121103.62

As the results show, feature x5 has the highest VIF value, it would be the first feature to be dropped. Note that the other features also have high values, because the generation of the linearly dependent feature also make other features dependent on it.

vif.ipynb