Splunk Machine Learning Toolkit: Does ML Toolkit categorical data generate biased (Logistic Regression) models?

Explorer

Hi Splunk Experts--

A colleague of mine and I are exploring the Splunk Machine Learning
Toolkit and, more specifically, using the ML Toolkit to perform
Logistic Regression analysis on a dataset that includes categorical
data as independent variables.

When performing LR on categorical data, we've been taught the
statistical technique of creating "dummy variables" that, in effect,
transform the categorical data into a series of numeric variables.

Example: Imagine a single categorical attribute `color` with values
["red", "yellow" and "blue"]. That categorical data could be
transformed into three dummy variables (say, `is_red`, `is_yellow` and
`is_blue`) where each dummy variable would have a value [`0` or `1`].

Given this, a data record where `color` had value `yellow` would be
transformed into

``````  is_red:    0
is_yellow: 1
is_blue:   0
``````

We were both taught that when using dummy variables in Logistic
Regression, you need to omit one dummy variable from the set
representing a given categorical variable. Doing this prevents
double-counting of that omitted categorical value (say, `is_blue`)
because having zeros in all other dummy variables effectively
represents a one in the omitted dummy variable.

We've been crawling through the ML Toolkit (Logistic Regression)
source code to see how it handles categorical data and have found
something that surprises both of us: Specifically, the
`prepare_features` method in `df_util.py` (see below), which uses pandas
to create dummy variables for categorical data, by invoking the pandas
`get_dummies` method (see line 27 below).

``````  def prepare_features(X, variables, final_columns=None, get_dummies=True):
"""Prepare features.

This method defines conventional steps to prepare features:
- drop unused columns
- drop rows that have missing values
- optionally (if get_dummies==True)
- convert categorical fields into indicator dummy variables
- optionally (if final_column is provided)
- make the resulting dataframe match final_columns

Args:
X (dataframe): input dataframe
variables (list): column names
final_columns (list): finalized column names
get_dummies (bool): indicate if categorical variable should be converted

Returns:
X (dataframe): prepared feature dataframe
nans (np array): boolean array to indicate which rows have missing
values in the original dataframe
columns (list): sorted list of feature column names
"""
X, nans = drop_unused_and_missing(X, variables)
if get_dummies:
filter_non_numeric(X)
X = pd.get_dummies(X, prefix_sep='=', sparse=True)
if final_columns is not None:
drop_unused_fields(X, final_columns)
assert_any_fields(X)
fill_missing_fields(X, final_columns)
assert_any_rows(X)
assert_any_fields(X)
columns = sort_fields(X)
return (X, nans, columns)
``````

The ML toolkit seems to use pandas 0.17. In pandas 0.18 the
`get_dummies` method supports a `drop_first` parameter which omits the
first dummy variable for a categorical variable, but that's not
available in pandas 0.17. To us this means that the Splunk ML Toolkit
code should contain code to drop one of the dummy variables returned
by pandas-- and we don't see code that does this.

So (finally!) here are our questions:

• Are the assertions/interpretations above correct?

• If so does it follow that the ML Toolkit is not handling categorical
data correctly-- that it will produce biased models when the input
contains categorical data?

• And if so, is there a technique for using the ML Toolkit to perform
Logistic Regression on categorical data that allows creation of
models without this bias?

Tags (2)
1 Solution
Splunk Employee

Hi,

Thank you for asking, this is an incredibly valuable question! You have a very good understanding of dummy variables.

First, about the bias in the model. Let's assume you have dummy variables `x1`, `x2`, `x3`, such that `x1 + x2 + x3 = 1`,
With `m-1` dummy variables, your linear model can be expressed as
`y = α0 + α1 * x1 + α2 * x2`
With `m` dummy variables, your linear model is now:
`y = β0 + β1 * x1 + β2 * x2 + β3 * x3`
Since `x3 = 1 − x1 − x2`, you get
`y = β0 + β1 * x1 + β2 * x2 + β3 * (1 − x1 − x2) = (β0 + β3) + (β1 − β3) * x1 + (β2 − β3) * x2`
Essentially you have
`α0 = β0 + β3`, `α1 = β1 − β3`, `α2 = β2 − β3`

So, these two models are equivalent, and there is no bias introduced as you see in this exercise.

Now, the question is, what's introduced here? Collinearity is what you are after, since you can always tell the value of the left out dummy variable if you know `m-1` of them. Collinearity can cause computational problems for linear regression since the matrix inversion can not be performed. But for logistic regression, depending on the computational scheme under the hood, e.g. gradient descent, numerical instability may not be an issue. Moreover, the LogisticRegression model in sklearn uses a regularization, `penalty='l2'` and `C=1.0`, which means feature collinearity will be penalized. Therefore, using the full `m` dummy variables instead of `m-1` does not introduce bias to the model, except for potential numerical instability.

In practice, to avoid the potential numerical instability issue, if you decide to go with `m-1` dummy variables, you may have the following options:

1) With latest version of MLTK (you are right it uses pandas 0.17), you can modify the `prepare_features_and_target` method in `df_util.py`, instead of doing

``````    X = pd.get_dummies(X, prefix_sep='=', sparse=True)
``````

you can use the following code to drop the first column of the created dummy variables for each categorical variable:

``````    columns_to_encode = X.select_dtypes(include=['object', 'category']).columns
for col in columns_to_encode:
X = X.join(pd.get_dummies(X.pop(col), prefix=col, prefix_sep='=').iloc[:, 1:])
``````

2) As you already mentioned in your post, `drop_first=True` is supported in pandas 0.18+, you could use this when a future version of Python for Scientific Computing is released.

On the other hand, if you want to reduce the effect of collinearity in your model, you can also use some preprocessing methods, e.g. Field Selector to select features, or PCA to remove collinearity. You can also use algorithms like Random Forest that are least affected by feature multicollinearity.

Hope it helps clarify some of the issues.

zd

Splunk Employee

In addition to @yangzd's response below, you can do your own categorical encoding quite simply with eval. Say for example our field `color` with values `is_red`, `is_yellow`, and `is_blue`, and you'd like to to encode the 3 levels into two dummy variables (treating `is_blue` as the base):

``````| eval {color} = 1
| fillnull is_red is_yellow
| fields - is_blue
``````

The `{color}` on the left side of eval will take the value of the field and use it as the name of the new field.

Explorer

Hi aljohnson-- Thanks very much for this. This, it turns out, is the method we're using to do the comparison between letting the ML Toolkit handle categorical data (described above) and converting our categorical data to dummy variables prior to invoking the `fit` command. In fact, we generalize this so something like (for a categorical attribute "foo"):

``````...
| eval foo_is_{foo} = 1
...
| foreach foo_is_* [ eval <<FIELD>>=coalesce(<<FIELD>>,0) ]
``````

Thanks for taking the time to comment!

Splunk Employee

Hi,

Thank you for asking, this is an incredibly valuable question! You have a very good understanding of dummy variables.

First, about the bias in the model. Let's assume you have dummy variables `x1`, `x2`, `x3`, such that `x1 + x2 + x3 = 1`,
With `m-1` dummy variables, your linear model can be expressed as
`y = α0 + α1 * x1 + α2 * x2`
With `m` dummy variables, your linear model is now:
`y = β0 + β1 * x1 + β2 * x2 + β3 * x3`
Since `x3 = 1 − x1 − x2`, you get
`y = β0 + β1 * x1 + β2 * x2 + β3 * (1 − x1 − x2) = (β0 + β3) + (β1 − β3) * x1 + (β2 − β3) * x2`
Essentially you have
`α0 = β0 + β3`, `α1 = β1 − β3`, `α2 = β2 − β3`

So, these two models are equivalent, and there is no bias introduced as you see in this exercise.

Now, the question is, what's introduced here? Collinearity is what you are after, since you can always tell the value of the left out dummy variable if you know `m-1` of them. Collinearity can cause computational problems for linear regression since the matrix inversion can not be performed. But for logistic regression, depending on the computational scheme under the hood, e.g. gradient descent, numerical instability may not be an issue. Moreover, the LogisticRegression model in sklearn uses a regularization, `penalty='l2'` and `C=1.0`, which means feature collinearity will be penalized. Therefore, using the full `m` dummy variables instead of `m-1` does not introduce bias to the model, except for potential numerical instability.

In practice, to avoid the potential numerical instability issue, if you decide to go with `m-1` dummy variables, you may have the following options:

1) With latest version of MLTK (you are right it uses pandas 0.17), you can modify the `prepare_features_and_target` method in `df_util.py`, instead of doing

``````    X = pd.get_dummies(X, prefix_sep='=', sparse=True)
``````

you can use the following code to drop the first column of the created dummy variables for each categorical variable:

``````    columns_to_encode = X.select_dtypes(include=['object', 'category']).columns
for col in columns_to_encode:
X = X.join(pd.get_dummies(X.pop(col), prefix=col, prefix_sep='=').iloc[:, 1:])
``````

2) As you already mentioned in your post, `drop_first=True` is supported in pandas 0.18+, you could use this when a future version of Python for Scientific Computing is released.

On the other hand, if you want to reduce the effect of collinearity in your model, you can also use some preprocessing methods, e.g. Field Selector to select features, or PCA to remove collinearity. You can also use algorithms like Random Forest that are least affected by feature multicollinearity.

Hope it helps clarify some of the issues.

zd

Explorer

Hi yangzd! Thanks for an awesome (clear and thorough) answer. We really appreciate your taking the time it must have taken to write this up. It's not clear if (or to what extent) our specific use-cases will be subject to the stability problem you describe. After some conversation I believe we're going to clone the ML Toolkit and compare models created with the stock and modified version. Let me know if you're interested in hearing about the results. And thanks so much again for your help. ..j

Splunk Employee

Hi jsinnott_, that sounds great. Yes please do let us know the comparison results. Look forward to it. -zd

Splunk Employee

This question ignited some interesting discussion on the ML teams here, I'll try to nudge someone into answering. 🙂

Explorer

Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...