Same fields with different values in one event

[kkuminsky]

In the following Windows event log message field Account Name appears twice with different values. When I build a report by Account Name it looks like there were two events instead of one, because Splunk is indexing Account Name twice in this case.

Is there a way to avoid indexing the first one (under subject)?

---

LogName=SecuritySourceName=Microsoft Windows security auditing.

EventCode=4625

EventType=0

Type=Information

ComputerName=AD1.TEIG.production

TaskCategory=Logon

OpCode=Info

RecordNumber=1371401

Keywords=Audit

FailureMessage=An account failed to log on.

Subject:

Security ID: S-1-0-0

Account Name: -

Account Domain: -

Logon ID: 0x0

Logon Type: 3

Account For Which Logon Failed:

Security ID: S-1-0-0

Account Name: APP2$

Account Domain: TEIG.PRODUCTIONFailure

Information: Failure

Reason: An Error occured during Logon.

Status: 0xc0000133

Sub Status: 0x0

[Lowell · Accepted solution]

I don't think your issue is that splunk is indexing your event multiple times. Splunk will only index an event once. I think your questions is really about field extraction and multi-value fields. For WinEventLog:Security events, splunk already has setup the necessary field-extraction rules for you, so all the fields you needs should already be available. However, I think what you hitting is the fact that splunk allows multiple value for these fields. (Which is generally preferable over silently dropping one of the values.)

I'm assuming that in your example, your are ending up with a multi-value field named Account_Name that contains the values of '-' and APP2$.

So what you probably want to do is simply drop the "-" value, which can be done a couple of different ways. You can build your chart or report and simply drop the "-" value after the fact. Or you can tell splunk to keep only the first or last value of a multi-value field, or filter the multi-value field. Here are a few examples to give you an idea of what's possible:

Keep only the first value:

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, 0) | ...

Keep only the last value:

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, -1) | ...

Remove out the "-" value using mvfilter. (I think this is probably the best option for you)

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name!="-") | ...

Remove out the "-" value (using a post-search command)

sourcetype="WinEventLog:Security" | stats count by Account_Name | search Account_Name!="-"

Here is one last search that will NOT work for what you want. (I feel this is an important gotcha to understand when searching with multi-value fields)

sourcetype="WinEventLog:Security" Account_Name!="-" | ...

Sometimes this may actually appear to work (based on how many events actually contain a dash value, vs how many event contain non-dash values). Here is what's really going on here: Any event that has a field named Account_Name where any of the multi-values is "-" will be dropped from your search results. Therefore, the example event that you provided will be excluded; which means that you will be loosing both the "-" and "APP2$" at the same time. So this approach is not what you want.

Side note: If you are fairly new to splunk, I would suggest that you play around with these different options until you get a better feel for how these different approaches work. You may find that such an exercise will end up saving you time in future splunk searches. Just a thought.

---

Additional resources:

• Functions for eval and where -- For more information about mvindex and mvfilter (and others).
• Manipulate and evaluate fields with multiple values -- A more general introduction to dealing with multi-value fields.

[kkuminsky]

Guys, thank you for the detailed info. I'm new to splunk and I'm impressed by it's search functionality.

So, I've adopted this one:

eval Account_Name=mvindex(Account_Name, -1) | ...

Mostly because:

1. There are some events that have "-" value for both Account_name fields

2. I'm expecting different values for the first account field, not only "-", but the one that I'm interested in is the second one.

[Lowell]

Sounds like good reasoning to me.

[marcoscala · Accepted solution]

An alternative way could be to remap the Account Name field with the following directive in props.conf:

[WinEventLog:security]
EXTRACT-AccountName = Account Name:\s(?P<AccountName>[^\s^:^-]+)\s

This will prevent to have something like "Account Name: -" recognized as another value for the Account name field.

Regards, Marco Scala

[marcoscala]

this is useful! thanks Lowell.

[Lowell]

Your regex will also block any account name with a dash in it, not just "-". For example, a value of "APP-2" doesn't match. You could try this regex instead: EXTRACT-AccountName = ^\s*Account Name:\s(?P<AccountName>(?!-$).+)\s*$

[Lowell]

Yes, this will work, and in previous versions of Splunk this may have been the best option. Here's the downside: You now have multiple fields with the same info (AccountName and Account_Name) and some extra regex overhead. Also, if you do have events with multiple account names given (other than "-", of course), than this approach will only match the first one. Perhaps this doesn't happen in this type of event, but it's something to think about.

[Lowell · Accepted solution]

I don't think your issue is that splunk is indexing your event multiple times. Splunk will only index an event once. I think your questions is really about field extraction and multi-value fields. For WinEventLog:Security events, splunk already has setup the necessary field-extraction rules for you, so all the fields you needs should already be available. However, I think what you hitting is the fact that splunk allows multiple value for these fields. (Which is generally preferable over silently dropping one of the values.)

I'm assuming that in your example, your are ending up with a multi-value field named Account_Name that contains the values of '-' and APP2$.

So what you probably want to do is simply drop the "-" value, which can be done a couple of different ways. You can build your chart or report and simply drop the "-" value after the fact. Or you can tell splunk to keep only the first or last value of a multi-value field, or filter the multi-value field. Here are a few examples to give you an idea of what's possible:

Keep only the first value:

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, 0) | ...

Keep only the last value:

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, -1) | ...

Remove out the "-" value using mvfilter. (I think this is probably the best option for you)

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name!="-") | ...

Remove out the "-" value (using a post-search command)

sourcetype="WinEventLog:Security" | stats count by Account_Name | search Account_Name!="-"

Here is one last search that will NOT work for what you want. (I feel this is an important gotcha to understand when searching with multi-value fields)

sourcetype="WinEventLog:Security" Account_Name!="-" | ...

Sometimes this may actually appear to work (based on how many events actually contain a dash value, vs how many event contain non-dash values). Here is what's really going on here: Any event that has a field named Account_Name where any of the multi-values is "-" will be dropped from your search results. Therefore, the example event that you provided will be excluded; which means that you will be loosing both the "-" and "APP2$" at the same time. So this approach is not what you want.

Side note: If you are fairly new to splunk, I would suggest that you play around with these different options until you get a better feel for how these different approaches work. You may find that such an exercise will end up saving you time in future splunk searches. Just a thought.

---

Additional resources:

• Functions for eval and where -- For more information about mvindex and mvfilter (and others).
• Manipulate and evaluate fields with multiple values -- A more general introduction to dealing with multi-value fields.

[essklau]

Great answer; loved the spl breakouts

[Lowell]

I can't delete others comments. I removed mine. (and I can remove this one later)

[jkat54]

That was then, this is now... and NOW you have mod rights so you CAN delete others comments :-). Both of you! All of us!

Cheers!

[sideview]

Oh of course. Sorry i missed that. Feel free to delete this little comment failtrain. my bad.

[sideview]

Note that your last last example ('Remove out the "-" value') will actually filter out the events that have both values, not simply clean out the value.