topic Re: Same fields with different values in one event in Splunk Search

Same fields with different values in one event

kkuminsky — Tue, 18 May 2010 19:57:03 GMT

In the following Windows event log message field Account Name appears twice with different values. When I build a report by Account Name it looks like there were two events instead of one, because Splunk is indexing Account Name twice in this case.

Is there a way to avoid indexing the first one (under subject)?

LogName=SecuritySourceName=Microsoft Windows security auditing.

EventCode=4625

EventType=0

Type=Information

ComputerName=AD1.TEIG.production

TaskCategory=Logon

OpCode=Info

RecordNumber=1371401

Keywords=Audit

FailureMessage=An account failed to log on.

Subject:

Security ID: S-1-0-0

Account Name: -

Account Domain: -

Logon ID: 0x0

Logon Type: 3

Account For Which Logon Failed:

Security ID: S-1-0-0

Account Name: APP2$

Account Domain: TEIG.PRODUCTIONFailure

Information: Failure

Reason: An Error occured during Logon.

Status: 0xc0000133

Sub Status: 0x0

Re: Same fields with different values in one event

Lowell — Tue, 18 May 2010 20:33:55 GMT

I don't think your issue is that splunk is indexing your event multiple times. Splunk will only index an event once. I think your questions is really about field extraction and multi-value fields. For WinEventLog:Security events, splunk already has setup the necessary field-extraction rules for you, so all the fields you needs should already be available. However, I think what you hitting is the fact that splunk allows multiple value for these fields. (Which is generally preferable over silently dropping one of the values.)

I'm assuming that in your example, your are ending up with a multi-value field named Account_Name that contains the values of '-' and APP2$.

So what you probably want to do is simply drop the "-" value, which can be done a couple of different ways. You can build your chart or report and simply drop the "-" value after the fact. Or you can tell splunk to keep only the first or last value of a multi-value field, or filter the multi-value field. Here are a few examples to give you an idea of what's possible:

Keep only the first value:

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, 0) | ...

Keep only the last value:

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, -1) | ...

Remove out the "-" value using mvfilter. (I think this is probably the best option for you)

sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name!="-") | ...

Remove out the "-" value (using a post-search command)

sourcetype="WinEventLog:Security" | stats count by Account_Name | search Account_Name!="-"

Here is one last search that will NOT work for what you want. (I feel this is an important gotcha to understand when searching with multi-value fields)

sourcetype="WinEventLog:Security" Account_Name!="-" | ...

Sometimes this may actually appear to work (based on how many events actually contain a dash value, vs how many event contain non-dash values). Here is what's really going on here: Any event that has a field named Account_Name where any of the multi-values is "-" will be dropped from your search results. Therefore, the example event that you provided will be excluded; which means that you will be loosing both the "-" and "APP2$" at the same time. So this approach is not what you want.

Side note: If you are fairly new to splunk, I would suggest that you play around with these different options until you get a better feel for how these different approaches work. You may find that such an exercise will end up saving you time in future splunk searches. Just a thought.

Additional resources:

Functions for eval and where -- For more information about mvindex and mvfilter (and others).
Manipulate and evaluate fields with multiple values -- A more general introduction to dealing with multi-value fields.

Re: Same fields with different values in one event

marcoscala — Tue, 18 May 2010 21:10:57 GMT

An alternative way could be to remap the Account Name field with the following directive in props.conf:

[WinEventLog:security]
EXTRACT-AccountName = Account Name:\s(?P<AccountName>[^\s^:^-]+)\s

This will prevent to have something like "Account Name: -" recognized as another value for the Account name field.

Regards, Marco Scala

Re: Same fields with different values in one event

sideview — Tue, 18 May 2010 22:00:19 GMT

Note that your last last example ('Remove out the "-" value') will actually filter out the events that have both values, not simply clean out the value.

Re: Same fields with different values in one event

sideview — Wed, 19 May 2010 14:03:52 GMT

Oh of course. Sorry i missed that. Feel free to delete this little comment failtrain. my bad.

Re: Same fields with different values in one event

Lowell — Wed, 19 May 2010 21:38:25 GMT

Yes, this will work, and in previous versions of Splunk this may have been the best option. Here's the downside: You now have multiple fields with the same info (AccountName and Account_Name) and some extra regex overhead. Also, if you do have events with multiple account names given (other than "-", of course), than this approach will only match the first one. Perhaps this doesn't happen in this type of event, but it's something to think about.

Re: Same fields with different values in one event

Lowell — Wed, 19 May 2010 21:44:01 GMT

Your regex will also block any account name with a dash in it, not just "-". For example, a value of "APP-2" doesn't match. You could try this regex instead: EXTRACT-AccountName = ^\s*Account Name:\s(?P<AccountName>(?!-$).+)\s*$

Re: Same fields with different values in one event

Lowell — Wed, 19 May 2010 21:45:16 GMT

I can't delete others comments. I removed mine. (and I can remove this one later)

Re: Same fields with different values in one event

kkuminsky — Thu, 20 May 2010 00:02:39 GMT

Guys, thank you for the detailed info. I'm new to splunk and I'm impressed by it's search functionality.

So, I've adopted this one:

eval Account_Name=mvindex(Account_Name, -1) | ...

Mostly because:

There are some events that have "-" value for both Account_name fields
I'm expecting different values for the first account field, not only "-", but the one that I'm interested in is the second one.

Re: Same fields with different values in one event

Lowell — Thu, 20 May 2010 00:20:51 GMT

Sounds like good reasoning to me.

Re: Same fields with different values in one event

marcoscala — Mon, 24 May 2010 17:34:27 GMT

this is useful! thanks Lowell.

Re: Same fields with different values in one event

essklau — Thu, 15 May 2014 20:34:17 GMT

Great answer; loved the spl breakouts

Re: Same fields with different values in one event

jkat54 — Sat, 07 Dec 2019 03:13:17 GMT

That was then, this is now... and NOW you have mod rights so you CAN delete others comments :-). Both of you! All of us!

Cheers!