All Apps and Add-ons

How do I define the empty/default value for space delimited events?

I have some IIS logs with single line events which are space separated and use the "-" character to represent an null value.


#Fields: field1 field2 field3 field4 etc
2013-08-16 13:24:13 192.168.17.187 - 200

and my transform stanza is as follows:

[transform]
DELIMS = " "
FIELDS = field1 field2 field3 etc

My problem is that when viewing results in the search app it reports that field4 appears in 100% of results when in actual fact it might be only a small percentage. This also adversely affects the query language (I have to use field4!="-" instead of field4).

What I'm looking for is something like this:

EMPTY_VALS = "-"
KEEP_EMPTY_VALS = false

Does anything like this exist? Is there a better way of doing it?

Thanks in advance

Edit #1:
It should be noted that I need to do this for pretty much all fields across an increasing number of sourcetypes.

0 Karma
1 Solution

Splunk Employee
Splunk Employee

You might consider using an EVAL for the fields that might be null. I'd approach it with either trim() or null(). The syntax for these statements in your props.conf is EVAL-<name_of_new_field> = <eval_expression>. You can use this to remap the value of an existing field as well. In your case, I might try this:


EVAL-field4 = if(field4=="-", null(), field4)

Or possibly


EVAL-field4 = trim(field4, "-")

View solution in original post

Splunk Employee
Splunk Employee

Ok, since EVAL didn't work for you, you might consider a (painful) way to handle this with the field extractions themselves. Instead of using nice, happy, DELIMS, you could instead do regexes to extract each field. The regex could then be written to say "extract this thing only if it's not -", so the regex would fail to find a match, and the desired field would be null.

I think it's just pushing the problem down, though. I'd go with delims and managing it in search language. Ultimately, you'll get to a point where you're dashboarding rather than using the search bar (still useful, though!), so the presence or absence of something in the field picker won't be your driving factor any longer. Things like stats or timechart throw away the other fields, so the dashboard wouldn't have them anyway. You could also use "fields" explicitly in your search to name the ones you want (and no others).

0 Karma

I had a suspicion we'd end up with some sort of regex solution. I was actually wondering whether I could craft a regex transform to turn the space-delimited fields into comma-delimited fields but that's about as far as I got with that thought process.

In any case, I think on balance you're probably right about searching vs dashboarding and it's just a short term inconvenience so I'm going to mark your other answer as correct.

Thanks

0 Karma

Splunk Employee
Splunk Employee

You might consider using an EVAL for the fields that might be null. I'd approach it with either trim() or null(). The syntax for these statements in your props.conf is EVAL-<name_of_new_field> = <eval_expression>. You can use this to remap the value of an existing field as well. In your case, I might try this:


EVAL-field4 = if(field4=="-", null(), field4)

Or possibly


EVAL-field4 = trim(field4, "-")

View solution in original post

Thanks sowings,

EVAL looks pretty interesting. trim() doesn't work but the if()/null() statement has the effect I'm looking for.

I need to do this for pretty much all fields extracted from the DELIMS transform and I have quite a few similar sourcetypes that need the same operation on multiple fields.

There's probably a bit too much of a maintenance overhead with this approach but it's certainly a way to move forward.

0 Karma

Splunk Employee
Splunk Employee

Note, I would only do this for a) fields which may have the - placeholder and b) the set of those fields that you care about.

0 Karma