All Apps and Add-ons

How do I define the empty/default value for space delimited events?

chrismilleruk
Engager

I have some IIS logs with single line events which are space separated and use the "-" character to represent an null value.


#Fields: field1 field2 field3 field4 etc
2013-08-16 13:24:13 192.168.17.187 - 200

and my transform stanza is as follows:

[transform]
DELIMS = " "
FIELDS = field1 field2 field3 etc

My problem is that when viewing results in the search app it reports that field4 appears in 100% of results when in actual fact it might be only a small percentage. This also adversely affects the query language (I have to use field4!="-" instead of field4).

What I'm looking for is something like this:

EMPTY_VALS = "-"
KEEP_EMPTY_VALS = false

Does anything like this exist? Is there a better way of doing it?

Thanks in advance

Edit #1:
It should be noted that I need to do this for pretty much all fields across an increasing number of sourcetypes.

0 Karma
1 Solution

sowings
Splunk Employee
Splunk Employee

You might consider using an EVAL for the fields that might be null. I'd approach it with either trim() or null(). The syntax for these statements in your props.conf is EVAL-<name_of_new_field> = <eval_expression>. You can use this to remap the value of an existing field as well. In your case, I might try this:


EVAL-field4 = if(field4=="-", null(), field4)

Or possibly


EVAL-field4 = trim(field4, "-")

View solution in original post

sowings
Splunk Employee
Splunk Employee

Ok, since EVAL didn't work for you, you might consider a (painful) way to handle this with the field extractions themselves. Instead of using nice, happy, DELIMS, you could instead do regexes to extract each field. The regex could then be written to say "extract this thing only if it's not -", so the regex would fail to find a match, and the desired field would be null.

I think it's just pushing the problem down, though. I'd go with delims and managing it in search language. Ultimately, you'll get to a point where you're dashboarding rather than using the search bar (still useful, though!), so the presence or absence of something in the field picker won't be your driving factor any longer. Things like stats or timechart throw away the other fields, so the dashboard wouldn't have them anyway. You could also use "fields" explicitly in your search to name the ones you want (and no others).

0 Karma

chrismilleruk
Engager

I had a suspicion we'd end up with some sort of regex solution. I was actually wondering whether I could craft a regex transform to turn the space-delimited fields into comma-delimited fields but that's about as far as I got with that thought process.

In any case, I think on balance you're probably right about searching vs dashboarding and it's just a short term inconvenience so I'm going to mark your other answer as correct.

Thanks

0 Karma

sowings
Splunk Employee
Splunk Employee

You might consider using an EVAL for the fields that might be null. I'd approach it with either trim() or null(). The syntax for these statements in your props.conf is EVAL-<name_of_new_field> = <eval_expression>. You can use this to remap the value of an existing field as well. In your case, I might try this:


EVAL-field4 = if(field4=="-", null(), field4)

Or possibly


EVAL-field4 = trim(field4, "-")

chrismilleruk
Engager

Thanks sowings,

EVAL looks pretty interesting. trim() doesn't work but the if()/null() statement has the effect I'm looking for.

I need to do this for pretty much all fields extracted from the DELIMS transform and I have quite a few similar sourcetypes that need the same operation on multiple fields.

There's probably a bit too much of a maintenance overhead with this approach but it's certainly a way to move forward.

0 Karma

sowings
Splunk Employee
Splunk Employee

Note, I would only do this for a) fields which may have the - placeholder and b) the set of those fields that you care about.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...