Getting Data In

How to exclude Null Values from field extractions

dsofoulis
Path Finder

I am building a TA.

The issue I am having is the log file has a field error="". Even though it is null the error field is still there and causing CIM to tag the logs as error. I am hoping you can help me to only return the error field if there is a value other than null. Also note, I am looking for a way to do this without having to write a regex string as I have the same issue across a bunch of other sourcetypes.

<30>2017:08:27-10:30:12 sophos httpproxy[19742]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="1.1.1.1" dstip="1.1.1.1" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="855" request="0xdffdb" url="https://www.google.com.au/" referer="" error="" authtime="0" dnstime="579003" cattime="288" avscantime="0" fullreqtime="109809548" device="0" auth="0" ua="" exceptions="" category="145" reputation="trusted" categoryname="Search Engines" application="google" app-id="182"
0 Karma

woodcock
Esteemed Legend

You would probably be best to strip the null error completely out of the raw event with this on your Indexers:

SEDCMD_remove_empty_error_KVP = "s/\s+error=\"\"//"
0 Karma

ddrillic
Ultra Champion

Similar question at How do I remove a null field?

0 Karma

dsofoulis
Path Finder

thanks for sharing, but will only remove the null value when performing a search. I need to this to happen at index time.

0 Karma

ddrillic
Ultra Champion

Oh, maybe @somesoni2 solution can be useful - Is it possible to replace null fields at index-time?

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Did that solution work @dsofoulis? If so we can close the question.

0 Karma
Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...