Getting Data In

How to exclude Null Values from field extractions

dsofoulis
Path Finder

I am building a TA.

The issue I am having is the log file has a field error="". Even though it is null the error field is still there and causing CIM to tag the logs as error. I am hoping you can help me to only return the error field if there is a value other than null. Also note, I am looking for a way to do this without having to write a regex string as I have the same issue across a bunch of other sourcetypes.

<30>2017:08:27-10:30:12 sophos httpproxy[19742]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="1.1.1.1" dstip="1.1.1.1" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="855" request="0xdffdb" url="https://www.google.com.au/" referer="" error="" authtime="0" dnstime="579003" cattime="288" avscantime="0" fullreqtime="109809548" device="0" auth="0" ua="" exceptions="" category="145" reputation="trusted" categoryname="Search Engines" application="google" app-id="182"
0 Karma

woodcock
Esteemed Legend

You would probably be best to strip the null error completely out of the raw event with this on your Indexers:

SEDCMD_remove_empty_error_KVP = "s/\s+error=\"\"//"
0 Karma

ddrillic
Ultra Champion

Similar question at How do I remove a null field?

0 Karma

dsofoulis
Path Finder

thanks for sharing, but will only remove the null value when performing a search. I need to this to happen at index time.

0 Karma

ddrillic
Ultra Champion

Oh, maybe @somesoni2 solution can be useful - Is it possible to replace null fields at index-time?

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Did that solution work @dsofoulis? If so we can close the question.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...