Getting Data In

How to exclude Null Values from field extractions

dsofoulis
Path Finder

I am building a TA.

The issue I am having is the log file has a field error="". Even though it is null the error field is still there and causing CIM to tag the logs as error. I am hoping you can help me to only return the error field if there is a value other than null. Also note, I am looking for a way to do this without having to write a regex string as I have the same issue across a bunch of other sourcetypes.

<30>2017:08:27-10:30:12 sophos httpproxy[19742]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="1.1.1.1" dstip="1.1.1.1" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="855" request="0xdffdb" url="https://www.google.com.au/" referer="" error="" authtime="0" dnstime="579003" cattime="288" avscantime="0" fullreqtime="109809548" device="0" auth="0" ua="" exceptions="" category="145" reputation="trusted" categoryname="Search Engines" application="google" app-id="182"
0 Karma

woodcock
Esteemed Legend

You would probably be best to strip the null error completely out of the raw event with this on your Indexers:

SEDCMD_remove_empty_error_KVP = "s/\s+error=\"\"//"
0 Karma

ddrillic
Ultra Champion

Similar question at How do I remove a null field?

0 Karma

dsofoulis
Path Finder

thanks for sharing, but will only remove the null value when performing a search. I need to this to happen at index time.

0 Karma

ddrillic
Ultra Champion

Oh, maybe @somesoni2 solution can be useful - Is it possible to replace null fields at index-time?

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Did that solution work @dsofoulis? If so we can close the question.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...