Getting Data In

How to exclude Null Values from field extractions

dsofoulis
Path Finder

I am building a TA.

The issue I am having is the log file has a field error="". Even though it is null the error field is still there and causing CIM to tag the logs as error. I am hoping you can help me to only return the error field if there is a value other than null. Also note, I am looking for a way to do this without having to write a regex string as I have the same issue across a bunch of other sourcetypes.

<30>2017:08:27-10:30:12 sophos httpproxy[19742]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="1.1.1.1" dstip="1.1.1.1" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="855" request="0xdffdb" url="https://www.google.com.au/" referer="" error="" authtime="0" dnstime="579003" cattime="288" avscantime="0" fullreqtime="109809548" device="0" auth="0" ua="" exceptions="" category="145" reputation="trusted" categoryname="Search Engines" application="google" app-id="182"
0 Karma

woodcock
Esteemed Legend

You would probably be best to strip the null error completely out of the raw event with this on your Indexers:

SEDCMD_remove_empty_error_KVP = "s/\s+error=\"\"//"
0 Karma

ddrillic
Ultra Champion

Similar question at How do I remove a null field?

0 Karma

dsofoulis
Path Finder

thanks for sharing, but will only remove the null value when performing a search. I need to this to happen at index time.

0 Karma

ddrillic
Ultra Champion

Oh, maybe @somesoni2 solution can be useful - Is it possible to replace null fields at index-time?

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Did that solution work @dsofoulis? If so we can close the question.

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...