Getting Data In

Ignoring any data record that begins with a "#" character

teward001
Path Finder

I've got data being imported from a CSV file into a custom data type, but it's reading the first line (which begins with # in all the data sources) as an actual record.

Is it possible to get Splunk 6.0.x to ignore any line beginning with a # character?

Tags (2)
0 Karma
1 Solution

teward001
Path Finder

I figured this out, what was going on is it wasn't applying the CSV extractions, so header lines were getting included in event data.

Problem solved by switching the datatypes to all csv, but making one custom CSV with the necessary timestamp search modifications. ^.^

View solution in original post

0 Karma

nithish
Loves-to-Learn

I had similar issue and below props and transforms worked for me in my scenario where it ignored the lines started with #:

transforms.conf:


[setnull]
REGEX=^[#\r\n]
DEST_KEY=queue
FORMAT=nullQueue

props.conf:

[sourcetype]
TRANSFORMS-ignore_comments = setnull

0 Karma

teward001
Path Finder

I figured this out, what was going on is it wasn't applying the CSV extractions, so header lines were getting included in event data.

Problem solved by switching the datatypes to all csv, but making one custom CSV with the necessary timestamp search modifications. ^.^

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I've wondered the same thing and have an idea, but haven't had a chance to try it. Set your transforms.conf file to send lines beginning with '#' to nullQueue.

props.conf

[<sourcetype>]
 SHOULD_LINEMERGE = false
 TRANSFORMS-set = setnull,setparsing

transforms.conf

[setnull]
 REGEX = ^#.
 DEST_KEY = queue
 FORMAT = nullQueue

[setparsing]
 REGEX = logit
 DEST_KEY = queue
 FORMAT = indexQueue
---
If this reply helps you, Karma would be appreciated.

splunkranger
Path Finder

What he said ^

0 Karma

splunkranger
Path Finder

Can you post an example? Do you want to not index those records, or just not have them show up in search?

0 Karma

teward001
Path Finder

Try not posting comments as answers, it irks me.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...