Getting Data In

Lines starting with semicolon(;) needs to discarded completely from indexing . My props and transforms not working

NAVEEN_CTS
Path Finder

I would like to remove any lines that start with semicolon(;) from indexing. Below are my config files and sample data. Im not receiving logs to my splunk. Please help

Sample Log data:
123 123 12123
;123 123 123 123 123
; 123 123 123 123
214121 ; 214 ; 1212 ; 33

My inputs.conf
[monitor://$SPLUNK_HOME/etc/apps/testapp/log]
index=test
sourcetype=test

My props.conf
[source:://$SPLUNK_HOME/etc/apps/testapp/log]                  
TRANSFORMS-null= setnull

My transforms.conf
[setnull]
REGEX = ^;.*$
DEST_KEY = queue
FORMAT = nullQueue
  

Tags (2)
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try

My props.conf

[source:://$SPLUNK_HOME/etc/apps/testapp/log]            
SHOULD_LINEMERGE = false       
TRANSFORMS-null= setnull_colons

My transforms.conf

[setnull_colons]
REGEX = ^\;.+
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

NAVEEN_CTS
Path Finder

this didn't work as well.

Let me review my config first.

1) I have placed my log file at HF and it has full permission
2) inputs .conf is placed in /apps/local
3) props.conf and transforms.conf is placed at idx --> /apps/local/

My inputs.conf

[monitor://$SPLUNK_HOME/etc/apps/app_name/log/test.txt]
index=test
sourcetype=test

My props:
[source::/$SPLUNK_HOME/etc/apps/app_name/log/test.txt]
SHOULD_LINEMERGE = false
TRANSFORMS-drop = delLines

My Transforms:
[delLines]
REGEX = ^[^\;].+
DEST_KEY = queue
FORMAT = nullQueue

I have 2 problems
1) I get all the lines as single event.
2) Lines starting with (;) is not removed

Please let me know the missing config here. Thanks in advance

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The props and transforms should be in the first Splunk Enterprise instance in your data flow. If you've heavy forwarders in front of indexes, then heavy forwarders should have that config. Also, do remember to restart HF after applying those configurations.

For your line breaking, could you post sample events and show what are your event boundaries?

0 Karma

NAVEEN_CTS
Path Finder

@somesoni2 .

My log file looks like this. In splunk im seeing it as a single line And the line that is starting with ; needs to be removed

Sample Log:

;*************
; X ABCDEF
;*************
xxxxxxxx BE A X.XX.XX.XXX
xxxxxxxx BE A X.XX.XX.XXX
XXXXXXXXXXXXXXXXXXXXXXXXX IN A X.XX.XX.XXX
XXXXXXXXXXXXXXXX BE A XX.XX.XX.XXX
XXXXXXXXXXXXXXX BE A XX.XX.XX.XXX
XXXXXXX BE A XX.XX.XX.XXX

0 Karma

niketn
Legend

@NAVEEN_CTS do you have event breaking in props.conf for breaking every line? Do you have timestamp in the data? Does time get identified correctly? With nullQueue not working are you seeing each line as separate event with correct event raw data and correct time stamp?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

NAVEEN_CTS
Path Finder

@niketnilay no i have not set the event breaking and also my log doesn't have a timestamp.....it is just a dump of some report which i would like to index and use it as a lookup. But it has lot of junk data ....which i want to remove lines that starts with ;

So not setting the event breaking is the problem?

0 Karma

NAVEEN_CTS
Path Finder

Could you please help with line break regex? Also where should we keep these props and transforms? Both HF and IDX? Right now im keeping it in idx alone. Log file is monitored in HF

0 Karma

vnravikumar
Champion

Hi

Try with this regex

^[^\;].+
0 Karma

NAVEEN_CTS
Path Finder

@vnravikumar No it didnt work .....same as before. I get all the lines as a single event .... may be i have to try event breaking

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...