Knowledge Management

How to ignore lines in a text file

gltplus
New Member

I'm struggling with a data source creating daily log files of the following format

01:06:15.558 Server 1.1.1.1: no new files
01:06:35.211 Positive leap second is reported by NTP
Some configuration text
config = 12.12
bootloader = 01.01.2012
01:06:35.267 Positive leap second is reported by NTP
01:07:15.558 Server 1.1.1.1: no new files

After adjusting the props.conf with
SHOULD_LINEMERGE = False
These lines are pared alright in Splunk.

The time is derived from log lines and date from the filename or date. (I don;t know but it just works)

The problem however is that each logfiles starts with +/- 500 lines of start-up config. To make it worse, it is possible that these lines could also be in the middle of these files as well. Because these lines do not contain any time information but other numbers instead Splunk can't interpret these lines. They are showing up with all kind of different weird time stamps. As I don't need these lines I'd like to remove them.

Is there a way to read every line from a text file, parse it as single line=singe event EXCEPT every line that does not start with a timestamp like 01:06:15.558 ?

I tried the following by adding the TRANSFORMS.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{3}\s
DEST_KEY = queue
FORMAT = indexQueue

and in the props.conf:
TRANSFORMS-set = setnull, setparsing

but that does not seems to work.

How can I solve this?

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

gltplus
New Member

Great!! It works!

Thanx alot somesoni2.
This rexeg seems to do the trick!

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...