Knowledge Management

How to ignore lines in a text file

gltplus
New Member

I'm struggling with a data source creating daily log files of the following format

01:06:15.558 Server 1.1.1.1: no new files
01:06:35.211 Positive leap second is reported by NTP
Some configuration text
config = 12.12
bootloader = 01.01.2012
01:06:35.267 Positive leap second is reported by NTP
01:07:15.558 Server 1.1.1.1: no new files

After adjusting the props.conf with
SHOULD_LINEMERGE = False
These lines are pared alright in Splunk.

The time is derived from log lines and date from the filename or date. (I don;t know but it just works)

The problem however is that each logfiles starts with +/- 500 lines of start-up config. To make it worse, it is possible that these lines could also be in the middle of these files as well. Because these lines do not contain any time information but other numbers instead Splunk can't interpret these lines. They are showing up with all kind of different weird time stamps. As I don't need these lines I'd like to remove them.

Is there a way to read every line from a text file, parse it as single line=singe event EXCEPT every line that does not start with a timestamp like 01:06:15.558 ?

I tried the following by adding the TRANSFORMS.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{3}\s
DEST_KEY = queue
FORMAT = indexQueue

and in the props.conf:
TRANSFORMS-set = setnull, setparsing

but that does not seems to work.

How can I solve this?

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

gltplus
New Member

Great!! It works!

Thanx alot somesoni2.
This rexeg seems to do the trick!

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...