Knowledge Management

How to ignore lines in a text file

gltplus
New Member

I'm struggling with a data source creating daily log files of the following format

01:06:15.558 Server 1.1.1.1: no new files
01:06:35.211 Positive leap second is reported by NTP
Some configuration text
config = 12.12
bootloader = 01.01.2012
01:06:35.267 Positive leap second is reported by NTP
01:07:15.558 Server 1.1.1.1: no new files

After adjusting the props.conf with
SHOULD_LINEMERGE = False
These lines are pared alright in Splunk.

The time is derived from log lines and date from the filename or date. (I don;t know but it just works)

The problem however is that each logfiles starts with +/- 500 lines of start-up config. To make it worse, it is possible that these lines could also be in the middle of these files as well. Because these lines do not contain any time information but other numbers instead Splunk can't interpret these lines. They are showing up with all kind of different weird time stamps. As I don't need these lines I'd like to remove them.

Is there a way to read every line from a text file, parse it as single line=singe event EXCEPT every line that does not start with a timestamp like 01:06:15.558 ?

I tried the following by adding the TRANSFORMS.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{3}\s
DEST_KEY = queue
FORMAT = indexQueue

and in the props.conf:
TRANSFORMS-set = setnull, setparsing

but that does not seems to work.

How can I solve this?

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

gltplus
New Member

Great!! It works!

Thanx alot somesoni2.
This rexeg seems to do the trick!

0 Karma
Get Updates on the Splunk Community!

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...

Splunk MCP & Agentic AI: Machine Data Without Limits

Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization uses ...