Knowledge Management

How to ignore lines in a text file

gltplus
New Member

I'm struggling with a data source creating daily log files of the following format

01:06:15.558 Server 1.1.1.1: no new files
01:06:35.211 Positive leap second is reported by NTP
Some configuration text
config = 12.12
bootloader = 01.01.2012
01:06:35.267 Positive leap second is reported by NTP
01:07:15.558 Server 1.1.1.1: no new files

After adjusting the props.conf with
SHOULD_LINEMERGE = False
These lines are pared alright in Splunk.

The time is derived from log lines and date from the filename or date. (I don;t know but it just works)

The problem however is that each logfiles starts with +/- 500 lines of start-up config. To make it worse, it is possible that these lines could also be in the middle of these files as well. Because these lines do not contain any time information but other numbers instead Splunk can't interpret these lines. They are showing up with all kind of different weird time stamps. As I don't need these lines I'd like to remove them.

Is there a way to read every line from a text file, parse it as single line=singe event EXCEPT every line that does not start with a timestamp like 01:06:15.558 ?

I tried the following by adding the TRANSFORMS.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{3}\s
DEST_KEY = queue
FORMAT = indexQueue

and in the props.conf:
TRANSFORMS-set = setnull, setparsing

but that does not seems to work.

How can I solve this?

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

gltplus
New Member

Great!! It works!

Thanx alot somesoni2.
This rexeg seems to do the trick!

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...