Getting Data In

PREAMBLE_REGEX used to ignore line of log not working as stated in splunk docs

net1993
Path Finder

Hello
I found this attribute in mysql app in props.conf:
PREAMBLE_REGEX = #\sTime:\s\d+\s+\d{1,2}:\d{2}:\d{2}

test data: Time: 190807 13:26:28

The issue is that nothing happens and the line with this string is still comming in splunk and makes the whole parsing wrong as it creates a new event only for it instead of ignoring it:(

I tried the regex and is correct. Tried it also in splunk with regex command and works fine so I believe this is due to that command.

Some others to help or know more about it?

0 Karma

skalliger
SplunkTrust
SplunkTrust

If I recall correct, PREAMBLE_REGEX tells Splunk to ignore something unstructured in terms of extractions but that does not mean that it won't be indexed.

Skalli

0 Karma

net1993
Path Finder

PREAMBLE_REGEX =
* A regular expression that lets Splunk software ignore "preamble lines",
or lines that occur before lines that represent structured data.
* When set, Splunk software ignores these preamble lines,
based on the pattern you specify.
* Default: not set
* List item

Isn't suppose to ignore any lines which matches the specified pattern?
This attribute was in splunk_TA_mysql and not working. I was need to correct the regex as it was not matching but before or after , it doesn't work.
Maybe I understand wrong the purpose of that preamble liines and if so, can you give example in practice what kind of scenario this will work?

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

It's been an issue that is fixed in 7.2.0 and above. What is your splunk version?

0 Karma

net1993
Path Finder

It's 7.2.6

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

May be your preamble lines are in the middle of file?
Is the file content something like

===begin===
preamble line 1
structured data1
preamble line2
structured data2
preamble line3
structured data3
preamble line4
===end===

0 Karma

net1993
Path Finder

No , it's more like this:

===begin===
structured data1
preamble line 1
structured data2
preamble line2
structured data3
preamble line3
structured data4
structured data5
===end===

*The preamble line doesnt always exist.

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

Interesting use case. If this file growing? or it's one time file?
If it's growing, what is the time interval for file to grow?

0 Karma

net1993
Path Finder

It's growing.
time interval - no idea.
It's mysqld_slow sourcetype- there is an app for it but even that not parsing correct. I got this preamble_line from there
If important , I can try find this time interval but I think its different depending on mysql slow queries log.

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

try setting time_before_close=300 for this particular input and test.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please share all the props.conf settings for that sourcetype and a sample preamble line.

---
If this reply helps you, Karma would be appreciated.
0 Karma

net1993
Path Finder

[mysqld_slow]
pulldown_type = true
category = Database
description = Mysql Slow Query Logs
BREAK_ONLY_BEFORE = ^#\s*User@Host
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %s
TIME_PREFIX = timestamp=
SHOULD_LINEMERGE = true
PREAMBLE_REGEX = #\sTime:\s\d+\s+\d{1,2}:\d{2}:\d{2}
TRUNCATE = 20000

0 Karma

net1993
Path Finder

I have already provided the test data in my first post but I will post for you again:

Time: 190807 13:26:28

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...