Splunk Search

How to configure props.conf to index txt files as one new event when text "~~CTRL AS~~:" appears, not by timestamps?

vtsguerrero
Contributor

Hello everybody!

I could use some help with this project that I've been working with...
I have some .txt files which show timestamp in some lines like this " ---- FRIDAY, 05 DEC 2014 ---- "
But the point is, when I index it, it's counting every single datetime as new event, and it should consider the whole .txt as ONE EVENT.
The text I have in particular that defines this txt is unique is this:

~~CTRL AS~~:

Any idea how could I make a Regex for this to consider every time a " ~~CTRL AS~~: " is a new event, not based on the timestamps actually.
Thanks in adv!
Bst rgds!

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Start with these specs in the relevant props.conf stanza:

SHOULD_LINEMERGE = true
TRUNCATE = 0
MAX_EVENTS = 500
BREAK_ONLY_BEFORE = ~~CTRL AS~~
DATETIME_CONFIG = current
---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Start with these specs in the relevant props.conf stanza:

SHOULD_LINEMERGE = true
TRUNCATE = 0
MAX_EVENTS = 500
BREAK_ONLY_BEFORE = ~~CTRL AS~~
DATETIME_CONFIG = current
---
If this reply helps you, Karma would be appreciated.

vtsguerrero
Contributor

This props.conf should be placed inside the app folder right?
And I should re-index the data in the preview mode to see any changes...

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, app/local/props.conf. You must re-index.

I suggest using a test index until you've found the right settings. That makes it easier to clean up and keeps unusable events out of your regular indexes.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vtsguerrero
Contributor

Still tryin' to re-index it, but when I applyin' this new stanza if keeps on loading, loading, and doesn't show any data at all in the preview mode...

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try cutting the file down as much as possible. Once you have it working with a few lines, add more data.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vtsguerrero
Contributor

Just did it, and it worked perfectly as one single event!
Just had to cut the last line DATETIME_CONFIG = current wasn't allowing to load the stanza config, but once removed, it worked... Thanks a lot @richgalloway !

0 Karma

vtsguerrero
Contributor

Okay @richgalloway
I'm gonna re-index it here, asap, I'll post results, thanks a lot!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

How large are the .txt files? If they're too large then Splunk won't be able to treat them as a single event.

If you can provide some sample data (not a whole file) we can better help you.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vtsguerrero
Contributor

Each txt has an average of 400 lines and all of'em start with this " ~~CTRL AS~~: " pattern...

The data is similar to this ( I don't have all the source too 😞

~~CTRL AS~~:FG8WT09UX86UBB929376293762376M92738263TROKOM S28628ITT86327UPK           293862397263755

*>>>>>>>>>>>>>> LOGS UTDNAME: HUTHUTHYGS <<<<<<<<<<<<<<<<<<*

06.52.22 UTF8556 ---- THURSDAY,  04 DEC 2014 ----
06.52.22 UTF8556 HASP HHIAO WLM IFOP
06.52.22 UTF8556 0PLLOAOKWMO

And all the rest should be considered as one event even though there's a datetime present.

0 Karma

thomrs
Communicator

Did you set line_breaker in your props? By default Splunk will break on the time.

http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/Propsconf

0 Karma

vtsguerrero
Contributor

A REGEX is required to set this prop right?
I know that " ~ " would match the beggining. but not the complete start of the event...

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You probably don't want the complete start of the event. The matching string is not included in the event so you'd want to use the smallest string. '~~CTRL AS~~' should work.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...