How to configure props.conf to index txt files as one new event when text "~~CTRL AS~~:" appears, not by timestamps?

[vtsguerrero]

Hello everybody!

I could use some help with this project that I've been working with...
I have some .txt files which show timestamp in some lines like this " ---- FRIDAY, 05 DEC 2014 ---- "
But the point is, when I index it, it's counting every single datetime as new event, and it should consider the whole .txt as ONE EVENT.
The text I have in particular that defines this txt is unique is this:

~~CTRL AS~~:

Any idea how could I make a Regex for this to consider every time a " ~~CTRL AS~~: " is a new event, not based on the timestamps actually.
Thanks in adv!
Bst rgds!

[richgalloway · Accepted solution]

Start with these specs in the relevant props.conf stanza:

SHOULD_LINEMERGE = true
TRUNCATE = 0
MAX_EVENTS = 500
BREAK_ONLY_BEFORE = ~~CTRL AS~~
DATETIME_CONFIG = current

[vtsguerrero]

This props.conf should be placed inside the app folder right?
And I should re-index the data in the preview mode to see any changes...

[richgalloway]

Yes, app/local/props.conf. You must re-index.

I suggest using a test index until you've found the right settings. That makes it easier to clean up and keeps unusable events out of your regular indexes.

[vtsguerrero]

Still tryin' to re-index it, but when I applyin' this new stanza if keeps on loading, loading, and doesn't show any data at all in the preview mode...

[richgalloway]

Try cutting the file down as much as possible. Once you have it working with a few lines, add more data.

[vtsguerrero]

Just did it, and it worked perfectly as one single event!
Just had to cut the last line DATETIME_CONFIG = current wasn't allowing to load the stanza config, but once removed, it worked... Thanks a lot @richgalloway !

[vtsguerrero]

Okay @richgalloway
I'm gonna re-index it here, asap, I'll post results, thanks a lot!

[richgalloway]

How large are the .txt files? If they're too large then Splunk won't be able to treat them as a single event.

If you can provide some sample data (not a whole file) we can better help you.

[vtsguerrero]

Each txt has an average of 400 lines and all of'em start with this " ~~CTRL AS~~: " pattern...

The data is similar to this ( I don't have all the source too 😞

~~CTRL AS~~:FG8WT09UX86UBB929376293762376M92738263TROKOM S28628ITT86327UPK           293862397263755

*>>>>>>>>>>>>>> LOGS UTDNAME: HUTHUTHYGS <<<<<<<<<<<<<<<<<<*

06.52.22 UTF8556 ---- THURSDAY,  04 DEC 2014 ----
06.52.22 UTF8556 HASP HHIAO WLM IFOP
06.52.22 UTF8556 0PLLOAOKWMO

And all the rest should be considered as one event even though there's a datetime present.

[thomrs]

Did you set line_breaker in your props? By default Splunk will break on the time.

http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/Propsconf

[vtsguerrero]

A REGEX is required to set this prop right?
I know that " ~ " would match the beggining. but not the complete start of the event...

[richgalloway]

You probably don't want the complete start of the event. The matching string is not included in the event so you'd want to use the smallest string. '~~CTRL AS~~' should work.