Getting Data In

Why are my props.conf configurations not merging lines into one event in Splunk 5?

Path Finder

Hello,

I have a problem with merging events: I search in this forum's posts and documentation and tried a lot of combinations, but never worked!

My config: Test environment = Splunk v5 on a single machine (indexer and search head are on the same machine)

My log:
alt text

My problem: Splunk treats each line as an event
What I want: Event just for line which have "...New SMTP ICID...", so with the 6 lines above, I should have 2 events
alt text

PROPS.CONF tested:

Tried with different LINE_BREAKER (All tested on regex101 with success)

    [iron_log]
    TIME_PREFIX = ^<\d\d>
    TIME_FORMAT = %b %d %H:%M:%S
    MAX_TIMESTAMP_LOOKAHEAD = 15
    LINE_BREAKER = ^.*New SMTP ICID.*
    #LINE_BREAKER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    #LINE_BREAKER = ([\r\n]+)<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    SHOULD_LINEMERGE = false
    TRUNCATE = 999999 

Try with SHOULD_LINEMERGE and MUST_BREAK_AFTER

    [iron_log]
    MUST_BREAK_AFTER = ^.*New
    #MUST_BREAK_AFTER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    BREAK_ONLY_BEFORE_DATE = false
    MAX_TIMESTAMP_LOOKAHEAD = 15
    TIME_PREFIX = ^<\d\d>

Tried without SHOULD_LINEMERGE

    [iron_log]
    SHOULD_LINEMERGE = false
    MUST_BREAK_AFTER = ^.*New
    #MUST_BREAK_AFTER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    BREAK_ONLY_BEFORE_DATE = false
    MAX_TIMESTAMP_LOOKAHEAD = 15
    TIME_PREFIX = ^<\d\d>

I always have one event per line. (I restarted Splunk for each modification and there is no error during the boot check)

If someone has an idea, it would be great!

Thanks by advance

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Unrelated to the actual question: Splunk 5 is quite old, consider upgrading to 6.3 - many great things await.

The props.conf setting you're looking for should be BREAK_ONLY_BEFORE = New SMTP ICID.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Unrelated to the actual question: Splunk 5 is quite old, consider upgrading to 6.3 - many great things await.

The props.conf setting you're looking for should be BREAK_ONLY_BEFORE = New SMTP ICID.

View solution in original post

0 Karma

Path Finder

Hi Martin,
The problem is solved by your last sentence.
I was sure that when you restart, it always reindex....
(Then all my props's change never be applied because restart splunk didn't reindex!)
I just change my log file and launch a search to see the same result as yours.

Thanks a lot for your time Martin
Regards

0 Karma

SplunkTrust
SplunkTrust

It'd be terrible if Splunk reindexed Petabytes of data every restart for some of the larger installs...

0 Karma

SplunkTrust
SplunkTrust

Setting only BREAK_ONLY_BEFORE = New SMTP ICID seems to work fine for me:

alt text

Make sure you configure this on the correct machine (any heavy forwarder involved?), there's no other configuration involved (props.conf on the source::, host::?), and you actually reindex that data after the restart.

Path Finder

Hi Martin,

Thanks for your help.
- For SplunkV6.3 => I understand but for now we can't improve to V6 then i have to find a solution with V5
- I did a basic config and try your solution but the same (6lines, 6 events):

indexes.conf
[iron_idx]
homePath = $SPLUNK_DB/iron_idx/db
coldPath = $SPLUNK_DB/iron_idx/colddb
thawedPath = $SPLUNK_DB/iron_idx/thaweddb

inputs.conf
[monitor:///var/log/iron.log]
index=iron_idx
sourcetype=iron_log

props.conf (I try several solutions but still the same...)
[iron_log]
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = New SMTP ICID

[iron_log]
BREAK_ONLY_BEFORE = ^.+New

[iron_log]
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = ^.+New
BREAK_ONLY_BEFORE_DATE = false

If you have other idea you're welcome
thks

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!