topic Re: How to handle staggered logs in Splunk Search

How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 08:09:12 GMT

Hi ,
Basically their server send logs one line at a time. When it came to Splunk it ingest automatically and not following the line breaker configuration. Out target is to line break the logs before "C:\Users\localserver>systeminfo".
Can Splunk wait for the line breaker to be visible before it linebreak ? Or what is the best way to handle this issue.

Example log:

C:\Users\localserver>systeminfo
    Host Name:                 localserver
    OS Name:                   Microsoft Windows 10 Enterprise
    OS Version:                10.0.18362 N/A Build 18362
    OS Manufacturer:           Microsoft Corporation
    OS Configuration:          Member Workstation
    OS Build Type:             Multiprocessor Free
    Registered Owner:          company
    Registered Organization:   OOO
    Original Install Date:     01/01/2020, 7:10:02 PM
    System Boot Time:          4/28/2020, 12:43:21 PM
    System Model:              HP Samplebook
    System Type:               x64-based PC
    Processor(s):              1 Processor(s) Installed.
    Windows Directory:         C:\WINDOWS
    System Directory:          C:\WINDOWS\system32
    Boot Device:               \Device\HarddiskVolume1

    C:\Users\localserver>
    Host Name:                 localserver
    OS Name:                   Microsoft Windows 10 Enterprise
    OS Version:                10.0.18362 N/A Build 18362
    OS Manufacturer:           Microsoft Corporation
    OS Configuration:          Member Workstation
    OS Build Type:             Multiprocessor Free
    Registered Owner:          company
    Registered Organization:   OOO
    Original Install Date:     01/01/2020, 7:10:02 PM
    System Boot Time:          4/28/2020, 12:43:21 PM
    System Model:              HP Samplebook
    System Type:               x64-based PC
    Processor(s):              1 Processor(s) Installed.
    Windows Directory:         C:\WINDOWS
    System Directory:          C:\WINDOWS\system32
    Boot Device:               \Device\HarddiskVolume1

Here's the situation that their server sending the logs, every 1minute it will sent the nextline.

C:\Users\localserver>systeminfo <After 1min it will send the next line>
Host Name:                 localserver <After 1min it will send the next line>
OS Name:                   Microsoft Windows 10 Enterprise <After 1min it will send the next line>
OS Version:                10.0.18362 N/A Build 18362 <After 1min it will send the next line>
OS Manufacturer:           Microsoft Corporation <After 1min it will send the next line>
OS Configuration:          Member Workstation <After 1min it will send the next line>

Props.conf

[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (C\:\\Users)
TRUNCATE = 8000

If i ingest the log as a bulk it will show the "GREEN BOX" in the picture whole and complete.
But in my case it's staggered and ingesting 1line per minute "RED BOX".

Re: How to handle staggered logs

PavelP — Tue, 28 Apr 2020 08:18:51 GMT

can you post your props and inputs?

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 08:39:08 GMT

i'll update the question to include the current props.conf

Re: How to handle staggered logs

to4kawa — Tue, 28 Apr 2020 08:47:13 GMT

| makeresults 
| eval _raw="C:\Users\localserver>systeminfo
     Host Name:                 localserver
     OS Name:                   Microsoft Windows 10 Enterprise
     OS Version:                10.0.18362 N/A Build 18362
     OS Manufacturer:           Microsoft Corporation
     OS Configuration:          Member Workstation
     OS Build Type:             Multiprocessor Free
     Registered Owner:          company
     Registered Organization:   OOO
     Original Install Date:     01/01/2020, 7:10:02 PM
     System Boot Time:          4/28/2020, 12:43:21 PM
     System Model:              HP Samplebook
     System Type:               x64-based PC
     Processor(s):              1 Processor(s) Installed.
     Windows Directory:         C:\WINDOWS
     System Directory:          C:\WINDOWS\system32
     Boot Device:               \Device\HarddiskVolume1

C:\Users\localserver>
     Host Name:                 localserver
     OS Name:                   Microsoft Windows 10 Enterprise
     OS Version:                10.0.18362 N/A Build 18362
     OS Manufacturer:           Microsoft Corporation
     OS Configuration:          Member Workstation
     OS Build Type:             Multiprocessor Free
     Registered Owner:          company
     Registered Organization:   OOO
     Original Install Date:     01/01/2020, 7:10:02 PM
     System Boot Time:          4/28/2020, 12:43:21 PM
     System Model:              HP Samplebook
     System Type:               x64-based PC
     Processor(s):              1 Processor(s) Installed.
     Windows Directory:         C:\WINDOWS
     System Directory:          C:\WINDOWS\system32
     Boot Device:               \Device\HarddiskVolume1
 "
| rex mode=sed "s/(?ms)[\r\n]+^C:/#C/g"
| makemv delim="#" _raw
| stats count by _raw

props.conf

[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+^)C:
TRUNCATE = 0

Re: How to handle staggered logs

PavelP — Wed, 30 Sep 2020 05:14:34 GMT

are the events coming one in time (say 1 event/minute) or multiple are send at once?

please try:
[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)C:
TRUNCATE = 8000

Re: How to handle staggered logs

to4kawa — Tue, 28 Apr 2020 09:09:20 GMT

what a many slashes!

@jadengoho
what's your inputs.conf
I don't understand why commands displays and is logging.

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 09:21:07 GMT

we don't have inputs. their server is sending logs to our HF .

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 09:23:09 GMT

hi @PavelP - it s 1 line per minute

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 09:24:29 GMT

Will this work even their server send data 1 line per minute, cause that's my problem?

Re: How to handle staggered logs

to4kawa — Tue, 28 Apr 2020 09:29:10 GMT

The problem was that HF was sending more than one event at a time.
If the event breaks up, there's no problem.

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 09:32:26 GMT

The problem was that HF was sending more than one event at a time.
- How can i set the HF to not send the logs if it doesn't saw the linebreaker ?

Also the client want to see the logs as a complete not line by line..

Re: How to handle staggered logs

to4kawa — Tue, 28 Apr 2020 09:34:06 GMT

https://answers.splunk.com/answers/458237/setting-up-propsconf-at-the-heavy-forwarders.html
set props.conf on HF

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 09:42:32 GMT

I do know how to setup the props on heavy forwarder using deployment server.

The question is How can i make HF not send the logs if it doesn't saw the linebreaker?
OR how to make the HF wait until he saw the linebreaker?

Re: How to handle staggered logs

PavelP — Tue, 28 Apr 2020 09:51:17 GMT

as @to4kawa mentioned, too many backslashes - I've corrected, the simplified LINE_BREAKER with C: will work, I've just tested successfully with your data

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 10:16:31 GMT

hi @PavelP -the configuration works for "one time ingestion", please see the GREEN BOX in the image.
But when the staggered data came it doesn't follow the linebreaking - please see the RED BOX.

Re: How to handle staggered logs

jadengoho — Tue, 28 Apr 2020 11:01:27 GMT

@to4kawa i've tried your code and it shows similar output.
When i ingest the log as whole - it shows complete - please see GREEN BOX
but in my case it send 1line per minute shows line per line - please see RED BOX

Re: How to handle staggered logs

PavelP — Thu, 30 Apr 2020 10:18:18 GMT

Hi @jadengoho,

I've just checked it successfully with this configuration:

[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)C:
TRUNCATE = 8000

don't forget to debug/refresh or restart Splunk after you changed the configuration.

Re: How to handle staggered logs

jadengoho — Wed, 30 Sep 2020 05:16:00 GMT

@PavelP - yes this works on bulk ingestion even logs 1second apart.
But in my case logs are being ingested 1minute apart.

I resolve the issue by using "inputs.conf" time_before_close