Splunk Search

How to handle staggered logs

jadengoho
Builder

alt textHi ,
Basically their server send logs one line at a time. When it came to Splunk it ingest automatically and not following the line breaker configuration. Out target is to line break the logs before "C:\Users\localserver>systeminfo".
Can Splunk wait for the line breaker to be visible before it linebreak ? Or what is the best way to handle this issue.

Example log:

C:\Users\localserver>systeminfo
    Host Name:                 localserver
    OS Name:                   Microsoft Windows 10 Enterprise
    OS Version:                10.0.18362 N/A Build 18362
    OS Manufacturer:           Microsoft Corporation
    OS Configuration:          Member Workstation
    OS Build Type:             Multiprocessor Free
    Registered Owner:          company
    Registered Organization:   OOO
    Original Install Date:     01/01/2020, 7:10:02 PM
    System Boot Time:          4/28/2020, 12:43:21 PM
    System Model:              HP Samplebook
    System Type:               x64-based PC
    Processor(s):              1 Processor(s) Installed.
    Windows Directory:         C:\WINDOWS
    System Directory:          C:\WINDOWS\system32
    Boot Device:               \Device\HarddiskVolume1

    C:\Users\localserver>
    Host Name:                 localserver
    OS Name:                   Microsoft Windows 10 Enterprise
    OS Version:                10.0.18362 N/A Build 18362
    OS Manufacturer:           Microsoft Corporation
    OS Configuration:          Member Workstation
    OS Build Type:             Multiprocessor Free
    Registered Owner:          company
    Registered Organization:   OOO
    Original Install Date:     01/01/2020, 7:10:02 PM
    System Boot Time:          4/28/2020, 12:43:21 PM
    System Model:              HP Samplebook
    System Type:               x64-based PC
    Processor(s):              1 Processor(s) Installed.
    Windows Directory:         C:\WINDOWS
    System Directory:          C:\WINDOWS\system32
    Boot Device:               \Device\HarddiskVolume1

Here's the situation that their server sending the logs, every 1minute it will sent the nextline.

C:\Users\localserver>systeminfo <After 1min it will send the next line>
Host Name:                 localserver <After 1min it will send the next line>
OS Name:                   Microsoft Windows 10 Enterprise <After 1min it will send the next line>
OS Version:                10.0.18362 N/A Build 18362 <After 1min it will send the next line>
OS Manufacturer:           Microsoft Corporation <After 1min it will send the next line>
OS Configuration:          Member Workstation <After 1min it will send the next line>

Props.conf

[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (C\:\\Users)
TRUNCATE = 8000

If i ingest the log as a bulk it will show the "GREEN BOX" in the picture whole and complete.
But in my case it's staggered and ingesting 1line per minute "RED BOX".

Tags (1)
0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="C:\Users\localserver>systeminfo
     Host Name:                 localserver
     OS Name:                   Microsoft Windows 10 Enterprise
     OS Version:                10.0.18362 N/A Build 18362
     OS Manufacturer:           Microsoft Corporation
     OS Configuration:          Member Workstation
     OS Build Type:             Multiprocessor Free
     Registered Owner:          company
     Registered Organization:   OOO
     Original Install Date:     01/01/2020, 7:10:02 PM
     System Boot Time:          4/28/2020, 12:43:21 PM
     System Model:              HP Samplebook
     System Type:               x64-based PC
     Processor(s):              1 Processor(s) Installed.
     Windows Directory:         C:\WINDOWS
     System Directory:          C:\WINDOWS\system32
     Boot Device:               \Device\HarddiskVolume1

C:\Users\localserver>
     Host Name:                 localserver
     OS Name:                   Microsoft Windows 10 Enterprise
     OS Version:                10.0.18362 N/A Build 18362
     OS Manufacturer:           Microsoft Corporation
     OS Configuration:          Member Workstation
     OS Build Type:             Multiprocessor Free
     Registered Owner:          company
     Registered Organization:   OOO
     Original Install Date:     01/01/2020, 7:10:02 PM
     System Boot Time:          4/28/2020, 12:43:21 PM
     System Model:              HP Samplebook
     System Type:               x64-based PC
     Processor(s):              1 Processor(s) Installed.
     Windows Directory:         C:\WINDOWS
     System Directory:          C:\WINDOWS\system32
     Boot Device:               \Device\HarddiskVolume1
 "
| rex mode=sed "s/(?ms)[\r\n]+^C:/#C/g"
| makemv delim="#" _raw
| stats count by _raw

props.conf

[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+^)C:
TRUNCATE = 0
0 Karma

jadengoho
Builder

Will this work even their server send data 1 line per minute, cause that's my problem?

0 Karma

to4kawa
Ultra Champion

The problem was that HF was sending more than one event at a time.
If the event breaks up, there's no problem.

0 Karma

jadengoho
Builder

The problem was that HF was sending more than one event at a time.
- How can i set the HF to not send the logs if it doesn't saw the linebreaker ?

Also the client want to see the logs as a complete not line by line..

0 Karma

to4kawa
Ultra Champion
0 Karma

jadengoho
Builder

I do know how to setup the props on heavy forwarder using deployment server.

The question is How can i make HF not send the logs if it doesn't saw the linebreaker?
OR how to make the HF wait until he saw the linebreaker?

0 Karma

jadengoho
Builder

@to4kawa i've tried your code and it shows similar output.
When i ingest the log as whole - it shows complete - please see GREEN BOX
but in my case it send 1line per minute shows line per line - please see RED BOX

0 Karma

PavelP
Motivator

can you post your props and inputs?

0 Karma

jadengoho
Builder

i'll update the question to include the current props.conf

0 Karma

PavelP
Motivator

are the events coming one in time (say 1 event/minute) or multiple are send at once?

please try:
[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)C:
TRUNCATE = 8000

0 Karma

jadengoho
Builder

hi @PavelP - it s 1 line per minute

0 Karma

PavelP
Motivator

Hi @jadengoho,

I've just checked it successfully with this configuration:

[sourcetype_name]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)C:
TRUNCATE = 8000

don't forget to debug/refresh or restart Splunk after you changed the configuration.

0 Karma

jadengoho
Builder

@PavelP - yes this works on bulk ingestion even logs 1second apart.
But in my case logs are being ingested 1minute apart.

I resolve the issue by using "inputs.conf" time_before_close

0 Karma

to4kawa
Ultra Champion

what a many slashes!

@jadengoho
what's your inputs.conf
I don't understand why commands displays and is logging.

0 Karma

PavelP
Motivator

as @to4kawa mentioned, too many backslashes - I've corrected, the simplified LINE_BREAKER with C: will work, I've just tested successfully with your data

0 Karma

jadengoho
Builder

hi @PavelP -the configuration works for "one time ingestion", please see the GREEN BOX in the image.
But when the staggered data came it doesn't follow the linebreaking - please see the RED BOX.

0 Karma

jadengoho
Builder

we don't have inputs. their server is sending logs to our HF .

0 Karma
Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...