Getting Data In

Need assist in setting json sourcetype

NanSplk01
Communicator

I have been trying to get the following sourcetype into Splunk for PI.  This whole stanza should go in as 1 event, but I've been unable to get the breakdown to multiple events from happening:

{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718196855107)\/",
"Message": "User query failed: Connection ID: 55, User: xxxxx, User ID: 1, Point ID: 247000, Type: summary, Start: 12-Jun-24 08:52:45, End: 12-Jun-24 08:54:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "sssssss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718196855.10703",
"Severity": "Warning"
},

I have even tried using the _json defaulted with Splunk, but it keeps breaking it into multiple lines/events.  Any suggestions would be helpful.  

Labels (2)
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@NanSplk01- I would suggest to assign a custom sourcetype, ex. my:pi:data

[my:pi:data]
SHOULD_LINEMERGE = false
LINE_BREAKER = [\}\[](,?[\s\n]*)\{[\s\n]*"Parameters"
TIME_PREFIX = Date\(
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %s%3N
TRUNCATE = 999999

 

This above props.conf config on the Indexers or Heavy Forwarder (first full Splunk instance) should work based on the data that you have provided.

 

I hope this helps!!!

0 Karma

NanSplk01
Communicator

Forgot to say, thank you everyone for the assist.

0 Karma

NanSplk01
Communicator

What I need is for the line that starts with Start: to be the break after line.

Start: 14-Jun-24 07:55:05, End: 14-Jun-24 07:56:35, Mode: 5, Status: [-11059] No Good Data For Calculation",

Break after the ", but since there are a few ",  and not only the ", how do I get it to break at that last comma?

0 Karma

NanSplk01
Communicator

NanSplk01_0-1718903493626.png

Unfortunately, as you can see, it's still splitting the two lines.

0 Karma

NanSplk01
Communicator

unfortunately it still breaks into two events and I wanted to receive only 1 event:

Time Event
1 6/14/24
7:56:39.168 AM
        "TimeStamp":  "\/Date(1718366199168)\/",
        "ID":  7082,
        "Parameters":  null,
    {
    },
Show all 6 lines
 
------------------------------------------------
2 6/14/24
7:56:39.013 AM
        "SplunkTime":  "1718366199.01303",
        "Source3":  null,
        "Source2":  null,
        "Source1":  null,
        "ProcessPIUser":  null,
Show all 15 lines
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Please provide multiple _raw events as raw, so community can help you write Line breaking configuration.

0 Karma

NanSplk01
Communicator

[
{
"Parameters": null,
"ID": 2185,
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122575.10669",
"Severity": "Warning"
}
]
"TimeStamp": "\/Date(1718122575106)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:45, End: 11-Jun-24 12:16:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122570.13029",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122570130)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:40, End: 11-Jun-24 12:16:10, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122565.16875",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122565168)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:35, End: 11-Jun-24 12:16:05, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122564.42661",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122564426)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:34, End: 11-Jun-24 12:16:04, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122555.14693",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122555146)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:25, End: 11-Jun-24 12:15:55, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122550.12819",
"Severity": "Warning"
},

0 Karma

NanSplk01
Communicator

It's hard to see, but what is need is for the "Message": line to be the breaking line and for the "TimeStamp': line to be the first line of the whole event.

"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 14-Jun-24 07:54:50, End: 14-Jun-24 07:56:20, Mode: 5, Status: [-11059] No Good Data For Calculation",-------event break here

"TimeStamp": "\/Date(1718366180157)\/",  ----event start here

In the example I sent it's hard to see the break after message and before Timestamp clearly because they look like one big line.

 

 

 

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

What Is Splunk? Here’s What You Can Do with Splunk

Hey Splunk Community, we know you know Splunk. You likely leverage its unparalleled ability to ingest, index, ...

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...