Getting Data In

Need assist in setting json sourcetype

NanSplk01
Communicator

I have been trying to get the following sourcetype into Splunk for PI.  This whole stanza should go in as 1 event, but I've been unable to get the breakdown to multiple events from happening:

{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718196855107)\/",
"Message": "User query failed: Connection ID: 55, User: xxxxx, User ID: 1, Point ID: 247000, Type: summary, Start: 12-Jun-24 08:52:45, End: 12-Jun-24 08:54:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "sssssss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718196855.10703",
"Severity": "Warning"
},

I have even tried using the _json defaulted with Splunk, but it keeps breaking it into multiple lines/events.  Any suggestions would be helpful.  

Labels (2)
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@NanSplk01- I would suggest to assign a custom sourcetype, ex. my:pi:data

[my:pi:data]
SHOULD_LINEMERGE = false
LINE_BREAKER = [\}\[](,?[\s\n]*)\{[\s\n]*"Parameters"
TIME_PREFIX = Date\(
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %s%3N
TRUNCATE = 999999

 

This above props.conf config on the Indexers or Heavy Forwarder (first full Splunk instance) should work based on the data that you have provided.

 

I hope this helps!!!

0 Karma

NanSplk01
Communicator

Forgot to say, thank you everyone for the assist.

0 Karma

NanSplk01
Communicator

What I need is for the line that starts with Start: to be the break after line.

Start: 14-Jun-24 07:55:05, End: 14-Jun-24 07:56:35, Mode: 5, Status: [-11059] No Good Data For Calculation",

Break after the ", but since there are a few ",  and not only the ", how do I get it to break at that last comma?

0 Karma

NanSplk01
Communicator

NanSplk01_0-1718903493626.png

Unfortunately, as you can see, it's still splitting the two lines.

0 Karma

NanSplk01
Communicator

unfortunately it still breaks into two events and I wanted to receive only 1 event:

Time Event
1 6/14/24
7:56:39.168 AM
        "TimeStamp":  "\/Date(1718366199168)\/",
        "ID":  7082,
        "Parameters":  null,
    {
    },
Show all 6 lines
 
------------------------------------------------
2 6/14/24
7:56:39.013 AM
        "SplunkTime":  "1718366199.01303",
        "Source3":  null,
        "Source2":  null,
        "Source1":  null,
        "ProcessPIUser":  null,
Show all 15 lines
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Please provide multiple _raw events as raw, so community can help you write Line breaking configuration.

0 Karma

NanSplk01
Communicator

[
{
"Parameters": null,
"ID": 2185,
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122575.10669",
"Severity": "Warning"
}
]
"TimeStamp": "\/Date(1718122575106)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:45, End: 11-Jun-24 12:16:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122570.13029",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122570130)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:40, End: 11-Jun-24 12:16:10, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122565.16875",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122565168)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:35, End: 11-Jun-24 12:16:05, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122564.42661",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122564426)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:34, End: 11-Jun-24 12:16:04, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122555.14693",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122555146)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:25, End: 11-Jun-24 12:15:55, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122550.12819",
"Severity": "Warning"
},

0 Karma

NanSplk01
Communicator

It's hard to see, but what is need is for the "Message": line to be the breaking line and for the "TimeStamp': line to be the first line of the whole event.

"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 14-Jun-24 07:54:50, End: 14-Jun-24 07:56:20, Mode: 5, Status: [-11059] No Good Data For Calculation",-------event break here

"TimeStamp": "\/Date(1718366180157)\/",  ----event start here

In the example I sent it's hard to see the break after message and before Timestamp clearly because they look like one big line.

 

 

 

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Data Persistence in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. What happens if the OpenTelemetry collector ...

Introducing Splunk 10.0: Smarter, Faster, and More Powerful Than Ever

Now On Demand Whether you're managing complex deployments or looking to future-proof your data ...

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...