topic Re: Need assist in setting json sourcetype in Getting Data In

Need assist in setting json sourcetype

NanSplk01 — Wed, 19 Jun 2024 16:41:59 GMT

I have been trying to get the following sourcetype into Splunk for PI. This whole stanza should go in as 1 event, but I've been unable to get the breakdown to multiple events from happening:

{ "Parameters": null, "ID": 2185, "TimeStamp": "\/Date(1718196855107)\/", "Message": "User query failed: Connection ID: 55, User: xxxxx, User ID: 1, Point ID: 247000, Type: summary, Start: 12-Jun-24 08:52:45, End: 12-Jun-24 08:54:15, Mode: 5, Status: [-11059] No Good Data For Calculation", "ProgramName": "sssssss", "Category": null, "OriginatingHost": null, "OriginatingOSUser": null, "OriginatingPIUser": null, "ProcessID": 5300, "Priority": 10, "ProcessHost": null, "ProcessOSUser": "SYSTEM", "ProcessPIUser": null, "Source1": "piarcset", "Source2": "Historical", "Source3": null, "SplunkTime": "1718196855.10703", "Severity": "Warning" },

I have even tried using the _json defaulted with Splunk, but it keeps breaking it into multiple lines/events. Any suggestions would be helpful.

Re: Need assist in setting json sourcetype

VatsalJagani — Tue, 18 Jun 2024 18:17:24 GMT

Please provide multiple _raw events as raw, so community can help you write Line breaking configuration.

Re: Need assist in setting json sourcetype

NanSplk01 — Tue, 18 Jun 2024 18:38:21 GMT

[
{
"Parameters": null,
"ID": 2185,
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122575.10669",
"Severity": "Warning"
}
]
"TimeStamp": "\/Date(1718122575106)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:45, End: 11-Jun-24 12:16:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122570.13029",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122570130)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:40, End: 11-Jun-24 12:16:10, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122565.16875",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122565168)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:35, End: 11-Jun-24 12:16:05, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122564.42661",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122564426)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:34, End: 11-Jun-24 12:16:04, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122555.14693",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122555146)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:25, End: 11-Jun-24 12:15:55, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122550.12819",
"Severity": "Warning"
},

Re: Need assist in setting json sourcetype

VatsalJagani — Wed, 19 Jun 2024 05:15:55 GMT

@NanSplk01- I would suggest to assign a custom sourcetype, ex. my:pi:data

[my:pi:data] SHOULD_LINEMERGE = false LINE_BREAKER = [\}\[](,?[\s\n]*)\{[\s\n]*"Parameters" TIME_PREFIX = Date\( MAX_TIMESTAMP_LOOKAHEAD = 128 TIME_FORMAT = %s%3N TRUNCATE = 999999

This above props.conf config on the Indexers or Heavy Forwarder (first full Splunk instance) should work based on the data that you have provided.

I hope this helps!!!

Re: Need assist in setting json sourcetype

NanSplk01 — Wed, 19 Jun 2024 14:45:17 GMT

unfortunately it still breaks into two events and I wanted to receive only 1 event:

Time Event

1 6/14/24

7:56:39.168 AM

"TimeStamp": "\/Date(1718366199168)\/",

"ID": 7082,

"Parameters": null,

{

Show all 6 lines

------------------------------------------------

2 6/14/24

7:56:39.013 AM

"SplunkTime": "1718366199.01303",

"Source3": null,

"Source2": null,

"Source1": null,

"ProcessPIUser": null,

Show all 15 lines

Re: Need assist in setting json sourcetype

NanSplk01 — Thu, 20 Jun 2024 17:12:30 GMT

Unfortunately, as you can see, it's still splitting the two lines.

Re: Need assist in setting json sourcetype

NanSplk01 — Fri, 21 Jun 2024 13:46:22 GMT

What I need is for the line that starts with Start: to be the break after line.

Start: 14-Jun-24 07:55:05, End: 14-Jun-24 07:56:35, Mode: 5, Status: [-11059] No Good Data For Calculation",

Break after the ", but since there are a few ", and not only the ", how do I get it to break at that last comma?

Re: Need assist in setting json sourcetype

NanSplk01 — Fri, 21 Jun 2024 13:46:46 GMT

Forgot to say, thank you everyone for the assist.

Re: Need assist in setting json sourcetype

NanSplk01 — Mon, 24 Jun 2024 15:31:14 GMT

It's hard to see, but what is need is for the "Message": line to be the breaking line and for the "TimeStamp': line to be the first line of the whole event.

"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 14-Jun-24 07:54:50, End: 14-Jun-24 07:56:20, Mode: 5, Status: [-11059] No Good Data For Calculation",-------event break here

"TimeStamp": "\/Date(1718366180157)\/", ----event start here

In the example I sent it's hard to see the break after message and before Timestamp clearly because they look like one big line.