Getting Data In

Need assist in setting json sourcetype

NanSplk01
Path Finder

I have been trying to get the following sourcetype into Splunk for PI.  This whole stanza should go in as 1 event, but I've been unable to get the breakdown to multiple events from happening:

{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718196855107)\/",
"Message": "User query failed: Connection ID: 55, User: xxxxx, User ID: 1, Point ID: 247000, Type: summary, Start: 12-Jun-24 08:52:45, End: 12-Jun-24 08:54:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "sssssss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718196855.10703",
"Severity": "Warning"
},

I have even tried using the _json defaulted with Splunk, but it keeps breaking it into multiple lines/events.  Any suggestions would be helpful.  

Labels (2)
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@NanSplk01- I would suggest to assign a custom sourcetype, ex. my:pi:data

[my:pi:data]
SHOULD_LINEMERGE = false
LINE_BREAKER = [\}\[](,?[\s\n]*)\{[\s\n]*"Parameters"
TIME_PREFIX = Date\(
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %s%3N
TRUNCATE = 999999

 

This above props.conf config on the Indexers or Heavy Forwarder (first full Splunk instance) should work based on the data that you have provided.

 

I hope this helps!!!

0 Karma

NanSplk01
Path Finder

Forgot to say, thank you everyone for the assist.

0 Karma

NanSplk01
Path Finder

What I need is for the line that starts with Start: to be the break after line.

Start: 14-Jun-24 07:55:05, End: 14-Jun-24 07:56:35, Mode: 5, Status: [-11059] No Good Data For Calculation",

Break after the ", but since there are a few ",  and not only the ", how do I get it to break at that last comma?

0 Karma

NanSplk01
Path Finder

NanSplk01_0-1718903493626.png

Unfortunately, as you can see, it's still splitting the two lines.

0 Karma

NanSplk01
Path Finder

unfortunately it still breaks into two events and I wanted to receive only 1 event:

Time Event
1 6/14/24
7:56:39.168 AM
        "TimeStamp":  "\/Date(1718366199168)\/",
        "ID":  7082,
        "Parameters":  null,
    {
    },
Show all 6 lines
 
------------------------------------------------
2 6/14/24
7:56:39.013 AM
        "SplunkTime":  "1718366199.01303",
        "Source3":  null,
        "Source2":  null,
        "Source1":  null,
        "ProcessPIUser":  null,
Show all 15 lines
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Please provide multiple _raw events as raw, so community can help you write Line breaking configuration.

0 Karma

NanSplk01
Path Finder

[
{
"Parameters": null,
"ID": 2185,
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122575.10669",
"Severity": "Warning"
}
]
"TimeStamp": "\/Date(1718122575106)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:45, End: 11-Jun-24 12:16:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122570.13029",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122570130)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:40, End: 11-Jun-24 12:16:10, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122565.16875",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122565168)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:35, End: 11-Jun-24 12:16:05, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122564.42661",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122564426)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:34, End: 11-Jun-24 12:16:04, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122555.14693",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122555146)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:25, End: 11-Jun-24 12:15:55, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122550.12819",
"Severity": "Warning"
},

0 Karma

NanSplk01
Path Finder

It's hard to see, but what is need is for the "Message": line to be the breaking line and for the "TimeStamp': line to be the first line of the whole event.

"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 14-Jun-24 07:54:50, End: 14-Jun-24 07:56:20, Mode: 5, Status: [-11059] No Good Data For Calculation",-------event break here

"TimeStamp": "\/Date(1718366180157)\/",  ----event start here

In the example I sent it's hard to see the break after message and before Timestamp clearly because they look like one big line.

 

 

 

0 Karma
Get Updates on the Splunk Community!

Let’s Talk Terraform

If you’re beyond the first-weeks-of-a-startup stage, chances are your application’s architecture is pretty ...

Cloud Platform | Customer Change Announcement: Email Notification is Available For ...

The Notification Team is migrating our email service provider. As the rollout progresses, Splunk has enabled ...

Save the Date: GovSummit Returns Wednesday, December 11th!

Hey there, Splunk Community! Exciting news: Splunk’s GovSummit 2024 is returning to Washington, D.C. on ...