Getting Data In

Need assist in setting json sourcetype

NanSplk01
Communicator

I have been trying to get the following sourcetype into Splunk for PI.  This whole stanza should go in as 1 event, but I've been unable to get the breakdown to multiple events from happening:

{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718196855107)\/",
"Message": "User query failed: Connection ID: 55, User: xxxxx, User ID: 1, Point ID: 247000, Type: summary, Start: 12-Jun-24 08:52:45, End: 12-Jun-24 08:54:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "sssssss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718196855.10703",
"Severity": "Warning"
},

I have even tried using the _json defaulted with Splunk, but it keeps breaking it into multiple lines/events.  Any suggestions would be helpful.  

Labels (2)
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@NanSplk01- I would suggest to assign a custom sourcetype, ex. my:pi:data

[my:pi:data]
SHOULD_LINEMERGE = false
LINE_BREAKER = [\}\[](,?[\s\n]*)\{[\s\n]*"Parameters"
TIME_PREFIX = Date\(
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %s%3N
TRUNCATE = 999999

 

This above props.conf config on the Indexers or Heavy Forwarder (first full Splunk instance) should work based on the data that you have provided.

 

I hope this helps!!!

0 Karma

NanSplk01
Communicator

Forgot to say, thank you everyone for the assist.

0 Karma

NanSplk01
Communicator

What I need is for the line that starts with Start: to be the break after line.

Start: 14-Jun-24 07:55:05, End: 14-Jun-24 07:56:35, Mode: 5, Status: [-11059] No Good Data For Calculation",

Break after the ", but since there are a few ",  and not only the ", how do I get it to break at that last comma?

0 Karma

NanSplk01
Communicator

NanSplk01_0-1718903493626.png

Unfortunately, as you can see, it's still splitting the two lines.

0 Karma

NanSplk01
Communicator

unfortunately it still breaks into two events and I wanted to receive only 1 event:

Time Event
1 6/14/24
7:56:39.168 AM
        "TimeStamp":  "\/Date(1718366199168)\/",
        "ID":  7082,
        "Parameters":  null,
    {
    },
Show all 6 lines
 
------------------------------------------------
2 6/14/24
7:56:39.013 AM
        "SplunkTime":  "1718366199.01303",
        "Source3":  null,
        "Source2":  null,
        "Source1":  null,
        "ProcessPIUser":  null,
Show all 15 lines
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Please provide multiple _raw events as raw, so community can help you write Line breaking configuration.

0 Karma

NanSplk01
Communicator

[
{
"Parameters": null,
"ID": 2185,
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122575.10669",
"Severity": "Warning"
}
]
"TimeStamp": "\/Date(1718122575106)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:45, End: 11-Jun-24 12:16:15, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122570.13029",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122570130)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:40, End: 11-Jun-24 12:16:10, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122565.16875",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122565168)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:35, End: 11-Jun-24 12:16:05, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122564.42661",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122564426)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:34, End: 11-Jun-24 12:16:04, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122555.14693",
"Severity": "Warning"
},
{
"Parameters": null,
"ID": 2185,
"TimeStamp": "\/Date(1718122555146)\/",
"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 11-Jun-24 12:14:25, End: 11-Jun-24 12:15:55, Mode: 5, Status: [-11059] No Good Data For Calculation",
"ProgramName": "piarchss",
"Category": null,
"OriginatingHost": null,
"OriginatingOSUser": null,
"OriginatingPIUser": null,
"ProcessID": 5300,
"Priority": 10,
"ProcessHost": null,
"ProcessOSUser": "SYSTEM",
"ProcessPIUser": null,
"Source1": "piarcset",
"Source2": "Historical",
"Source3": null,
"SplunkTime": "1718122550.12819",
"Severity": "Warning"
},

0 Karma

NanSplk01
Communicator

It's hard to see, but what is need is for the "Message": line to be the breaking line and for the "TimeStamp': line to be the first line of the whole event.

"Message": "User query failed: Connection ID: 55, User: piadmin, User ID: 1, Point ID: 247000, Type: summary, Start: 14-Jun-24 07:54:50, End: 14-Jun-24 07:56:20, Mode: 5, Status: [-11059] No Good Data For Calculation",-------event break here

"TimeStamp": "\/Date(1718366180157)\/",  ----event start here

In the example I sent it's hard to see the break after message and before Timestamp clearly because they look like one big line.

 

 

 

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...