Getting Data In

Can you suggest a format for a source type event break that doesn't truncate the following JSON logs?

pdantuuri0411
Explorer

I have a JSON log that is getting truncated because of the event break pattern in the source type. I cloned the source type to create a similar one with different event break pattern so that the log doesn't get truncated. My current event break is \d\d?:\d\d:\d\d.

The log I have is

"USER_STAR_BADGE" : "160",
"USER_TYPE_CD" : "INSPECTOR",
"USER_EMAIL" : "atp@sgifn.com",
"SECUR_ROLE_ID" : "1",
"USER_STATUS_CD" : "A",
"USER_STATUS_DATE" : "2012-12-27 11:23:20.0",
"CREATE_TIMESTAMP" : "2012-12-27 11:23:20.0",
"CREATE_USERID" : "32845",
"UPDATE_TIMESTAMP" : "2016-02-26 12:33:42.0",
"UPDATE_USERID" : "186"
The event is breaking before "USER_STATUS_DATE" : "2012-12-27 11:23:20.0", because of the event break pattern. Please suggest a format for event break so that the log doesn't get truncated.

Tags (2)
0 Karma

Rob2520
Communicator

Not sure if this is what you are asking for. Every event breaks at "USER_STAR_BADGE".

alt text

0 Karma

pdantuuri0411
Explorer

Thank for the reply Rob. I am sorry to confuse you with the lack of detail in the post. I have a log file where 8 lines in the middle are getting truncated. I want to set my source type configuration so they donot get truncated. Here is the original log file

"AppData" : {
"user_info" : {
"USERID" : "",
"USER_CHICAGO_ID" : "",
"USER_PASSWORD" : "*****",
"USER_LAST_NAME" : "",
"USER_FIRST_NAME" : "",
"DEPT_ID" : "",
"USER_STAR_BADGE" : "",
"USER_TYPE_CD" : "",
"USER_EMAIL" : "",
"SECUR_ROLE_ID" : "",
"USER_STATUS_CD" : "",
"USER_STATUS_DATE" : "",
"CREATE_TIMESTAMP" : "",
"CREATE_USERID" : "",
"UPDATE_TIMESTAMP" : "2018-06-27 09:39:13.0",
"UPDATE_USERID" : ""
},
"image_info" : {
"image_key" : "359764084248580-img15_jpg-1539965048401",
"image_len" : 903988,
"create_time" : 1539965048401,
"location" : {

Here is what splunk is extracting

},
"AppData" : {
"user_info" : {
"USERID" : "607",
"USER_CHICAGO_ID" : "",
"USER_PASSWORD" : "",
"USER_LAST_NAME" : "",
"USER_FIRST_NAME" : "",
"DEPT_ID" : "13",
"USER_STAR_BADGE" : "",
"USER_TYPE_CD" : "",
"USER_EMAIL" : "",
"SECUR_ROLE_ID" : "",
"USER_STATUS_CD" : "",
"create_time" : ,
"location" : {

If you see the lines between "USER_STATUS_CD" : "", and "create_time" : , are missing. I don't want those lines are missing. I want Splunk to extract those lines too.

0 Karma

Rob2520
Communicator

I don't think you shared the complete log event. The one you shared is not fully formed JSON. I would check with removing the event breaker and see if i get the whole event.

0 Karma

sudosplunk
Motivator

I see there are 2-3 timestamps in the event. What are the event boundaries that you'd like to break on?

0 Karma

pdantuuri0411
Explorer

I do not want to break it at all. Is that possible? I want the whole event to be logged.

0 Karma

Vijeta
Influencer

probably its breaking at date timestamp settings. What is the configuration for date timestamp for this event, also you can change DATE TIMESTAMP to current if it does not impact you.

0 Karma

pdantuuri0411
Explorer

It is breaking at "USER_STATUS_DATE" : "2012-12-27 11:23:20.0". I get logs before that line. I was suggested to change the event break value in the source type option. I am not sure what to change it to.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...