Getting Data In

Why am I getting sourcetype errors after trying to configure Splunk to parse json and proper timestamp recognition?

tomeumir
Engager

Hi,
I am trying Splunk and try to evaluate it as a tool for managing the logs of our in-house applications. I am uploading a file with thousands of lines like the following ones (2 sample lines):
{"time_stamp":"2014-10-06 18:00:00.025956","message_id":2459316836,"rq_id":8036007057,"rs_id":8036007059,"status":"OK","message_type":"OTA_HotelAvailRQ","host":"0.0.0.0","elapsed_time":155,"subsystem":"Player","connectorhost"":localhost","resstatus":"null","requestorid":"TRPS"}
{"time_stamp":"2014-10-06 18:00:00.041755","message_id":2459316838,"rq_id":8036007025,"rs_id":8036007010,"status":"OK","message_type":"OTA_HotelAvailRQ","host":"0.0.0.0","elapsed_time":31,"subsystem":"Player","connectorhost""localhost","resstatus":"null","requestorid":"TREP"}

But after uploading the file I cannot set the proper SourceType. I am choosing _json and setting
- TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N %Z
- TIME_FIELD= time_stamp

But I get the following message: No results found. Please change Sourcetype, adjust Sourcetype settings, or check your source file.

I will appreciate any help.

Thanks a lot in advanced.

Tom.

1 Solution

tomeumir
Engager

Hi there,
It seems that the time stamp field must be the first one. If so, then choosing just _json as source type all fields are parsed well automatically and _time is calculated based on my time stamp.
Rgds,
T.

View solution in original post

aweitzman
Motivator

First of all, this needs to be correct as you upload the file, so changing anything here will not change what's in Splunk unless you delete it out of Splunk and re-upload your file.

Second, your TIME_FORMAT differs from your actual input in the following three ways:

  1. You have a "T" separating the date from the time in the TIME_FORMAT, but it's not there in the data.
  2. You have a "%Z" for the timezone in your TIME_FORMAT, but it's not there in your input.
  3. You have a "%3N" for milliseconds in your TIME_FORMAT, but you appear to have microseconds there, which would mean you want to use "%6N" instead.

Third, you can't use TIME_FIELD here since that's used for search-time parsing, not index-time parsing, which is when the sourcetype is assigned. At the point that indexing happens, the JSON parsing has not taken place, and you don't have a time_stamp field yet. You need to use the TIME_PREFIX line in your stanza, like this:

TIME_PREFIX = time_stamp":

Read this for more details: http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Configuretimestamprecognition

What happens when you fix those?

0 Karma

tomeumir
Engager

Hi there,
It seems that the time stamp field must be the first one. If so, then choosing just _json as source type all fields are parsed well automatically and _time is calculated based on my time stamp.
Rgds,
T.

aweitzman
Motivator

Glad it works!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...