topic Re: .kvp file forwarding from the server but not appearing in Splunk Indexer in Dashboards & Visualizations

.kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Wed, 05 Jul 2017 16:10:26 GMT

Hi,

Suddenly data not appearing in the indexer and dash board but it is forwarding from the server. No configuration changed but earlier it is forwarding and stopped suddenly at end of the month at last hour.

Is there any Indexer configuration limit or what is the issue ?

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

richgalloway — Wed, 05 Jul 2017 17:20:25 GMT

Are you using SSL? That the forwarding stopped at the end of the month makes me think a certificate expired.

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Wed, 05 Jul 2017 17:26:18 GMT

Hi Richgalloway,

How we will check whether SSL expired or not in Splunk forwarder ? If it is expired, how we will get renewal this ? We install forwarder downloading from Splunk.com .

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

richgalloway — Wed, 05 Jul 2017 18:29:36 GMT

If you or your admin did not specifically set up SSL then you are not using it.
That means something else changed at the end of the month. Perhaps someone pushed out a firewall change before leaving on Friday.
Have you checked Splunk's logs? Maybe something is preventing it from writing to the index.

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Tue, 29 Sep 2020 14:46:06 GMT

Hi Richgalloway,

This issue in windows forwarder.

.kvp file forwarding if we place without timestamp in the file.
But same file if we place with timestamp it is forwarding but not appearing in Splunk.
But from the server all Splunk logs, metric logs, system event logs everything appearing. Only script data is not appearing in Splunk indexer.

appearing: number_of_Images=20
not appearing : timestamp="06/07/2017 13:26:05" number_of_Images=20

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

richgalloway — Thu, 06 Jul 2017 12:47:03 GMT

What are your props.conf settings for that file's sourcetype?

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

davebrooking — Thu, 06 Jul 2017 12:57:47 GMT

Mmmm, the events stopped indexing at the end of the month. I suspect there may be a timestamp mismatch and the events for 1st July were indexed with a date of 7th January, the events for 2nd July were indexed with a date of 7th February. Will these events magically start indexing tonight at midnight?

Dave

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Thu, 06 Jul 2017 13:36:16 GMT

Yes Dave, last month same issue. stopped at 31st may midnight 11pm and started at 6th june 12am. But this month stopped at 11pm june but started yet.

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

davebrooking — Thu, 06 Jul 2017 14:08:57 GMT

The documentation covers timestamp recognition. I suspect you will need to specifically define a TIME_FORMAT attribute for this data, so that Splunk can correctly interpret the timestamp in the event.

Dave

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Thu, 06 Jul 2017 17:25:44 GMT

Hi Dave,
Like below we are data forwarding and appearing in Splunk indexer.

data stopped at 11pm end of the month and started at 12am like below.

Feb - 1st no data , from 2nd we have data
Mar - 1st and 2nd no data , From 3rd we have data
April - 1st - 3rd no data , from 4th we have data
May - 1st - 4th no data , from 5th we have data
June - 1st - 5th no data, , from 6th we have data
July - 1st - 6th no data – hope we have data from this mid night 12am

Here how we can fix the issue, this is in Production Environment.

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

richgalloway — Thu, 06 Jul 2017 20:00:53 GMT

I think @davebrooking has hit the nail on the head. Splunk defaults to US date format (mm/dd/yyyy) and is easily confused by dd/mm/yyyy dates. The best fix, which really should be done by everyone for every sourcetype, is to specify a TIME_FORMAT attribute in props.conf.

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Tue, 29 Sep 2020 14:46:32 GMT

Hi richgalloway,

Is the TIME_FORMAT we need to change to mm/dd/yyyy or dd/mm/yyyy ?

my soucename is Import_Count-kvp, so need to place in the props.conf like below,

[Import_Count-kvp]
TIME_FORMAT=mm/dd/yyy

And in one server we don't have source name we are monitoring with Indexer name splunk, so for this props.conf like,

[splunk]
TIME_FORMAT=mm/dd/yyy

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

richgalloway — Fri, 07 Jul 2017 12:22:16 GMT

Use TIME_FORMAT = %d/%m/%Y, assuming your dates are in that format (day/month/year).

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Fri, 07 Jul 2017 16:21:32 GMT

time format is correct but It is not working richgalloway.

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Mon, 10 Jul 2017 13:09:41 GMT

Is this configuration need to be done at forwarder level or Indexer level ?

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

richgalloway — Mon, 10 Jul 2017 15:22:26 GMT

The indexer level.

Re: .kvp file forwarding from the server but not appearing in Splunk Indexer

RAYUDU_NARA — Tue, 25 Jul 2017 11:57:23 GMT

Hi Richgalloway,

It is working fine now. Thank you.

Regards,
Rayudu