topic Re: Indexer ignoring Time_Format settings in forwarder props.conf in Getting Data In

Indexer ignoring Time_Format settings in forwarder props.conf

hastingsjay — Thu, 03 Oct 2013 12:20:26 GMT

I have events in plain text format like this:

"[Process Id:3952 Thread Id: 4152] 03/10/2013 12:44:58 GetComponentDetailsFromXMLLookup - sXMLCategory = General"

however, the event times are being indexed as:
10/03/2013 12:44:58.000 "[Process Id:3952 Thread Id: 4152] 03/10/2013 12:44:58 GetComponentDetailsFromXMLLookup - sXMLCategory = General"

The fowarder is the Windows machine.

The forwarder props.conf:

[splunkd]

EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<message>.+)

[splunk_web_service]

EXTRACT-useragent = userAgent=(?P<browser>[^ (]+)

[sourcetype:<mysourcetype>]

MAX_TIMESTAMP_LOOKAHEAD = 19

TIME_FORMAT = %d/%m/%Y %H:%M:%S

TIME_PREFIX = ]

and the forwarder inputs.conf:

[monitor://<mylogfilepath>]

disabled = false

host = <myhost>

sourcetype = <mysourcetype>

source = <mysource>

index = <myindex>

I have attempted to delete the local forwarder fishbucket directory and restart the forwarder to no avail. Is there some configuration missing?

Re: Indexer ignoring Time_Format settings in forwarder props.conf

Ayn — Thu, 03 Oct 2013 12:25:24 GMT

You should check the splunkd.log on your forwarder (it's in $SPLUNK_HOME\var\log\splunk\splunkd.log). I see a couple of issues with your config:

You have a stanza with [sourcetype:<mysourcetype>]. You don't need to specify sourcetype in the stanza, but if you do, it should be with TWO colons. So, either [sourcetype::mysourcetype] or just [mysourcetype].

Then, you have a TIME_PREFIX consisting of a sole ]. I'm not entirely sure how this is handled, but ] is a special character in regular expressions, which is what's used for TIME_PREFIX. If you want to match a literal ] you should escape it - TIME_PREFIX = \]

Finally it's useless to have field extractions on a forwarder. Field extraction happens at search-time, so all those definitions should go on the Splunk instance you're searching on only. It doesn't throw an error if you put that kind of stuff on a forwarder, there's just no reason for it to be there.

Re: Indexer ignoring Time_Format settings in forwarder props.conf

sdaniels — Thu, 03 Oct 2013 12:29:38 GMT

Extracting the time stamps is done on the indexer and not the forwarder (unless it's a heavy forwarder of course). Assuming you are using a heavy forwarder here? If it's the universal forwarder then you'll need to move those settings to the indexer for that sourcetype.

It looks like you've got the right settings on TIME_FORMAT except you have '/' in there and i'm guessing that will also be an issue. Try this:

TIME_FORMAT = %d %m %Y %H:%M:%S

Take a look at the docs for the different types of forwarders:

http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Typesofforwarders

Re: Indexer ignoring Time_Format settings in forwarder props.conf

sdaniels — Thu, 03 Oct 2013 12:30:56 GMT

Ayn always beats me to it...and he picked up two other issues...DOH!

Re: Indexer ignoring Time_Format settings in forwarder props.conf

hastingsjay — Thu, 03 Oct 2013 13:15:12 GMT

Couldn't see any errors being generated in the splunkd.log on the forwarder. To clarify we are using a universal forwarder.

Do I need to include the stanza entry (and parameters) in the props.conf on the indexer instance $SPLUNK_HOME/etc/system/local ?

Re: Indexer ignoring Time_Format settings in forwarder props.conf

hastingsjay — Thu, 03 Oct 2013 13:15:24 GMT

Couldn't see any errors being generated in the splunkd.log on the forwarder. To clarify we are using a universal forwarder.

Do I need to include the stanza entry (and parameters) in the props.conf on the indexer instance $SPLUNK_HOME/etc/system/local ?