Getting Data In

How to troubleshoot why my Windows universal forwarders are not recognizing the correct timestamp for my data?

Communicator

Hi,

I have complex events in files forwarded from Windows hosts with Universal Forwarders.
These files are zip-compressed, and have "TRA" in filename.

They look similar to this:

20150422|20150721|grtghtyrt|teghtrhher(... some text)
20150427|20150630|grtghtyrt|teghtrhher(... some text)

Date of each event is THE SECOND column (first is for something else).
So for those 2 events, I expect _time to be 2015-07-21 00:00:00 and 2015-06-30 00:00:00

I made a simple app with props.conf: http://pastebin.com/LGCUNpPp

When I add input directly to Splunk, _time is correct.
When I forward data with Splunk Universal Forwarder, _time is set to modification date of those files, which is wrong.
Sourcetype is set correctly.

Why does Splunk Forwarder ignore my settings? How to debug this and what to do?

0 Karma
1 Solution

Path Finder

This may be helpful:

The time is extracted where the log data is parsed. This is on the indexer if you are using a lightweight forwarder, and on a forwarder if you are using a heavy forwarder. (Parsing the data is the essential difference between a light and heavy forwarding.)

Update: I wrote this which goes into a more thorough explanation.

Found here: https://answers.splunk.com/answers/612/timestamp-logic-config-on-forwarder-or-indexer.html

View solution in original post

Esteemed Legend

According to this:

http://docs.splunk.com/Documentation/Splunk/6.3.1/admin/Propsconf

You need to know this:

When setting a [<spec>] stanza, you can use the following regex-type syntax:
... recurses through directories until the match is met
    or equivalently, matches any number of characters.
*   matches anything but the path separator 0 or more times.
    The path separator is '/' on unix, or '\' on windows.
    Intended to match a partial or complete directory or filename.
|   is equivalent to 'or'
( ) are used to limit scope of |.

So instead of this:

[source::*TRA*]

Try this (or similar) instead:

[source::...TRA...]

Communicator

Thank you very much

0 Karma

Path Finder

This may be helpful:

The time is extracted where the log data is parsed. This is on the indexer if you are using a lightweight forwarder, and on a forwarder if you are using a heavy forwarder. (Parsing the data is the essential difference between a light and heavy forwarding.)

Update: I wrote this which goes into a more thorough explanation.

Found here: https://answers.splunk.com/answers/612/timestamp-logic-config-on-forwarder-or-indexer.html

View solution in original post

Communicator

I tried both changes at the same time - and don't know if changing '*' to '...' solved my problem, or moving props to indexers was enough.

Thank you guys!

0 Karma

Communicator

Thank you for the link to wiki. It helped me a lot!

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!