Timestamps not recognised correctly

[SRIVATSAN_IYER]

I am quite new to Splunk. I'd be really grateful if you could point me towards the fix of the problem.

Environment : I have a splunk forwarder set up on another machine that forwards the logs in realtime to a central splunk server.

Below are the configurations ( on the forwarder machine ) and an example log file structure

---

Log File Structure Example

2013-10-07:04:00:26,x.y.z.w| x.y.z.w| a.b.c.d,11977EA89F5CC5,1381118419818,1381118426978,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.
2013-10-07:11:43:23,x.y.z.w| x.y.z.w| a.b.c.d,1209A270E6F5BF,1381146195657,1381146203190,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.
2013-10-07:13:27:12,x.y.z.w| x.y.z.w| a.b.c.d,EC3F8D2FFE67,1381152428564,1381152432796,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.

splunkforwarder/etc/system/local/inputs.conf

[default]
host = ip-x-y-z-w

[monitor:///home/jboss/jboss-as-7.1.1.Final/standalone/log/xyzlog/transactions.log]
sourcetype = XYZ_TRANSACTIONS

splunkforwarder/etc/system/local/props.conf

[XYZ_TRANSACTIONS]
SHOULD_LINEMERGE = False
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
#TIME_FORMAT=%Y-%m-%d %H:%M:%S        # I tried both with and w/o ":" between the date and time part; doesn't work.
TRANSFORMS-Transaction_Timestamp=Transaction_Timestamp

splunkforwarder/etc/system/local/props.conf

[Transaction_Timestamp]
DEST_KEY = _time
REGEX = (\d{4}-\d{2}-\d{2}):(\d{2}:\d{2}:\d{2})

---

Problem

• Splunk parses the timestamp string 2013-10-07:14:06:30 as 10/7/13 7:14:06.300 AM.

What I tried:

• Initially, there was no sourcetype at all. I used to build dashboard (forms) around queries which used to start like "source=....". (This was where the problem started: I found out that Splunk was parsing the time wrongly)
• I then used sourcetype specification in inputs.conf file, and specified the corresponding properties for the source type in props.conf file. One on the properties I used was TIME_FORMAT. This is shown as commented above. (There was no change in the final timestamp recognition)
• I tried to make a transform by replacing the ":" that appears after the date part with a space. I now added transforms.conf. Result: No change. I tried removing the TIME_FORMAT from props.conf. Result: No change.
• I tried to add a Data Input (of a few lines of the same log file) from Web UI on my main splunk server. The props.conf file it generates does not have TIME_FORMAT and is also able to recognize time correctly. I tried setting the TIME_FORMAT of %Y-%m-%d %H:%M:%S (which I have in props.conf), splunk server was able to recognize the timestamp from sample log file perfectly.

Splunk Server version: 5.0.4, build 172409
Splunk forwarder version: Splunk Universal Forwarder 5.0.4 (build 172409)

The only problem is that Splunk forwarder doesn't seem to use those properties I am specifying. Am I doing something wrong? Can this be improved? Is there a way I can fix this problem?

Please let me know if you need any further info about the environment/configurations/etc. Thanks.

---

@Sowing mentions about heavy/light/universal forwarders. From the answers mentioned here about finding the type of forwarder, I tried a search query like: "index=_internal source=*metrics.log group=tcpin_connections". I find that most of the results have "fwdType=uf". From this I understand that its a universal forwarder thats forwarding the above logs. Any pointers from here ?

[SRIVATSAN_IYER]

@kristian.kolb Thanks for the link 🙂

[kristian_kolb]

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

always useful.

[SRIVATSAN_IYER]

@Sowings I will give that a shot. Thanks! 🙂

[sowings]

Since you indicate that the forwarders are universal, they are doing no parsing of the data. The TIME_FORMAT, etc, will be ignored on that host. These props.conf and transforms.conf entries should be on the indexer host.

[sowings · Accepted solution]

1) Your TIME_FORMAT needs the colon between the date and the time.

2) If the forwarder doesn't have a monitor:// declaration that sets a sourcetype, it will attempt to figure one out before sending the data to the indexer. It may not be choosing the correct name (XYZ_TRANSACTIONS), so when the data arrives, it's left to the default parsing rules. Adding the sourcetype key = value pair to your inputs.conf on the forwarder should be enough.

3) If the forwarder is a heavy forwarder (i.e., a full instance of Splunk), it's doing the parsing there, rather than waiting for the indexer to do it.

[SRIVATSAN_IYER]

@sowings +1. Thanks a ton! :). Having props.conf in the central indexer fixed it.

[sowings]

1) Use TIME_FORMAT; trying to set DEST_KEY of _time in transforms is not likely to work.

2) Inputs.conf is fine.

3) The configs for parsing (the props.conf from the forwarder) should be on the indexer.

[SRIVATSAN_IYER]

Hi! Thanks for your answer.

About 1) Please check my config file again, I just made an update. I tried with and without ":" character.

About 2) I am already setting the sourcetype in inputs.conf, can you please re-check, and tell me if its misplaced somehow?

About 3) How should I figure out if the forwarder is heavy/non-heavy? Also, if the forwarder is non-heavy, and the central server parses the files, does that mean I have to put the *.conf in the main server?

Thanks.