Getting Data In

Timestamps not recognised correctly

SRIVATSAN_IYER
Explorer

I am quite new to Splunk. I'd be really grateful if you could point me towards the fix of the problem.

Environment : I have a splunk forwarder set up on another machine that forwards the logs in realtime to a central splunk server.

Below are the configurations ( on the forwarder machine ) and an example log file structure


Log File Structure Example

2013-10-07:04:00:26,x.y.z.w| x.y.z.w| a.b.c.d,11977EA89F5CC5,1381118419818,1381118426978,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.
2013-10-07:11:43:23,x.y.z.w| x.y.z.w| a.b.c.d,1209A270E6F5BF,1381146195657,1381146203190,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.
2013-10-07:13:27:12,x.y.z.w| x.y.z.w| a.b.c.d,EC3F8D2FFE67,1381152428564,1381152432796,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.

splunkforwarder/etc/system/local/inputs.conf

[default]
host = ip-x-y-z-w

[monitor:///home/jboss/jboss-as-7.1.1.Final/standalone/log/xyzlog/transactions.log]
sourcetype = XYZ_TRANSACTIONS

splunkforwarder/etc/system/local/props.conf

[XYZ_TRANSACTIONS]
SHOULD_LINEMERGE = False
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
#TIME_FORMAT=%Y-%m-%d %H:%M:%S        # I tried both with and w/o ":" between the date and time part; doesn't work.
TRANSFORMS-Transaction_Timestamp=Transaction_Timestamp

splunkforwarder/etc/system/local/props.conf

[Transaction_Timestamp]
DEST_KEY = _time
REGEX = (\d{4}-\d{2}-\d{2}):(\d{2}:\d{2}:\d{2})

Problem

  • Splunk parses the timestamp string 2013-10-07:14:06:30 as 10/7/13 7:14:06.300 AM.

What I tried:

  • Initially, there was no sourcetype at all. I used to build dashboard (forms) around queries which used to start like "source=....". (This was where the problem started: I found out that Splunk was parsing the time wrongly)
  • I then used sourcetype specification in inputs.conf file, and specified the corresponding properties for the source type in props.conf file. One on the properties I used was TIME_FORMAT. This is shown as commented above. (There was no change in the final timestamp recognition)
  • I tried to make a transform by replacing the ":" that appears after the date part with a space. I now added transforms.conf. Result: No change. I tried removing the TIME_FORMAT from props.conf. Result: No change.
  • I tried to add a Data Input (of a few lines of the same log file) from Web UI on my main splunk server. The props.conf file it generates does not have TIME_FORMAT and is also able to recognize time correctly. I tried setting the TIME_FORMAT of %Y-%m-%d %H:%M:%S (which I have in props.conf), splunk server was able to recognize the timestamp from sample log file perfectly.

Splunk Server version: 5.0.4, build 172409
Splunk forwarder version: Splunk Universal Forwarder 5.0.4 (build 172409)

The only problem is that Splunk forwarder doesn't seem to use those properties I am specifying. Am I doing something wrong? Can this be improved? Is there a way I can fix this problem?

Please let me know if you need any further info about the environment/configurations/etc. Thanks.


@Sowing mentions about heavy/light/universal forwarders. From the answers mentioned here about finding the type of forwarder, I tried a search query like: "index=_internal source=*metrics.log group=tcpin_connections". I find that most of the results have "fwdType=uf". From this I understand that its a universal forwarder thats forwarding the above logs. Any pointers from here ?

Tags (2)
1 Solution

sowings
Splunk Employee
Splunk Employee

1) Your TIME_FORMAT needs the colon between the date and the time.

2) If the forwarder doesn't have a monitor:// declaration that sets a sourcetype, it will attempt to figure one out before sending the data to the indexer. It may not be choosing the correct name (XYZ_TRANSACTIONS), so when the data arrives, it's left to the default parsing rules. Adding the sourcetype key = value pair to your inputs.conf on the forwarder should be enough.

3) If the forwarder is a heavy forwarder (i.e., a full instance of Splunk), it's doing the parsing there, rather than waiting for the indexer to do it.

View solution in original post

SRIVATSAN_IYER
Explorer

@kristian.kolb Thanks for the link 🙂

0 Karma

kristian_kolb
Ultra Champion

SRIVATSAN_IYER
Explorer

@Sowings I will give that a shot. Thanks! 🙂

0 Karma

sowings
Splunk Employee
Splunk Employee

Since you indicate that the forwarders are universal, they are doing no parsing of the data. The TIME_FORMAT, etc, will be ignored on that host. These props.conf and transforms.conf entries should be on the indexer host.

0 Karma

sowings
Splunk Employee
Splunk Employee

1) Your TIME_FORMAT needs the colon between the date and the time.

2) If the forwarder doesn't have a monitor:// declaration that sets a sourcetype, it will attempt to figure one out before sending the data to the indexer. It may not be choosing the correct name (XYZ_TRANSACTIONS), so when the data arrives, it's left to the default parsing rules. Adding the sourcetype key = value pair to your inputs.conf on the forwarder should be enough.

3) If the forwarder is a heavy forwarder (i.e., a full instance of Splunk), it's doing the parsing there, rather than waiting for the indexer to do it.

SRIVATSAN_IYER
Explorer

@sowings +1. Thanks a ton! :). Having props.conf in the central indexer fixed it.

0 Karma

sowings
Splunk Employee
Splunk Employee

1) Use TIME_FORMAT; trying to set DEST_KEY of _time in transforms is not likely to work.

2) Inputs.conf is fine.

3) The configs for parsing (the props.conf from the forwarder) should be on the indexer.

SRIVATSAN_IYER
Explorer

Hi! Thanks for your answer.

About 1) Please check my config file again, I just made an update. I tried with and without ":" character.

About 2) I am already setting the sourcetype in inputs.conf, can you please re-check, and tell me if its misplaced somehow?

About 3) How should I figure out if the forwarder is heavy/non-heavy? Also, if the forwarder is non-heavy, and the central server parses the files, does that mean I have to put the *.conf in the main server?

Thanks.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...