- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am quite new to Splunk. I'd be really grateful if you could point me towards the fix of the problem.
Environment : I have a splunk forwarder set up on another machine that forwards the logs in realtime to a central splunk server.
Below are the configurations ( on the forwarder machine ) and an example log file structure
Log File Structure Example
2013-10-07:04:00:26,x.y.z.w| x.y.z.w| a.b.c.d,11977EA89F5CC5,1381118419818,1381118426978,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.
2013-10-07:11:43:23,x.y.z.w| x.y.z.w| a.b.c.d,1209A270E6F5BF,1381146195657,1381146203190,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.
2013-10-07:13:27:12,x.y.z.w| x.y.z.w| a.b.c.d,EC3F8D2FFE67,1381152428564,1381152432796,62B55DF2C81A,No Facility,SUCCESS,Transaction completed successfully.
splunkforwarder/etc/system/local/inputs.conf
[default]
host = ip-x-y-z-w
[monitor:///home/jboss/jboss-as-7.1.1.Final/standalone/log/xyzlog/transactions.log]
sourcetype = XYZ_TRANSACTIONS
splunkforwarder/etc/system/local/props.conf
[XYZ_TRANSACTIONS]
SHOULD_LINEMERGE = False
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
#TIME_FORMAT=%Y-%m-%d %H:%M:%S # I tried both with and w/o ":" between the date and time part; doesn't work.
TRANSFORMS-Transaction_Timestamp=Transaction_Timestamp
splunkforwarder/etc/system/local/props.conf
[Transaction_Timestamp]
DEST_KEY = _time
REGEX = (\d{4}-\d{2}-\d{2}):(\d{2}:\d{2}:\d{2})
Problem
- Splunk parses the timestamp string
2013-10-07:14:06:30
as10/7/13 7:14:06.300 AM
.
What I tried:
- Initially, there was no
sourcetype
at all. I used to build dashboard (forms) around queries which used to start like "source=....". (This was where the problem started: I found out that Splunk was parsing the time wrongly) - I then used
sourcetype
specification ininputs.conf
file, and specified the corresponding properties for the source type inprops.conf
file. One on the properties I used wasTIME_FORMAT
. This is shown as commented above. (There was no change in the final timestamp recognition) - I tried to make a transform by replacing the ":" that appears after the date part with a space. I now added
transforms.conf
. Result: No change. I tried removing theTIME_FORMAT
fromprops.conf
. Result: No change. - I tried to add a Data Input (of a few lines of the same log file) from Web UI on my main splunk server. The props.conf file it generates does not have
TIME_FORMAT
and is also able to recognize time correctly. I tried setting the TIME_FORMAT of%Y-%m-%d %H:%M:%S
(which I have in props.conf), splunk server was able to recognize the timestamp from sample log file perfectly.
Splunk Server version: 5.0.4, build 172409
Splunk forwarder version: Splunk Universal Forwarder 5.0.4 (build 172409)
The only problem is that Splunk forwarder doesn't seem to use those properties I am specifying. Am I doing something wrong? Can this be improved? Is there a way I can fix this problem?
Please let me know if you need any further info about the environment/configurations/etc. Thanks.
@Sowing mentions about heavy/light/universal forwarders. From the answers mentioned here about finding the type of forwarder, I tried a search query like: "index=_internal source=*metrics.log group=tcpin_connections". I find that most of the results have "fwdType=uf". From this I understand that its a universal forwarder thats forwarding the above logs. Any pointers from here ?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
1) Your TIME_FORMAT needs the colon between the date and the time.
2) If the forwarder doesn't have a monitor:// declaration that sets a sourcetype, it will attempt to figure one out before sending the data to the indexer. It may not be choosing the correct name (XYZ_TRANSACTIONS), so when the data arrives, it's left to the default parsing rules. Adding the sourcetype key = value pair to your inputs.conf on the forwarder should be enough.
3) If the forwarder is a heavy forwarder (i.e., a full instance of Splunk), it's doing the parsing there, rather than waiting for the indexer to do it.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@kristian.kolb Thanks for the link 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sowings I will give that a shot. Thanks! 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since you indicate that the forwarders are universal, they are doing no parsing of the data. The TIME_FORMAT, etc, will be ignored on that host. These props.conf and transforms.conf entries should be on the indexer host.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
1) Your TIME_FORMAT needs the colon between the date and the time.
2) If the forwarder doesn't have a monitor:// declaration that sets a sourcetype, it will attempt to figure one out before sending the data to the indexer. It may not be choosing the correct name (XYZ_TRANSACTIONS), so when the data arrives, it's left to the default parsing rules. Adding the sourcetype key = value pair to your inputs.conf on the forwarder should be enough.
3) If the forwarder is a heavy forwarder (i.e., a full instance of Splunk), it's doing the parsing there, rather than waiting for the indexer to do it.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@sowings +1. Thanks a ton! :). Having props.conf in the central indexer fixed it.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
1) Use TIME_FORMAT; trying to set DEST_KEY of _time in transforms is not likely to work.
2) Inputs.conf is fine.
3) The configs for parsing (the props.conf from the forwarder) should be on the indexer.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi! Thanks for your answer.
About 1) Please check my config file again, I just made an update. I tried with and without ":" character.
About 2) I am already setting the sourcetype
in inputs.conf, can you please re-check, and tell me if its misplaced somehow?
About 3) How should I figure out if the forwarder is heavy/non-heavy? Also, if the forwarder is non-heavy, and the central server parses the files, does that mean I have to put the *.conf in the main server?
Thanks.