If I have a custom sourcetype with fields delimited by
,, the first field in the data is what I want to extract as the event time. What should be in the transforms.conf file for the FIELDS = ?
The data looks like:
05-Oct-2016 12:45:17, Jon, Sally, Sue,
How should I configure transforms.conf?
FIELDS = ????, Name1, Name2, Name3
The easiest way is to take an example in a file and follow the guided ingestion, in this way you can be sure to take correct timestamp and fields.
After you can use this sourcetype for your usual ingestion.
I have configured on my indexer in the
TIME_PREFIX = ^
SHOULD_LINEMERGE = False
pulldown_type = 1
DELIMS = ","
FIELDS = Name1,Name2,Name3
The best way to identify the correct timestamp format is to export an example of your logs in a file and try to ingest it using the menu function "Add data".
In this way you can create your sourcetype step by step modifying timestamp format as you need.
Every way your TIMESTAMP line to insert in props.conf is
TIME_FORMAT = %d-%b-%Y\s%H:%M:%S
The timestamp recognition is configured in the props.conf and not the transforms.conf (all transforms.conf settings are executed after Timestamp recognition). This this link for details on what all attributes that you need to setup for timestamp recognition. It's recommended that you explicitly specify these attributes for better indexing performance as automatic recognition can cause extra processing.
I generally configure these 3 attributes at the least. Here is what you can use for your situation
props.conf (on indexer/heavy forwarder)
[yoursourcetype] TIME_PREFIX = ^ TIME_FORMAT = %d-%b-%Y %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 20 ..other settings..
So, when you say noting gets indexed at all, do you mean you've an input.conf on your forwarder which is monitoring this CSV file and sending to Indexer but Indexer is dropping all events? (That doesn't seem right)