I have a problem when indexing the events through a forwarder. The forwarder is listening to a log file with first event (first record) as header event. But I could see the events are not coming as is into the indexer. It is getting sorted on datetime.
REQID~INPCD~RQTPCD~WKFWNM~INSTNM~NODENM~PRTNCT~STRTTS~ENDTS~DURMSCT~VOLCT~REVOLCT~APPNM~RULENO~NMSCORE~CONMSCORE~ADDRSCORE~PHSCORE~SSNSCORE~YOBSCORE~MATCHIND 4472276843203022016174103072 ~RT ~RT ~node01~RUNINSTANCE4~PCIS_ABCDPB_01~1~2016-03-02 17:41:02.771~2016-03-02 17:41:03.072 ~1~1~1~1 4477576843003022016175856617 ~RT ~RT ~node01~RUNINSTANCE2~PCIS_ABCDPB_01~1~2016-03-02 17:58:56.022~2016-03-02 17:58:56.617 ~1~1~1~1 ~NA~NA~NA~NA~NA~NA~NA~N
4477576843003022016175856617 ~RT ~RT ~node01~RUNINSTANCE2~PCIS_ABCDPB_01~1~2016-03-02 17:58:56.022~2016-03-02 17:58:56.617 ~1~1~1~1 ~NA~NA~NA~NA~NA~NA~NA~N 4472276843203022016174103072 ~RT ~RT ~node01~RUNINSTANCE4~PCIS_ABCDPB_01~1~2016-03-02 17:41:02.771~2016-03-02 17:41:03.072 ~1~1~1~1
Props.conf in indexer
NO_BINARY_CHECK=1 SHOULD_LINEMERGE=false TIME_FORMAT=%Y-%m-%d TRUNCATE=300000 pulldown_type = 1
Please advise the correct configuration to get the event as is from the log file.
Splunk Indexes data as soon as they are received and they might not know when your file writing finishes. These events are persisted to disk in their original arrival order. However, in search results events are retrieved in inverse time order to show the latest events first.
You can sort events during search to see events in arrival order.
The first record in the file is header record. In the search head,I'm trying to extract the fields by creating the props.conf and transforms.conf in $splunk_HOME/etc/apps/appname/local . But I couldn't see the fields getting extracted ,Is it because of order I'm not able to extract the fields ? Please advise.
[12345678_abcd_Analytic_logs] SHOULD_LINEMERGE=false KV_MODE = none CHECK_FOR_HEADER = TRUE REPORT-ext = detail_abcd_fields pulldown_type = 1
[detail_abcd_fields] DELIMS = "~" FIELDS = REQ_ID,INP_CHAN_CD,REQ_TYP_CD,INFA_WKFLOW_NM,INFA_INSTNC_NM,INFA_NODE_NM,INFA_PRTN_CT,PRCS_STRT_TS,PRCS_END_TS,PRCS_DUR__CT,INP_VOL_CT,REFER_VOL_CT,APPL_NM
The order of indexing should not affect the extraction process. From the above configuration, it more looks like an indexer time extraction and for that you have to place the configuration on indexers.
If you just want search time extractions, try with splunk web
If you still want to do
search time field extractions using config files, refer to http://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/Createandmaintainsearch-timefieldextract...
Also remember, whenever you do configurations using files, then you need to restart splunk and with the web, it's not required.