topic Re: log file parsing on IDX in Getting Data In

log file parsing on IDX

secuc2r83 — Fri, 28 Feb 2020 13:46:26 GMT

Hello,
I just want to parse a log file. I try every solution found on forum but never work.
(Splunk 7.3.3)

Log:

<event action=....>
  <start>2020-02-22 12:49:21:596</start>
  <client .../>
  <sent ...></sent>
</event>
<event action=...>
  <start>2020-02-22 12:49:20:435</start>
  <client .../>
  <sent ...></sent>
</event>

What i want on Splunk SH:

_time               _raw
2020-02-22 13:49:21 <event action=...>
                      <start>2020-02-22 12:49:21:596</start>
                      <client .../>
                      <sent ...></sent>
                    </event>
2020-02-22 13:45:20 <event action=...>
                      <start>2020-02-22 12:49:21:596</start>
                      <client .../>
                      <sent ...></sent>
                    </event>

What i have on Splunk SH:

_time               _raw
2020-02-22 13:49:21 <event action=...>
2020-02-22 13:45:20 <event action=...>

inputs.conf on UF:

index = test
sourcetype = my_sourcetype
disabled = 0

4 props.conf tried on Indexer (based on forum's solution):

[my_sourcetype]
LINE_BREAKER = ([\r\n]+)\<event
SHOULD_LINEMERGE = false
DATETIME_CONFIG = NONE

[my_sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\s*\<event

[my_sourcetype]
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER = \</event\>

[my_sourcetype]
DATETIME_CONFIG = CURRENT
KV_MODE = xml
LINE_BREAKER = ([\r\n]+)(?=\s*\<event.*?\>)   
BREAK_ONLY_BEFORE_DATE = False
MUST_BREAK_AFTER = \</event\>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 0

props.conf on Search Head:

[my_sourcetype]
KV_MODE = xml

After each change, i reboot splunk on Idx and SH
Thanks for help
Regards

Re: log file parsing on IDX

richgalloway — Fri, 28 Feb 2020 17:07:14 GMT

Try these props on your indexers. Keep in mind changing props.conf will not affect data that is already indexed; only new data will be affected.

[my_sourcetype]
  KV_MODE = xml
  LINE_BREAKER = ([\r\n]+)\<event 
  TIME_PREFIX = start>
  TIME_FORMAT = %Y-%d-%n %H:%M:%S:%3N

Re: log file parsing on IDX

secuc2r83 — Fri, 28 Feb 2020 18:21:54 GMT

Hi richgalloway,

Yes i generate new events for each test

<event action=...>
  <start>2020-02-28 18:04:14:706</start>
  <end>2020-02-28 18:04:32:629</end>
  <client .../>
  <sent ...></sent>
 </event>

With your conf, same result, juste one line in SH:

 _time                      _raw
2020-02-28 18:04:37     <event action=...>

Here are all settings on indexer with btool, perhaps could see the problem:

# ./splunk btool props list test --debug
.../etc/slave-apps/app_test/local/props.conf [test]
.../etc/system/default/props.conf             ADD_EXTRA_TIME_FIELDS = True
.../etc/system/default/props.conf             ANNOTATE_PUNCT = True
.../etc/system/default/props.conf             AUTO_KV_JSON = true
.../etc/system/default/props.conf             BREAK_ONLY_BEFORE =
.../etc/system/default/props.conf             BREAK_ONLY_BEFORE_DATE = True
.../etc/system/default/props.conf             CHARSET = UTF-8
.../etc/system/default/props.conf             DATETIME_CONFIG = /etc/datetime.xml
.../etc/system/default/props.conf             DEPTH_LIMIT = 1000
.../etc/system/default/props.conf             HEADER_MODE =
.../etc/slave-apps/app_test/local/props.conf  KV_MODE = xml
.../etc/system/default/props.conf             LEARN_MODEL = true
.../etc/system/default/props.conf             LEARN_SOURCETYPE = true
.../etc/slave-apps/app_test/local/props.conf  LINE_BREAKER = ([\r\n]+)\<event
.../etc/system/default/props.conf             LINE_BREAKER_LOOKBEHIND = 100
.../etc/system/default/props.conf             MATCH_LIMIT = 100000
.../etc/system/default/props.conf             MAX_DAYS_AGO = 2000
.../etc/system/default/props.conf             MAX_DAYS_HENCE = 2
.../etc/system/default/props.conf             MAX_DIFF_SECS_AGO = 3600
.../etc/system/default/props.conf             MAX_DIFF_SECS_HENCE = 604800
.../etc/system/default/props.conf             MAX_EVENTS = 256
.../etc/system/default/props.conf             MAX_TIMESTAMP_LOOKAHEAD = 128
.../etc/system/default/props.conf             MUST_BREAK_AFTER =
.../etc/system/default/props.conf             MUST_NOT_BREAK_AFTER =
.../etc/system/default/props.conf             MUST_NOT_BREAK_BEFORE =
.../etc/system/default/props.conf             SEGMENTATION = indexing
.../etc/system/default/props.conf             SEGMENTATION-all = full
.../etc/system/default/props.conf             SEGMENTATION-inner = inner
.../etc/system/default/props.conf             SEGMENTATION-outer = outer
.../etc/system/default/props.conf             SEGMENTATION-raw = none
.../etc/system/default/props.conf             SEGMENTATION-standard = standard
.../etc/system/default/props.conf             SHOULD_LINEMERGE = True
.../etc/slave-apps/app_test/local/props.conf  TIME_FORMAT = %Y-%d-%n %H:%M:%S:%3N
.../etc/slave-apps/app_test/local/props.conf  TIME_PREFIX = start>
.../etc/system/default/props.conf             TRANSFORMS =
.../etc/system/default/props.conf             TRUNCATE = 10000
.../etc/system/default/props.conf             detect_trailing_nulls = false
.../etc/system/default/props.conf             maxDist = 100
.../etc/system/default/props.conf             priority =
.../etc/system/default/props.conf             sourcetype =