Hello,
I have some issues with field extraction using props.conf and transforms.conf files. Sample data (3 sample events), my props.conf and transforms.conf files are provided below. I am getting events but field extraction is not responding (or getting events without extracted fields). Any recommendations would be highly appreciated. Thank you!
3 Sample Raw Events
tP1380158753BMFPG68006701522000000000000000000000000000000 Y1021210324010157DFTJ450757015#26I040
tP1380158753BMFPG68016702522000000000000000000000000000000 Y1023210324010156DFTJ450757015#25I040
tP1380158753BMFPG68026703522000000000000000000000000000000 Y1023210324010155DFTJ450757015#26I040
props.conf
[abcd:tests]
SHOULD_LINEMERGE = false
CHARSET=UTF-8
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK=true
TIME_PREFIX=[^"]{62}
TIME_FORMAT=%Y%m%d%H%M%S
REPORT-abcdCod6 = abcdCod6
transforms.conf
[abcdCod6]
REGEX =
(^)(?i)(?<a>.{1})(?i)(?<b>.{1})(?i)(?<e>.{10})(?i)(?<f>.{5})(?i)(?<g>(?:6))(?i)(?<h>.{8})(?i)(?<i>.{1})(?i)(?<j>.{1})(?i)(?<k>.{30})(?i)(?<l>.{1})(?i)(?<m>.{1})(?i)(?<n>.{2})(?i)(?<o>.{14})(?i)(?<p>.{4})(?i)(?<q>.{9})(?i)(?<r>.{1})(?i)(?<s>.{2})(?i)(?<t>.{1})(?i)(?<u>.{2})(?i)(?<v>.{1})
DEST_KEY = _raw
Hello @richgalloway ,
Thank you so much again. Yes, it has and marked as bold.
tP1380158753BMFPG68006701522000000000000000000000000000000 Y1021202224010157DFTJ450757015#26I040
tP1380158753BMFPG68016702522000000000000000000000000000000 Y1022202224010156DFTJ450757015#25I040
tP1380158753BMFPG68026703522000000000000000000000000000000 Y1023202224010155DFTJ450757015#26I040
The bolded text, Y10212022, is not even close to the format specified in TIME_FORMAT.
%Y%m%d%H%M%S needs 14 characters, but only 9 are bold. Based on the format string, I expected to see "2023" somewhere in the event, but that's not the case. What's more, the highlighted text is 3 fields (m,n,o) rather than the one or two fields we usually see for timestamps.
Make sure the sourcetype specified in inputs.conf matches the stanza name in props.conf.
Confirm the search heads were restarted since the .conf files were modified.
Hello @richgalloway
Thank you for your respond.
All looked good, as you mentioned. But still field extractions are not working, getting events under that sourcetype without extracted fields. Are there any other recommendations?
Do we have the complete raw events? I ask because the sample events do not appear to have timetamps in them even though the TIME_PREFIX and TIME_FORMAT settings are present. Without a complete event, it's impossible to confirm the accuracy of the regular expression.