Basically, when I try to search for mf4 values on their own, index="sean-testing" mf4=w, the data found is zero or blank but if I use the full search with rex as seen below then click the mf4 value from the table view, the full event data will come up. I use regex101.com to drop the rex into the expression line and the data sample, one at a time, into the test sample box to test things out.
*Q: I'm thinking that I need to place this field extraction in a promps as a report or extract on the indexers? *
Any advice, examples or links to a better understanding would be great! Thank you. -Sean
full Search with rex
index="sean-testing" sourcetype="mfsource1"
| rex "\s{2}(?P<mf1>\d+)\s(?P<mf2>.{8})\s(?P<mf3>\S{10}){1}(?P<mf4>\S+)\s(?P<mf5>.+)[0-2][0-9]:[0-9][0-9]:[0-9][0-9]$"
| search mf4=W OR mf4=I
| table _time mf1 mf2 mf3 mf4 mf5
Event Data Sample 1
200401 07595444 17476 CARDS5 EXSFJM1083I EJM1: Using LWASP provider module "LWASP32.DLL", from API version 2009 07:59:54
Event Data Sample 2
200331 18250270 1764 CARDS1 CASHO00200I AB2P XA interface loaded. Name(AB2 for WINDOWS), Registration Mode(Dynamic) 18:25:02
Event Data Sample 3
200331 18250131 6508 CARDS3 CASZS50110W Failed to open port 21661 for TCPIPSERVICE ZTGIPP1 18:25:01
sourcetype
Basic stuff - break at everyline, no regex added
So it turns out this old blog post fixed my issue. https://www.splunk.com/en_us/blog/tips-and-tricks/cannot-search-based-on-an-extracted-field.html entitled "Cannot search based on an extracted field" There seems to be an issue when I try to create the extraction for MF3 and MF4 ...
I created a fields.conf file within the app folder on the search head ... (ie ...\apps\MF-Servers\local\fields.conf ... and added the stanza for the fields giving issues and it worked
========== The fields.conf file: ========
[mf3]
INDEXED_VALUE = False
[mf4]
I also found some specific field lengths so I expanded on my rex - "(^(?P<mf01>\d+\s\d+).{6}(?P<mf02>.{5}).(?P<mf02a>.{8}).(?P<mf3>.{9})(?<mf4>.).(?<mf5>.+)(?P<mf6>[0-2][0-9]:[0-9][0-9]:[0-9][0-9])$)"
props.conf
[mfsource1]
EXTRACT-mf = ^\S+\s\S+\s+(?P<mf1>\d+)\s(?P<mf2>\w+)\s+(?P<mf3>\S{10})(?P<mf4>\S)\s(?P<mf5>.+)\s+\S+$
It is better to extract with REGEX from the beginning.
Thanks for the input, i'll try the props.conf suggestion
| search mf_status=W OR mf_status=I ... this should read | search mf4=W OR mf4=I
I've updated the original post.