Greetings Splunkers (and Splunkettes),
I have a large amount of raw data in the default index of sourcetype "hslf5web", which when I put the search of:
in the default Search app gives me a steady stream of raw data in the main pane, as well as the fields it has managed to extract in the field picker pane. So far, so good...
When I mouseover one of the "Other interesting fields", such as the
client_ip field and click on one of the values (ie. 220.127.116.11), sure enough the search query changes to:
However, I am returned zero results, despite the Splunk itself telling me that it's in there?
Can some nice Splunker tell me what's going on here? I suspect permission issues, but when I can see the raw data, before specifying the actual field name,and when Splunk even sees the name-value pairs, I'm kinda scratching my head!
This can happen with odd field extractions. Can you tell me if, when you do the above, but then remove the field name, whether you still get results. e.g., if you originally search for:
And you click on an "interesting field" value to get:
Can you then change it to:
and let us know if you get any results that way?
Searching sourcetype="hslf5web" the results come in as you'd expect, with raw data coming out in the results pane, & the left hand pane populating with the extracted fields.
Clicking on an interesting field value client_ip="18.104.22.168" as above starts the scan through all 44 million results, however no results are returned, despite this value supposedly being present in ~50% of the events (search was allowed to run through all events). Interestingly, the left fields pane did not populate at all, despite field discovery being on.
Removing the client_ip= did return results.
As a follow up to this, if I issue the following search, I get results:
sourcetype="hsl_f5_web" | search client_ip="22.214.171.124"
Very confusing... would still appreciate any assistance you can provide.
Interesting. It would be helpful to see the full props.conf for the hs1f5web sourcetype. I wonder if there is also any sourcetype renaming or a conflicting field extraction (maybe in the host or source stanzas as well).
Another possible though unlikely explanation is that the
client_ip field extraction is defined in an "eventtype" props.conf stanza, but I'm not even sure this is possible in 4.x
I am grabbing a copy of the props.conf for you now. It should be noted that this issue with the search is not limited to the 'client_ip' field, but any field (i.e Searches with
#[hsl_f5_web] #REPORT-internet_gateway = hsl_f5_web-extractions [eventtype::f5_ltm_http_resp] REPORT-extrac = widgetco.f5.ltm.http.resp [eventtype::f5_ltm_sssss_block] REPORT-extrac = widgetco.f5.ltm.sssss.block [eventtype::f5_asm_http_block] REPORT-extrac = widgetco.f5.asm.http.block
#[hsl_f5_web-extractions] #DELIMS = "," #FIELDS = "timestamp","device_id","event_type","client_ip","client_port","server_ip","server_port","request_num","http_status","cs-bytes","sc-bytes","uri-scheme","http_response_time","http_server","http_method","uri","x-forwarded-for","http_user_agent","http_referrer"
Right so it's commented out... increasingly confused.
Well there is the problem. You ahve extractions defined based on
eventtype::. That is what these. That's because eventtype is not known until an event is actually retrieved, and therefor it's not possible to use any such extractions to retrieve those events in the first place. I actually thought that eventtype field extractions would not work in current Splunk versions, but it looks like they still do. However, because of the behavior you see, they aren't really encouraged and I don't know if they're supported.