Splunk Search
Highlighted

Raw data is searchable until I specify a field... then nothing?

Builder

Greetings Splunkers (and Splunkettes),

I have a large amount of raw data in the default index of sourcetype "hslf5web", which when I put the search of:

sourcetype="hsl_f5_web

in the default Search app gives me a steady stream of raw data in the main pane, as well as the fields it has managed to extract in the field picker pane. So far, so good...

When I mouseover one of the "Other interesting fields", such as the client_ip field and click on one of the values (ie. 1.2.3.4), sure enough the search query changes to:

sourcetype="hsl_f5_web" client_ip="1.2.3.4"

However, I am returned zero results, despite the Splunk itself telling me that it's in there?

Can some nice Splunker tell me what's going on here? I suspect permission issues, but when I can see the raw data, before specifying the actual field name,and when Splunk even sees the name-value pairs, I'm kinda scratching my head!

Confused,

Ryan 🙂

0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Splunk Employee
Splunk Employee

This can happen with odd field extractions. Can you tell me if, when you do the above, but then remove the field name, whether you still get results. e.g., if you originally search for:

sourcetype="hsl_f5_web"

And you click on an "interesting field" value to get:

sourcetype="hsl_f5_web" client_ip="1.2.3.4"

Can you then change it to:

sourcetype="hsl_f5_web" "1.2.3.4"

and let us know if you get any results that way?

View solution in original post

Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Builder

Searching sourcetype="hslf5web" the results come in as you'd expect, with raw data coming out in the results pane, & the left hand pane populating with the extracted fields.

Clicking on an interesting field value client_ip="1.2.3.4" as above starts the scan through all 44 million results, however no results are returned, despite this value supposedly being present in ~50% of the events (search was allowed to run through all events). Interestingly, the left fields pane did not populate at all, despite field discovery being on.

Removing the client_ip= did return results.

0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Builder

As a follow up to this, if I issue the following search, I get results:

sourcetype="hsl_f5_web" | search client_ip="1.2.3.4"

Very confusing... would still appreciate any assistance you can provide.

0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Splunk Employee
Splunk Employee

Interesting. It would be helpful to see the full props.conf for the hs1f5web sourcetype. I wonder if there is also any sourcetype renaming or a conflicting field extraction (maybe in the host or source stanzas as well).

0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Splunk Employee
Splunk Employee

Another possible though unlikely explanation is that the client_ip field extraction is defined in an "eventtype" props.conf stanza, but I'm not even sure this is possible in 4.x

0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Builder

I am grabbing a copy of the props.conf for you now. It should be noted that this issue with the search is not limited to the 'client_ip' field, but any field (i.e Searches with = do not return any results.)

0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Builder

props.conf

#[hsl_f5_web] 
#REPORT-internet_gateway = hsl_f5_web-extractions

[eventtype::f5_ltm_http_resp]
REPORT-extrac = widgetco.f5.ltm.http.resp

[eventtype::f5_ltm_sssss_block]
REPORT-extrac = widgetco.f5.ltm.sssss.block

[eventtype::f5_asm_http_block]
REPORT-extrac = widgetco.f5.asm.http.block
0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Builder

transforms.conf

#[hsl_f5_web-extractions]
#DELIMS = ","
#FIELDS = "timestamp","device_id","event_type","client_ip","client_port","server_ip","server_port","request_num","http_status","cs-bytes","sc-bytes","uri-scheme","http_response_time","http_server","http_method","uri","x-forwarded-for","http_user_agent","http_referrer"

Right so it's commented out... increasingly confused.

0 Karma
Highlighted

Re: Raw data is searchable until I specify a field... then nothing?

Splunk Employee
Splunk Employee

Well there is the problem. You ahve extractions defined based on eventtype::. That is what these. That's because eventtype is not known until an event is actually retrieved, and therefor it's not possible to use any such extractions to retrieve those events in the first place. I actually thought that eventtype field extractions would not work in current Splunk versions, but it looks like they still do. However, because of the behavior you see, they aren't really encouraged and I don't know if they're supported.

0 Karma