Raw data is searchable until I specify a field... then nothing?

[rturk]

Greetings Splunkers (and Splunkettes),

I have a large amount of raw data in the default index of sourcetype "hsl_f5_web", which when I put the search of:

sourcetype="hsl_f5_web

in the default Search app gives me a steady stream of raw data in the main pane, as well as the fields it has managed to extract in the field picker pane. So far, so good...

When I mouseover one of the "Other interesting fields", such as the client_ip field and click on one of the values (ie. 1.2.3.4), sure enough the search query changes to:

sourcetype="hsl_f5_web" client_ip="1.2.3.4"

However, I am returned zero results, despite the Splunk itself telling me that it's in there?

Can some nice Splunker tell me what's going on here? I suspect permission issues, but when I can see the raw data, before specifying the actual field name,and when Splunk even sees the name-value pairs, I'm kinda scratching my head!

Confused,

Ryan 🙂

[sideview]

This can happen in at least one way that I can think of:

a) the app you are in has specified in its fields.conf that INDEXED_VALUE=true for the [clientip] field.

http://www.splunk.com/base/Documentation/latest/Admin/Fieldsconf

What this tells splunkd is that the full value of each clientip will be present in as a basic searchterm in the index. So whenever it sees a search for clientip="foo", it can speed up the search dramatically by throwing in an extra raw searchterm of "foo"

b) The problem arises then when the clientip values are NOT actually indexed as raw tokens in the index, or perhaps not consistently indexed as such. Generally this is when they're prefixed or suffixed with something unusual.

The search gkanapathy provided will help troubleshoot a lot.
You can also try other permutations like:

"1.2.3.4" | top clientip  

ie to make sure that it does actually yield a table where "1.2.3.4" is marked as a 'clientip' field.

[alexiri]

I'm having this problem as well, but I don't have a fields.conf file. My extraction is defined as follows in props.conf:

EXTRACT-server = /var/log/tsm/(?\w+)/dsmaccnt.log in source

The field seems to be extracted just fine, but this query doesn't return results:

index=tsm-accnt node=cviclr01fc tsmserver=tsm91

while this one does:

index=tsm-accnt node=cviclr01fc | search tsmserver=tsm91

Any ideas?

[gkanapathy · Accepted solution]

This can happen with odd field extractions. Can you tell me if, when you do the above, but then remove the field name, whether you still get results. e.g., if you originally search for:

sourcetype="hsl_f5_web"

And you click on an "interesting field" value to get:

sourcetype="hsl_f5_web" client_ip="1.2.3.4"

Can you then change it to:

sourcetype="hsl_f5_web" "1.2.3.4"

and let us know if you get any results that way?

[gkanapathy]

Well there is the problem. You ahve extractions defined based on eventtype::. That is what these. That's because eventtype is not known until an event is actually retrieved, and therefor it's not possible to use any such extractions to retrieve those events in the first place. I actually thought that eventtype field extractions would not work in current Splunk versions, but it looks like they still do. However, because of the behavior you see, they aren't really encouraged and I don't know if they're supported.

[rturk]

transforms.conf

#[hsl_f5_web-extractions]
#DELIMS = ","
#FIELDS = "timestamp","device_id","event_type","client_ip","client_port","server_ip","server_port","request_num","http_status","cs-bytes","sc-bytes","uri-scheme","http_response_time","http_server","http_method","uri","x-forwarded-for","http_user_agent","http_referrer"

Right so it's commented out... increasingly confused.

[rturk]

props.conf

#[hsl_f5_web] 
#REPORT-internet_gateway = hsl_f5_web-extractions

[eventtype::f5_ltm_http_resp]
REPORT-extrac = widgetco.f5.ltm.http.resp

[eventtype::f5_ltm_sssss_block]
REPORT-extrac = widgetco.f5.ltm.sssss.block

[eventtype::f5_asm_http_block]
REPORT-extrac = widgetco.f5.asm.http.block

[rturk]

I am grabbing a copy of the props.conf for you now. It should be noted that this issue with the search is not limited to the 'client_ip' field, but any field (i.e Searches with = do not return any results.)

[gkanapathy]

Another possible though unlikely explanation is that the client_ip field extraction is defined in an "eventtype" props.conf stanza, but I'm not even sure this is possible in 4.x

[gkanapathy]

Interesting. It would be helpful to see the full props.conf for the hs1_f5_web sourcetype. I wonder if there is also any sourcetype renaming or a conflicting field extraction (maybe in the host or source stanzas as well).

[rturk]

As a follow up to this, if I issue the following search, I get results:

sourcetype="hsl_f5_web" | search client_ip="1.2.3.4"

Very confusing... would still appreciate any assistance you can provide.

[rturk]

Searching sourcetype="hsl_f5_web" the results come in as you'd expect, with raw data coming out in the results pane, & the left hand pane populating with the extracted fields.

Clicking on an interesting field value client_ip="1.2.3.4" as above starts the scan through all 44 million results, however no results are returned, despite this value supposedly being present in ~50% of the events (search was allowed to run through all events). Interestingly, the left fields pane did not populate at all, despite field discovery being on.

Removing the client_ip= did return results.