topic Re: Fields search doesn't work when filtering internal fields in Getting Data In

Fields search doesn't work when filtering internal fields

sansay — Tue, 25 Sep 2012 00:17:16 GMT

Here is the search string:

MissingUserData exchange rate | rex "ID :(?.+)" | fields ORDERID

This returns 8 records on a specific time frame. But it returns the whole log string.

If I add " | fields - _*"

to only get the values of ORDERID in the results like this:

MissingUserData exchange rate | rex "ID :(?.+)" | fields ORDERID | fields - _*

I get nothing.

That behavior just baffles me. Anyone has an idea why that might be?

Re: Fields search doesn't work when filtering internal fields

Ayn — Tue, 25 Sep 2012 06:02:30 GMT

"The whole log string" is stored in the field _raw, so when you do | fields - _* you're removing that field as well, which is why you're not seeing any events - you should only be seeing your values for ORDERID.

Re: Fields search doesn't work when filtering internal fields

sansay — Tue, 25 Sep 2012 16:23:21 GMT

Here is a sample log that shows up, instead of just giving me the values for ORDERID:
2012-02-06T03:07:54.739-08:00 lvp-p2-nppaysys01 BibitServer[9081]: WARNING: LogCategory [MissingUserData] No exchange rate for Order ID :2646042148

Note that I can get only ORDERID values displayed with:

MissingUserData exchange rate | rex "ID :(?<ORDERID>.+)" | fields ORDERID | table ORDERID

But then I can't use that to feed it to another search, as it fails.

Re: Fields search doesn't work when filtering internal fields

Runals — Tue, 25 Sep 2012 17:02:27 GMT

I'm just a noob but I think the results of the search will be "ORDERID=2646042148." If you need to pass just "2646042148" you might need to rename the ORDERID field to query and return just that. I think I read this on post Ayn responded to once. Your query (subquery actually) would be something like:

[ MissingUserData exchange rate | rex "ID :(?<ORDERID>.+)" | rename ORDERID as query | fields query ]

I did that for a query where I was looking up IPs in a lookup table and throwing them against some very unstructured data. My query was as follows where the column name in test2.csv was IP. This does impact search performance and like I said I'm a noob so in the interest of full disclosure YMMV

index=netflow [| inputlookup test2.csv | rename IP as query | fields query ]

Note the following link: http://docs.splunk.com/Documentation/Splunk/4.3.4/User/HowSubsearchesWork

Re: Fields search doesn't work when filtering internal fields

sansay — Tue, 25 Sep 2012 18:26:41 GMT

Thank you Runals. You hit it on the nail with your answer.