using source for a field extractions

[kkalmbach]

I seem to be having some problems with extracting fields from the "source"

In by props.conf, I have:

[my_source]
SHOULD_LINEMERGE = false
LINE_BREAKER = (---------*)
REPORT-get_service = get_service

in transforms.conf, I have:

[get_service]
SOURCE_KEY=source
REGEX = ([^/]*)_sndmsg.out
FORMAT = service::"$1"

This only seems to be half working. In the search, if I do

index=myindex sourcetype=my_source service=abc

nothing comes back.

If I split it out into a separate search:

index=myindex sourcetype=my_source | search service=abc

then everything works fine.

Also if I prepend a wildcard, everything works fine, so this also works..

index=myindex sourcetype=my_source service=*abc

If I do an

eval l=len(service)

it always come back with what I expect. Everything looks fine when I print out the service field, I just can't put it in the inital search.

Any ideas would be greatly appreciated.

Thanks

-Kevin

[gkanapathy · Accepted solution]

That's because field searches don't look properly at fields that aren't extracted from the raw event text (vs the source field). You can deal with this by setting

[service]
INDEXED_VALUE=false

in fields.conf. But this is terrible and will make your searches run terribly terribly slow. It will work, but the price is way too high.

You can also modifying search term from

service="abc" 

to

(service="abc" source="*abc*")

This is inconvenient. But you could instead define a macro:

[service(1)]
args = sv
definition = ((service="$sv$" source="*$sv$*"))

and invoke it in your search string with:

index=myindex myterm1 `service(abc)`

instead.

This is something that should have a prettier solution in Splunk, but that's what we can do today. It would not be bad if you filed an Enhancement Request with Splunk Support to help get this in a future release.

[gkanapathy]

There is a big difference performance-wise, yes. Doing a second search tells Splunk to actually bring back all the data (in the initial search), then filter it. Specifying it all in the initial search makes use of the full-text index, which means retrieving only the data you specify in the first place. Note that the cost of the initial query is what typically dominates search time. If there a a lot of stuff in the index that does not match the "service", then there will be a significant difference.

[kkalmbach]

I think that makes sense. Thanks for you answer.
I will probably go the macro route.
Also, is there a big difference (performance wise)
in doing index="..." | search service="abc"
compared to putting it all in the same inital search?
Thanks again,