Splunk Search

Index Time Fields Extraction on Summary Index

omerl
Path Finder

Hey

I’m trying to extract fields in index time on my summary index, in order to use ‘tstats’ command.

I used ‘collect’ to index the data, Setting sourcetype=_json, but I couldn’t make the fields extracted in index time.

I tested the command by using ‘makeresults’, and manually building the _raw field, but the fields were only extracted in search time (with KV_MODE=auto). Using KV_MODE=none and INDEXED_EXTRACTIONS=json, the fields were not indexed.

So I made a different test. I copied the generated _raw to a local file, and added it using the Upload File option. This time the fields were extracted at index time, as desired.

Is it possible to index fields using the collect command? Or am I doing something wrong?

Also, I’ve checked the Accelerated Data Model, but it didn’t fit my needs (due to non streaming commands).

0 Karma

didhavn
Engager

Hey @omerl

I know it's a long time ago but I just ran into the exact same issue. Did you find an explanation and/or solution?

Thanks! 

0 Karma

cpatadobe
Explorer

Have you looked at this article:
https://answers.splunk.com/answers/52753/extraction-of-an-indexed-field-in-summary-indexes.html
Is your situation very similar?

0 Karma

omerl
Path Finder

No, I think it’s more an issue with the ‘collect’ command, since adding data manually does extract the fields, but using collect the fields are not being extracted at index time, event though I’m setting sourcetype=_json

0 Karma

adonio
SplunkTrust
SplunkTrust

if you look at the raw data collect provides, it looks something like this:
08/12/2019 07:00:00 -0400, info_min_time=1565607600.000, info_max_time=1565695530.000, info_search_time=1565695530.437, <your_field_name_with_json_values>="<json_structure_values>"
this format is not a JSON format and therefore doesnt match the _json sourcetype.
you can create a unique props and transforms to build your index extractions. However, without fully knowing your use case, i assume it will be easier to complete your first search with a stats command and send the results in key=value pairs and write indexed extractions to that format
hope it helps

0 Karma

omerl
Path Finder

Yes, this might be a good solution. How should the additional fields be considered? Is ‘=‘ as a delimiter enough?
Thanks

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!