topic Re: collect index="based on values" in Knowledge Management

How to collect index="based on values"

fklink — Mon, 13 Jun 2022 17:34:24 GMT

Hi everybody,

is it possible to create several summary index within one search?

Example:
"Index A" has a field "OS" with values "Windows", "Linux"...

Is there a way to tell splunk to loop something like: index=A | collect index="OS" (where OS is the field-value)

So that each result based on field OS will be stored in (allready created) index "Windows", "Linux", ....

Many thanks in advance.

Re: collect index="based on values"

gcusello — Fri, 06 Sep 2019 14:17:13 GMT

Hi fklink,
why do you need to use only one search?
Bye.
Giuseppe

Re: collect index="based on values"

fklink — Fri, 06 Sep 2019 14:27:34 GMT

Hi Guiseppe,

the values Windows and Linux are exemplary.

In fact, there are about 150 different values that vary.

My goal is to use one query to fill the 150 different indexes (automatically)

Re: collect index="based on values"

gcusello — Fri, 06 Sep 2019 14:42:55 GMT

Hi fklink,
are you sure about this?
an elevated quantity of indexes isn't a best practice!
You should analyze your needs and define the correct number of indexes thinking to the main reason to have different indexes:

different retention periods,
different access rights.

In addition you can take in consideration also the quantity of logs ingested (e.g. it isn't a good practice to have in the same index logs from large flows with logs from little flows).
But anyway 150 indexes are very many indexes!
Don't think to indexes as DB tables, Splunk is different!

Bye.
Giuseppe

Re: collect index="based on values"

DalJeanis — Fri, 06 Sep 2019 20:51:15 GMT

Here's how I've written that before

search that generates events with some field rectype

| rename COMMENT as "send foo to index foo"
| appendpipe [ 
    | where rectype="foo" 
    | collect index="foo" testmode=f other parameters
    | where false()
    ]

| rename COMMENT as "send bar to index bar"
 | appendpipe [ 
    | where rectype="bar" 
    | collect index="bar" testmode=f other parameters
    | where false()
    ]

| rename COMMENT as "repeat for each destination index"


| rename COMMENT as "This last one doesn't go to an index but it lets you see as the search results what would have done so for all types"
| collect index="goingnowhere" testmode=t other parameters

Re: collect index="based on values"

fklink — Sat, 07 Sep 2019 10:16:05 GMT

Hi DalJeanis,

many thanks for your help.

Since the values within "rectype" can differ, I'm looking for a solution that fills the field variably.

someting like: | collect index="$value_of_rectype$"

Re: collect index="based on values"

fklink — Sat, 07 Sep 2019 10:20:28 GMT

Hi Guiseppe,

thanks for your help.

Because of permissions i have to create 150 index 😞

Maybe i will find an other solution.....

Re: collect index="based on values"

woodcock — Mon, 09 Sep 2019 14:22:23 GMT

Like this:

search that generates events with some field rectype
| _SI_Name_{index} = "IgnoreMe"
| foreach _SI_Name_* [
| appendpipe [ 
   where rectype="$MATCHSTR$"
| collect index="$MATCHSTR$" testmode=f other parameters
| where false()
] ]
| rename COMMENT as "Anything left at this point did not go into an index, which may be an error"

Re: collect index="based on values"

scombs — Sat, 11 Jun 2022 18:33:17 GMT

The following is a subset of a larger solution I came up with to anonymize and re-index some previously ingested events while maintaining their original index, sourcetype, source, host, and _time fields.

Create a set of index-time props and transforms that:

Separately assign each event to their own index.
Restore default summary sourcetype of "stash".
Remove the pre-pended index name from each event.

Then pre-pend a target index name to each event record before calling |collect with a sourcetype matching the stanza from your props.conf (and which is reverted back to "stash" to not count against license).

#Index time props.conf [per_event_summary_index] TRANSFORMS-set_summary_index = reset_index, reset_stash_sourcetype, remove_prepended_field #Index time transforms.conf [reset_index] DEST_KEY = _MetaData:Index REGEX = (.*?), FORMAT = $1 [reset_stash_sourcetype] DEST_KEY = MetaData:Sourcetype FORMAT = stash [remove_prepended_field] INGEST_EVAL = _raw=replace(_raw,".*?,","") ```Send events to separate summary indexes based on OS.``` index=A |eval _raw=OS.","._raw |collect sourcetype=per_event_summary_index

Three years too late for your use but maybe it will help others.