Knowledge Management

How to collect index="based on values"

fklink
New Member

Hi everybody,

is it possible to create several summary index within one search?

Example:
"Index A" has a field "OS" with values "Windows", "Linux"...

Is there a way to tell splunk to loop something like: index=A | collect index="OS" (where OS is the field-value)

So that each result based on field OS will be stored in (allready created) index "Windows", "Linux", ....

Many thanks in advance.

Labels (1)
0 Karma

scombs
Explorer

The following is a subset of a larger solution I came up with to anonymize and re-index some previously ingested events while maintaining their original index, sourcetype, source, host, and _time fields.

Create a set of index-time props and transforms that:

  1. Separately assign each event to their own index.
  2. Restore default summary sourcetype of "stash".
  3. Remove the pre-pended index name from each event.

Then pre-pend a target index name to each event record before calling |collect with a sourcetype matching the stanza from your props.conf (and which is reverted back to "stash" to not count against license).

 

#Index time props.conf
[per_event_summary_index]
TRANSFORMS-set_summary_index = reset_index, reset_stash_sourcetype, remove_prepended_field

#Index time transforms.conf
[reset_index]
DEST_KEY = _MetaData:Index
REGEX = (.*?),
FORMAT = $1
[reset_stash_sourcetype]
DEST_KEY = MetaData:Sourcetype
FORMAT = stash
[remove_prepended_field]
INGEST_EVAL = _raw=replace(_raw,".*?,","")

```Send events to separate summary indexes based on OS.```
index=A 
|eval _raw=OS.","._raw 
|collect sourcetype=per_event_summary_index

 

 

Three years too late for your use but maybe it will help others.

0 Karma

woodcock
Esteemed Legend

Like this:

search that generates events with some field rectype
| _SI_Name_{index} = "IgnoreMe"
| foreach _SI_Name_* [
| appendpipe [ 
   where rectype="$MATCHSTR$"
| collect index="$MATCHSTR$" testmode=f other parameters
| where false()
] ]
| rename COMMENT as "Anything left at this point did not go into an index, which may be an error"
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Here's how I've written that before

search that generates events with some field rectype

| rename COMMENT as "send foo to index foo"
| appendpipe [ 
    | where rectype="foo" 
    | collect index="foo" testmode=f other parameters
    | where false()
    ]

| rename COMMENT as "send bar to index bar"
 | appendpipe [ 
    | where rectype="bar" 
    | collect index="bar" testmode=f other parameters
    | where false()
    ]

| rename COMMENT as "repeat for each destination index"


| rename COMMENT as "This last one doesn't go to an index but it lets you see as the search results what would have done so for all types"
| collect index="goingnowhere" testmode=t other parameters 
0 Karma

fklink
New Member

Hi DalJeanis,

many thanks for your help.

Since the values within "rectype" can differ, I'm looking for a solution that fills the field variably.

someting like: | collect index="$value_of_rectype$"

0 Karma

gcusello
Legend

Hi fklink,
why do you need to use only one search?
Bye.
Giuseppe

0 Karma

fklink
New Member

Hi Guiseppe,

the values Windows and Linux are exemplary.

In fact, there are about 150 different values that vary.

My goal is to use one query to fill the 150 different indexes (automatically)

0 Karma

gcusello
Legend

Hi fklink,
are you sure about this?
an elevated quantity of indexes isn't a best practice!
You should analyze your needs and define the correct number of indexes thinking to the main reason to have different indexes:

  • different retention periods,
  • different access rights.

In addition you can take in consideration also the quantity of logs ingested (e.g. it isn't a good practice to have in the same index logs from large flows with logs from little flows).
But anyway 150 indexes are very many indexes!
Don't think to indexes as DB tables, Splunk is different!

Bye.
Giuseppe

0 Karma

fklink
New Member

Hi Guiseppe,

thanks for your help.

Because of permissions i have to create 150 index 😞

Maybe i will find an other solution.....

0 Karma
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...