topic Re: Set the index with a field when using collect command in Splunk Search

Set the index with a field when using collect command

ejwade — Fri, 10 May 2024 00:39:23 GMT

Hello!

I'm looking to set the index parameter of the collect command with the value of a field from each event.

Here's an example.

| makeresults count=2 | streamstats count | eval index = case(count=1, "myindex1", count=2, "myindex2") | collect index=index testmode=true

This search creates two events. Both events have the index field, one with "myindex1" as the value, and the other with "myindex2". I would like to use these values to set the index in the collect command.

Re: Set the index with a field when using collect command

bowesmana — Fri, 10 May 2024 01:29:34 GMT

I don't believe it is possible to do - you can in theory do this

but you would need for the subsearch to know the index to select and that is run before the outer search, so you can't do what you are trying to do

Re: Set the index with a field when using collect command

danspav — Fri, 10 May 2024 05:15:29 GMT

Hi @ejwade,

I'm with @bowesmana on this - I don't think it's possible to run | collect with multiple index locations.

You could do this instead:

You will need an appendpipe command for each index you want to export to, but you should know the destination indexes in advance anyway.

Re: Set the index with a field when using collect command

PickleRick — Fri, 10 May 2024 05:34:41 GMT

You can't. Even with output_format=hec you can specify some metadata fields like source or sourcetype (which can affect your license usage) but the destination index has to be provided explicitly with the collect command invocation.

Re: Set the index with a field when using collect command

ejwade — Mon, 13 May 2024 15:44:17 GMT

After tooling with it more, I think the best approach uses the map command.

report_to_map_through_indexes

| inputlookup lookup_of_events where index="$index$" | collect index="$index$"

Re: Set the index with a field when using collect command

PickleRick — Mon, 13 May 2024 17:51:22 GMT

Be aware that map is a potentially unsafe command.

Also your approach with both map and an intermediate lookup seems strange. That's what passing fields to the subsearch is for.

Re: Set the index with a field when using collect command

bowesmana — Mon, 13 May 2024 22:54:05 GMT

It can be done with map, but the phrase best approach uses the map command is not a phrase that would normally be used when considering the map command. As @PickleRick indicates, it has to be used carefully.

In your pseudo example it's fine, but with real data remember that each result will initiate a new run of the saved search - if you have lots of results, as this runs collect for EACH and every row, it can place significant additional load on the server - and by default it will only run 10 iterations.

Re: Set the index with a field when using collect command

PickleRick — Tue, 14 May 2024 08:18:09 GMT

On top of that your use might simply be restricted from using such commands. And your dashboards may not run if powered by risky commands.

https://docs.splunk.com/Documentation/Splunk/latest/Security/SPLsafeguards

Re: Set the index with a field when using collect command

ejwade — Wed, 15 May 2024 21:31:28 GMT

The lookup reduces the iterations of the map command. In a real world scenario, I have a field called "dept" that lists one of ten departments for each result. The map command only needs to iterate through each one (ten times total), so the output lookup saves off the data, then the stats separates each dept, and the map iterates through.

Re: Set the index with a field when using collect command

Petermann — Tue, 29 Apr 2025 09:11:20 GMT

How to understand: "report_to_map_through_indexes",
I tried to built a macro but got server error or shall it become a custom command? or how tp implement?

Re: Set the index with a field when using collect command

ITWhisperer — Tue, 29 Apr 2025 12:10:07 GMT

@Petermann You have tried to piggy-back your question onto someone else's solved question without a clear indication as to how your question is related. Since this is already marked as solved, it is less likely to receive the attention you might wish. You would be better off starting your own question, clearly stating your usecase, providing sample data (anonymised as minimally as possible, of course), showing what your expected output would be, what you have tried, what errors/messages you are getting, and state why this is not what you want.

Re: Set the index with a field when using collect command

PickleRick — Tue, 29 Apr 2025 18:09:22 GMT

@ITWhispererIt wasn't obvious at first glance for me either but if you scroll back "report_to_map_through_indexes" was actually a name of a saved search used in the solution.

@PetermannAs you can see in the docs for the map command, it takes either a literal search as an argument or a name of a saved search. In this case @ejwade used the latter option. The map command references a report_to_map_through_indexes report definition of which is shown below in the original solution.