create summary index in transforms.conf

[markwymer]

Hi all,

I currently have a scheduled search that runs every minute and filters certain events for the previous minute and then creates a summary index.

However, I got to wondering whether this would be better done within the transforms.conf using REGEX and DEST? The only problem that I would have is that the events would be 'split' over the two indexes. What I would, ideally, like to achieve is - all events go to index 1 and REGEX filtered events go to index 1 AND index 2.

Is this at all possible?

Would another approach be to create multiple Monitor stanzas for the same log file in my universal forwarder inputs.conf to send ALL the events to the two indexes and use transforms.conf to send any events that I don't want in index 2 to a null index?

Thanks for any advise,
Mark.

[somesoni2 · Accepted solution]

I don't think so there is a native method to achieve this. Creating multiple monitoring stanza will not work as Splunk merges the configuration with same stanza names.

There are workarounds available. See this for more details

https://answers.splunk.com/answers/61433/have-forwarder-duplicating-data-to-2-indexes.html

[hartfoml]

@somesoni2, since the accepted answer in the "Have forwarder duplicating data to 2 indexes." answer is using the crcSalt command the duplicate log events sent to both indexes will be seen as individual records and count twice against the splunk license. If summery of accelerated search is done to put the data in a tsidx OR summarydb it will not be charged against the license. I get that if this is done at the forwarder or at index time rather than at search time this could potently save CPU cycles or at lease READ IO but overall I don't think you are really saving resources by doing it at index rather than at search time so why use extra licensing to index the same event twice??? -- Sorry just my opinion --( :-] )

See you all at the end of September >DOT CONF IS COMING<>DOT CONF IS COMING<

[somesoni2]

I absolutely agree. I was about to add the caveat with this approach when some hit me at work.

Apart from the point with @hartfoml, that above approach(s) will cause the double license usage (at least for the duplicate parts) which the summary indexing doesn't, I also want to bring the point that, if you wish to follow the approach above, you'd need to create a new sourcetype (for data to go to index2) with necessary event filter (transforms.conf), as you want to keep all data in index 1.

[Masa]

1. A kinda hidden built-in feature is available for this question. Well, not documented well. But, it works. Yes, you will use more license volume in this approach. (ref: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf )

   CLONE_SOURCETYPE

2. Summary Index is a good approach in general. Search time configuration can be changed, summary index data could be re-indexed.