Getting Data In

create summary index in transforms.conf

markwymer
Path Finder

Hi all,

I currently have a scheduled search that runs every minute and filters certain events for the previous minute and then creates a summary index.

However, I got to wondering whether this would be better done within the transforms.conf using REGEX and DEST? The only problem that I would have is that the events would be 'split' over the two indexes. What I would, ideally, like to achieve is - all events go to index 1 and REGEX filtered events go to index 1 AND index 2.

Is this at all possible?

Would another approach be to create multiple Monitor stanzas for the same log file in my universal forwarder inputs.conf to send ALL the events to the two indexes and use transforms.conf to send any events that I don't want in index 2 to a null index?

Thanks for any advise,
Mark.

0 Karma
1 Solution

somesoni2
Revered Legend

I don't think so there is a native method to achieve this. Creating multiple monitoring stanza will not work as Splunk merges the configuration with same stanza names.

There are workarounds available. See this for more details

https://answers.splunk.com/answers/61433/have-forwarder-duplicating-data-to-2-indexes.html

View solution in original post

somesoni2
Revered Legend

I don't think so there is a native method to achieve this. Creating multiple monitoring stanza will not work as Splunk merges the configuration with same stanza names.

There are workarounds available. See this for more details

https://answers.splunk.com/answers/61433/have-forwarder-duplicating-data-to-2-indexes.html

hartfoml
Motivator

@somesoni2, since the accepted answer in the "Have forwarder duplicating data to 2 indexes." answer is using the crcSalt command the duplicate log events sent to both indexes will be seen as individual records and count twice against the splunk license. If summery of accelerated search is done to put the data in a tsidx OR summarydb it will not be charged against the license. I get that if this is done at the forwarder or at index time rather than at search time this could potently save CPU cycles or at lease READ IO but overall I don't think you are really saving resources by doing it at index rather than at search time so why use extra licensing to index the same event twice??? -- Sorry just my opinion --( :-] )

See you all at the end of September >DOT CONF IS COMING<>DOT CONF IS COMING<

0 Karma

somesoni2
Revered Legend

I absolutely agree. I was about to add the caveat with this approach when some hit me at work.

Apart from the point with @hartfoml, that above approach(s) will cause the double license usage (at least for the duplicate parts) which the summary indexing doesn't, I also want to bring the point that, if you wish to follow the approach above, you'd need to create a new sourcetype (for data to go to index2) with necessary event filter (transforms.conf), as you want to keep all data in index 1.

Masa
Splunk Employee
Splunk Employee
  1. A kinda hidden built-in feature is available for this question. Well, not documented well. But, it works. Yes, you will use more license volume in this approach. (ref: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf )

    CLONE_SOURCETYPE

  2. Summary Index is a good approach in general. Search time configuration can be changed, summary index data could be re-indexed.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...