topic Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar in Getting Data In

How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwarder

vksplunk1 — Tue, 14 Jun 2022 17:57:12 GMT

Hi,

Could you please let me know How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwarder

I tried with the following configuration but all the data going to once index that is defined in inputs.conf. If I remove index from inputs.conf all the events are going to main index.

Thank you in ad

Here my configuration and data:

INPUTS.CONF
======
[monitor:///opt/splunk/var/log/tesData]
disabled = false
host = heaveforwarder1

PROPS.CONF
===========
[source::///opt/splunk/var/log/tesData]
TRANSFORMS-routing=vendorData,secureData

TRANSFORMS.conf
==========
[vendorData]
REGEX=5617605039838520
DEST_KEY=_MetaData:Index
FORMAT=index1

[secureData]
REGEX=6794850084423218
DEST_KEY=_MetaData:Index
FORMAT=index2

testdata:

[08/June/2022:18:23:07] VendorID=5038 Code=C AcctID=5617605039838520
[08/June/2022:18:23:22] VendorID=9109 Code=A AcctID=6794850084423218

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

vksplunk1 — Tue, 14 Jun 2022 18:04:14 GMT

Here is my correct INPUTS.conf. Missing index in the earlier inouts.conf

INPUTS.CONF
======
[monitor:///opt/splunk/var/log/testData]
disabled = false
host = Haveyforwarder1
index = index1

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

jamie00171 — Tue, 14 Jun 2022 21:56:55 GMT

Hi @vksplunk1

From the docs:

The REGEX must have at least one capturing group, even if the FORMAT does
    not reference any capturing groups.

so it might be worth trying to add a capturing group to the regex?

Thanks,

Jamie

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

PickleRick — Wed, 15 Jun 2022 04:45:28 GMT

Generally, there's no way to "split" or "duplicate" an event while indexing. After initial input the event is processed as a whole. You can modify it - trim it, add to it, rewrite parts of it, modify its metadata but it's still within the single event. The only thing that can be "done multiple" with a single event is routing.

So theoretically you could send an event to both the original indexer(s) to one index and to the tcp (so called "syslog") output where it would be re-ingested and processed from the start. But it's a very ugly solution.

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

vksplunk1 — Wed, 15 Jun 2022 04:59:02 GMT

Thank you for the response. To make it clear, I am NOT trying to split the event. I am trying to send the events from the same source to same indexer but to a different INDEXE( index1 or index2 based on the regex )

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

jamie00171 — Wed, 15 Jun 2022 08:23:36 GMT

It might be worth taking a look at this .conf presentation: PLA1154C - Advanced pipeline configurations with INGEST_EVAL and CLONE_SOURCETYPE from 2020 .conf. The last example looks very similar to what you are trying to do, you can watch it here: https://conf.splunk.com/watch/conf-online.html

All of the configuration for the examples is on github: https://github.com/silkyrich/ingest_eval_examples

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

PickleRick — Wed, 15 Jun 2022 10:55:22 GMT

OK. You're right. I forgot about this wonderfully misnamed feature 🙂

You can indeed split the processing pipeline and reinject the same event back for another run.

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

vksplunk1 — Wed, 15 Jun 2022 13:22:40 GMT

Thank you for the response.

Regardless of the REGEX ins transforms.conf in HF, the data always sending to the index in inputs.conf

Seems like splunk not allowing to overide index specified in inpouts.conf

TRANSFORMS.conf
======
#[cdmops]
#REGEX=.*service=cdmops2.*
#DEST_KEY=_MetaData:Index
#FORMAT=cdmops

#[cdmops2]
#REGEX=.*service=cdmops2.*
#DEST_KEY=_MetaData:Index
#FORMAT=cdmops2

Data:
=====
2022/06/01 10:45:50 service=cdmops server=node3 score=50 seq=55041
2022/06/01 10:45:50 service=cdmops2 server=node1 score=17 seq=55042

Re: How to split data to multiple indexes on the same indexer (index1,index2) from one input source from one Heavyforwar

PickleRick — Thu, 16 Jun 2022 06:40:39 GMT

No. Splunk allows you to manipulate metadata during event ingestion.

Question is - where are you trying to do that.

1) Do you indeed have Heavy Forwarder or a Universal Forwarder on the machine you're reading the logfiles on?

2) Do you send the data directly to indexer(s)? Or do you have any intermediate forwarder(s) in the path? If so, what kind of forwarders are those?

3) Where are you putting those props/transforms? (on which component?)