topic Splunk standalone: How to split main data in multiple indexes? in Installation

Splunk standalone: How to split main data in multiple indexes?

juju — Fri, 14 Apr 2023 20:45:14 GMT

I installed splunk standalone (9.0.4) with ansible https://github.com/splunk/splunk-ansible/ on Ubuntu jammy.

That has worked well. Data is ingested from port 9997 and for now, everything goes to main index.

I want to split things between multiple indexes aka windows, linux and other source types.

I think this would be through transforms as per https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Routeandfilterdatad but this seems to be only valid for heavy forwarder role.
Or cluster master as per https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_cluster_master/tasks/configure_indexes.yml
In role variable, only found smartstore with an index array but I believe it is different.
I tried

* forwarding working with transform in /opt/splunk/etc/system/local/props.conf and /opt/splunk/etc/system/local/transforms.conf but nok

$ sudo cat /opt/splunk/etc/system/local/props.conf # https://docs.splunk.com/Documentation/Splunk/9.0.4/Indexer/Setupmultipleindexes [SOURCE1] TRANSFORMS-index = SOURCE1Redirect $ sudo cat /opt/splunk/etc/system/local/transforms.conf [SOURCE1Redirect] #REGEX = ,"file":{"path":"\/var\/log\/SOURCE1\/SOURCE1.log"}},"message": REGEX = ^{.*SOURCE1.*}$ DEST_KEY = _MetaData:Index FORMAT = SOURCE1

* get tcp data input losing all the json fields extract and only raw unusable data. Similar to https://community.splunk.com/t5/Getting-Data-In/Splunk-is-adding-weird-strings-like-quot-linebreaker-x00-x00/m-p/21598
* set data receiver in forwarding section and setting index in inputs.conf but not getting data ingested even if data received from tcpdump. And not found how to associate a specific receiver port to an index.

tried

$ sudo more /opt/splunk/etc/system/local/inputs.conf [splunktcp://9997] disabled = 0 [splunktcp://9525] disabled = 0 index = sourcetype1

Any advices?

Thanks

Re: Splunk standalone: How to split main data in multiple indexes?

juju — Fri, 14 Apr 2023 23:41:04 GMT

I managed to split index with multiple splunk HEC and matching index as defined in /opt/splunk/etc/apps/search/local/inputs.conf and /opt/splunk/etc/apps/search/local/indexes.conf

Not sure if recommended way but so far, it works.

Re: Splunk standalone: How to split main data in multiple indexes?

gcusello — Sat, 15 Apr 2023 06:32:24 GMT

Hi @juju,

at first, it's possible to send data to different indexes that main only for new data, when a log is already indexed in one index, it isn't possible to move it in another one, so for the old data, the only way is reindex all of them.

If instead you're speaking of new data, the best approach is to define the index value in the inputs.conf, so the first question is: how do you ingest your data?

you spoke of port 9997, this means that you take data from other Forwarders, so the best and easiest approach is to insert the option "index=<your_index>" in the inputs.conf of the Add-Ons that you're using to take data.

You can also override the index value on the Indexer using the method that you shared in your question and that it's described in many answers in the Community.

The only constrain is that this job must be done by the first Splunk Full instance (not Universal Forwarder) that the data pass through.

In other words, if in your architecture you have one or more intermediate Heavy Forwarders, you must put the index on them to override the index configuration and send data to the correct index.

If instead you don't have any HF, you can put it on the Indexer, in other words the affirmation that it's a role for HFs, in general, is wrong!

Then the transformation is never applied on Cluster Master!

Ciao.

Giuseppe

Re: Splunk standalone: How to split main data in multiple indexes?

juju — Sat, 15 Apr 2023 12:42:37 GMT

Thanks @gcusello

Source is Cribl sending to Splunk Single instance as destination. There is no index option.

https://docs.cribl.io/stream/destinations-splunk

As splunk standalone, no other splunk servers.

Re: Splunk standalone: How to split main data in multiple indexes?

gcusello — Sat, 15 Apr 2023 15:29:07 GMT

Hi @juju,

the problem is that, reading the Cribl documentation, "From the perspective of the receiving Splunk Cloud instance, the data arrives cooked and parsed."

This means that it isn't possible to modify index assignment as described in documentation.

You should ask to Cribl support if it's possible to add the index definition on Cribl,

Ciao.

Giuseppe

Re: Splunk standalone: How to split main data in multiple indexes?

PickleRick — Sat, 15 Apr 2023 20:21:38 GMT

I've never used Cribl but as the data is supposed to be parsed, it means that with each event all metadata is sent along. So you should be able to set the index field within cribl pipeline.

Re: Splunk standalone: How to split main data in multiple indexes?

woodcock — Sun, 16 Apr 2023 21:01:20 GMT

The easiest way is to do that kind of thing is like this:
https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Assignmetadatatoeventsdynamically

Re: Splunk standalone: How to split main data in multiple indexes?

juju — Sun, 16 Apr 2023 21:36:18 GMT

Thanks all!

As said, I got it working with multiple splunk HEC collections.

I will check on cribl side if better way.

Re: Splunk standalone: How to split main data in multiple indexes?

gcusello — Mon, 17 Apr 2023 06:27:11 GMT

Hi @juju ,

if one answer solves your need, please accept one answer for the other people of Community or tell us how we can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)