Splunk Search

routing and transforming to two different indexers

ldnail_at_TI
Path Finder

Today I have a custom sourcetype = custom:access_combined this is routed in its entirety at the heavy forwarder to two different index clusters.

ldnail_at_TI_0-1629313184782.png

Indexer1 is the dev team, indexer2 is ops.

So the problem I'm running into is that I'd like to:
- route a full copy to indexer1
- for indexer2, run through transforms and drop a bunch of noise (like 75%) ops doesn't need to nullqueue

ldnail_at_TI_1-1629313536940.png

Any ideas on how to approach this? 

 

 

Labels (1)
Tags (1)
0 Karma
1 Solution

shivanshu1593
Builder

Like this:

On your HF, go to props.conf or create one in the directory $SPLUNK_HOME/etc/system/local or $SPLUNK_HOME/etc/apps/<app_name>/local and put the following:

[custom:access_combined]
TRANSFORMS-routing=devRouting
TRANSFORMS-routing1=opsRouting

 In the same path, edit or create transforms.conf and put the following:

[devRouting]
REGEX= .
DEST_KEY=_TCP_ROUTING
FORMAT=devGroup

[opsRouting]
REGEX= <put the regex to select & identify the data that you want to send to the ops indexer>
DEST_KEY=_TCP_ROUTING
FORMAT=opsGroup

 

Then in outputs.conf, append the following changes:

 

[tcpout:devGroup]
server=<ip of dev indexer>:<port number>

[tcpout:opsGroup]
server=<ip of ops indexer>:<port number>

 

Restart splunkd and check the results. If the changes aren't reflecting, please check the following:

1. Your regex is correct.

2. Entries about the IP and port are correct 

3. You or the team has the rights to access data.

4. Run btool and see what configurations are loaded for the app and sourcetype, for which we made the changes.

 

Hope this helps. Let me know if it works.

Thanks,

S

****If it helped, please upvote and accept it as a solution. It helps other Splunkers to find the solutions easily****

 

 

 

 

 

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###

View solution in original post

0 Karma

ldnail_at_TI
Path Finder

I didn't see that in the forest of trees... thanks.

0 Karma

shivanshu1593
Builder

Like this:

On your HF, go to props.conf or create one in the directory $SPLUNK_HOME/etc/system/local or $SPLUNK_HOME/etc/apps/<app_name>/local and put the following:

[custom:access_combined]
TRANSFORMS-routing=devRouting
TRANSFORMS-routing1=opsRouting

 In the same path, edit or create transforms.conf and put the following:

[devRouting]
REGEX= .
DEST_KEY=_TCP_ROUTING
FORMAT=devGroup

[opsRouting]
REGEX= <put the regex to select & identify the data that you want to send to the ops indexer>
DEST_KEY=_TCP_ROUTING
FORMAT=opsGroup

 

Then in outputs.conf, append the following changes:

 

[tcpout:devGroup]
server=<ip of dev indexer>:<port number>

[tcpout:opsGroup]
server=<ip of ops indexer>:<port number>

 

Restart splunkd and check the results. If the changes aren't reflecting, please check the following:

1. Your regex is correct.

2. Entries about the IP and port are correct 

3. You or the team has the rights to access data.

4. Run btool and see what configurations are loaded for the app and sourcetype, for which we made the changes.

 

Hope this helps. Let me know if it works.

Thanks,

S

****If it helped, please upvote and accept it as a solution. It helps other Splunkers to find the solutions easily****

 

 

 

 

 

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...