Getting Data In

Problem: Unable to send cooked data to two different Indexer ports

splunk_kk
Path Finder

Hello Experts,

I have an issue where I am unable to send cooked data to two different Indexer ports.

My flow of traffic is UF > HF > IDX

UF IP: a.a.a.a
HF IP: y.y.y.y
IDX IP: x.x.x.x

1) Universal Forwarder is monitoring more than one input with different sourcetypes.
2) My Indexer is listening on TCP 9997 and TCP 9996 (Same Indexer is listening on both the ports)
3) I want the data with sourcetype ABC to hit TCP 9996 and all other data to hit TCP 9997 on the Indexer.
4) When hitting TCP 9996, the index should change to pqr index but while hitting TCP 9997 the index should remain the same as mentioned in the monitor stanza in UF

My configuration is below:

UF config

inputs.conf

[monitor:///xyz/urgent]
index= my_index
sourcetype=ABC
disabled=false

[monitor:///var/important]
index= my_index
sourcetype=DEF
disabled=false

outputs.conf

[tcpout]
useACK=true
maxQueueSize=auto
readTimeout=300

[tcpout:HF]
server = y.y.y.y:9997

HF config

inputs.conf

[default]
host = def.sdf.ppp

[splunktcp://9997]

props.conf

[ABC]
TRANSFORMS-routing = route_data

transforms.conf

[route_data]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = urgent

outputs.conf

[tcpout]
defaultGroup = important
maxQueueSize = 50MB
useACK = true

[tcpout:important]
server = x.x.x.x:9997

[tcpout:urgent]
server = x.x.x.x:9996

Indexer config

inputs.conf

[default]
host = abcdef.xyz.ppp

[splunktcp://9997]

[splunktcp://9996]
index = pqr

This is not working for me. The sourcetype ABC is still sending data on TCP 9997 and not changing the index.

Have I missed something? is there anything wrong with the config?

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

In your inputs.conf on the indexers, for splunktcp, index = is not a valid option.

You cannot do it this way. You would need to rewrite the index metadata for the sourcetype on your HF by adding another transform and change the index metadata.

View solution in original post

esix_splunk
Splunk Employee
Splunk Employee

Think about your issue, I think you would be better to approach this differently. Instead of trying to assign the index or priority based on the inbound SplunkTCP port, why dont you add metadata at the ingest time. Add a metadata field such as PRIORITY-HIGH, and then at the HF layer, filter that to a specific index and all other data to general indexes...

See documentation here : http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Configureindex-timefieldextraction

Note, this does add an indexed field, which does effect indexing performance. But this is a better approach then trying to work with multiple TCP inputs and trasnforms.

0 Karma

splunk_kk
Path Finder

Thanks Esix, That worked for me.... 🙂 ok a quick question, If in case I need to send the data from sourcetype ABC to both the ports (9997 and 9996) and change the index when it hits 9996. will that be possible? if I change the index metadata at HF it will change the index for both the ports.

Do I need to write props and transforms in Index for that to work? I doubt

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Its not really possible to do this at the indexer side. Basically Splunk isnt able to change the meta data on multiple listening ports since it doesnt track which input port the data comes in on.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

In your inputs.conf on the indexers, for splunktcp, index = is not a valid option.

You cannot do it this way. You would need to rewrite the index metadata for the sourcetype on your HF by adding another transform and change the index metadata.

Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...