Getting Data In

overwrite index on heavy forwarder based on port

las
Contributor

Hi.

We are about to ingest logs from multiple suppliers, where the individual supplier has full control over their infrastructure.
My take was to to create a couple of heavy forwarders and dedicate a port to each supplier:
supplier_1 sends data to port 9991
supplier_2 sends data to port 9992
...
This part I think I have working.

The next problem is that I have a need to separate the data from supplier_1 from supplier_2, My thought was to create a index per supplier.
The problem is then how do I route data received from port 9991 to index_1 regardless of what is configured on the Universal Forwarder, except for Splunk stuff (_internal ...) the different suppliers might use the same source or sourcetype, so it is only the receiving port on the heavy forwarder I might use to separate the data.

Any help is much appreciated

Kind regards

0 Karma

dmuraleetcs
Explorer

Hi, 

Since you want to do it in HF, you can modify the input stanza to specify the default index. 

[tcp://9991]
index = supplier1

[tcp://9992]
index = supplier2

 

OR you can add the props & transforms

transforms.conf (if you want to filter you can use sourcekey & regex)

[tcp9991_syslog_supplier1]
SOURCE_KEY = MetaData:Host
REGEX = (10.*.*.*)
DEST_KEY = _MetaData:Index
FORMAT = supplier1

[tcp9992_syslog_supplier2]
SOURCE_KEY = MetaData:Host
REGEX = (10.*.*.*)
DEST_KEY = _MetaData:Index
FORMAT = supplier2

 

0 Karma

las
Contributor

Hi.

Yes, that is the easy way, my problem is that I'm getting data from a Universal Forwarder, that I do not have any control over, it is located at a vendor, but I'm also required to store the data. The Universal Forwarders have used the index stanza, so there is not an easy way to overwrite that.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @las,
suppliers systems send data by syslog or they are Universal Forwarders?

if they are syslogs, you can identify supplier from the source (tcp://9991 or tcp://9992) and you can use the source field to perform an index override:
On your heavy forwarders:

in transforms.conf

[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = index1

in props.conf

[source::tcp://9991]
TRANSFORMS-index = overrideindex

If they are Universal Forwarders, it's more difficoult, you need a way to identify them: e.g. a one or more values in one or more fields that identify supplier servers.
Or if you know the list of servers from each supplier, you could use a lookup to identify them.

Anyway, I don't like to have more indexes containing the same data, I prefer to identify sources in a different way (source or another field).

Ciao.
Giuseppe

las
Contributor

Hi Guiseppe.

I had the idea, that it would be from Universal Forwarders.

Generally I aggree with you, that it is a bad idea to have the same sourcetypes in multiple indexes, but here it makes sense, as only a select few will have access to data from a specific supplier.

Kind regards
Lars

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Lars,
find field/s to identify suppliers, then you can create an automatic field to classify events by Supplier.

e.g. if all hostnames of Supplier1 start with "srv" and hostnames of Supplier2 don't, you can create a calculated field: | eval Supplier=if(host="srv*","Supplier1","Supplier2")

Ciao.
Giuseppe

0 Karma

las
Contributor

Hi Guiseppe.

 

I still haven't put this one to rest.

If I use the _meta in inputs.conf

[splunktcp-ssl:9990]
disabled = 0
_meta userindex::index1

[splunktcp-ssl:9991]
disabled = 0
_meta userindex::index2

then I might be able to route everything thru to my transform where I could do something like this

[force_index]
DEST_KEY = _MetaData:Index
REGEX = userindex::(\w*)\s
FORMAT = $1

 

Do you think this might be a possible solution?

 

Kind regards.

Lars

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @las ,

does it run?

if yes it's a good idea,

I usually don't like to modify indexe value, but it's an idea of mine!

Ciao.

Giuseppe 

0 Karma

las
Contributor

Hi Guiseppe.

 

I have not been sucessfull yet, but I'm still working on the principal and have asked another question on how the reference the _meta variable in a transform.

Kind regards

Lars

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...