topic Re: Can [splunktcp://XXX] be forced to a specific index in Getting Data In

Can [splunktcp://XXX] be forced to a specific index

jimcroft — Mon, 28 Sep 2020 11:22:52 GMT

Hi all

I've configured the following in inputs.conf on our indexer:

[splunktcp://9998]

index=blah

The universal forwarders are configured to send to this port (I've disabled port 9997 on the indexer to be sure) however data still ends up in the main index.

I've checked './bin/splunk cmd btool transforms list --debug' and there are no transforms redirecting to other indexes. The only reference to 'index' in our transforms is:

system [splunk_index_history]

system CAN_OPTIMIZE = True

system CLEAN_KEYS = True

system DEFAULT_VALUE =

system DEST_KEY = _MetaData:Index

system FORMAT = history

system KEEP_EMPTY_VALS = False

system LOOKAHEAD = 4096

system MV_ADD = False

system REGEX = .

system SOURCE_KEY = _raw

system WRITE_META = False

... which shouldn't cause any problems.

Does anyone know why that global input stanza isn't working?

Many thanks

Jim

Re: Can [splunktcp://XXX] be forced to a specific index

itinney — Mon, 28 Sep 2020 11:23:02 GMT

The "splunktcp" input is not a "data" input, but instead an input to listen to Splunk Forwarders.

Instead you can do one of the following:

Set this in the Forwarder's inputs.conf for each stanza, i.e.:

[monitor:///var/log]
disabled = false
index = blah
Create a transform on the Indexer to override the _MetaData::index field at index time., i.e.

$ cat transforms.conf
[setIndexMeta]
DEFAULT_VALUE = unknown
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = blah

...and you can call the transforms from props.conf by specifying a host, source or source type upon which to apply the transform.

Re: Can [splunktcp://XXX] be forced to a specific index

jimcroft — Fri, 10 Feb 2012 17:30:35 GMT

Thanks itinney

So is the documentation for inputs.conf wrong or am I just reading it wrong? It says that 'index = ' is general setting and...

The following attribute/value pairs are valid for all input types (except file system change monitor,
which is described in a separate section in this file)

There's no indication that it shouldn't work with [splunktcp://XXXX] stanzas.

Do you mind if I run my use case by you and see if you have any ideas?

We're a managed service provider looking after various customers AWS estates. We collect their logs in Splunk. I'm building a new Splunk 4.3 setup and reviewing all the reports, security, etc.

What I'm trying to achieve is some kind of security and data isolation around receiving data from the forwarders. There are two main issues I'm trying resolve:

We cannot guarantee hostname uniqueness across clients (so I want a separate index per client).
Some clients are have admin rights to some servers meaning I cannot rely 100% on them not tampering with the forwarder config. So I want to rely on the [splunktcp://XXXX] port they (any only they) have access to.

What I wanted to do was setup a specific listener port per client and have that listener directed to the clients index. And I'd like this enforced at the indexer rather than forwarder.

Is this at all possible or do I need to rethink this?

Any thoughts you have would be welcome.

Thanks

Re: Can [splunktcp://XXX] be forced to a specific index

mce128 — Wed, 24 Oct 2012 21:12:16 GMT

I know that this is a very old question. As much searching has not yielded an appropriate solution, I was curious as to your end result. I too would like to be able to enforce the destination index on the indexer as opposed to relying on the forwarder specifying it. It seems to me that allowing the forwarder specify the destination index is a bit of a secuirty issue as one is essentially relying on user input which is generally a security no-no.Did you ever come up with a solution do enforce the destination index on the indexer? Or did you wind up relying on the forwarder to specify it?

Thanks much

Re: Can [splunktcp://XXX] be forced to a specific index

lguinn2 — Wed, 24 Oct 2012 21:29:38 GMT

Note that in itinney's answer, he says how to do this using transforms.conf

If you are using Universal Forwarders, transforms.conf must be on the indexers. This would accomplish your goal.

It will add overhead to the parsing to do it this way, though.

Re: Can [splunktcp://XXX] be forced to a specific index

sowings — Wed, 24 Oct 2012 22:14:05 GMT

If the forwarder's inputs.conf defines an index, that will override the splunktcp setting for index destination.

You may need a transforms.conf rule like the one cited above to enforce the "per host/port" destination index.

Re: Can [splunktcp://XXX] be forced to a specific index

broomcupboardsu — Mon, 18 May 2020 14:35:35 GMT

Thx so much, second solution is handy but look like a workaround.
First solution reflect how Splunk UF are meant to be configured to define a specific index for a source, and I totally missed that.

Re: Can [splunktcp://XXX] be forced to a specific index

ephemeric — Mon, 09 Aug 2021 14:10:32 GMT

"The "splunktcp" input is not a "data" input, but instead an input to listen to Splunk Forwarders."

Thank you, this clarified the problem for me as I was experiencing the same issue.