Getting Data In

Can [splunktcp://XXX] be forced to a specific index

jimcroft
Explorer

Hi all

I've configured the following in inputs.conf on our indexer:

[splunktcp://9998]

index=blah

The universal forwarders are configured to send to this port (I've disabled port 9997 on the indexer to be sure) however data still ends up in the main index.

I've checked './bin/splunk cmd btool transforms list --debug' and there are no transforms redirecting to other indexes. The only reference to 'index' in our transforms is:

system [splunk_index_history]

system CAN_OPTIMIZE = True

system CLEAN_KEYS = True

system DEFAULT_VALUE =

system DEST_KEY = _MetaData:Index

system FORMAT = history

system KEEP_EMPTY_VALS = False

system LOOKAHEAD = 4096

system MV_ADD = False

system REGEX = .

system SOURCE_KEY = _raw

system WRITE_META = False

... which shouldn't cause any problems.

Does anyone know why that global input stanza isn't working?

Many thanks

Jim

Tags (2)

mce128
Explorer

I know that this is a very old question. As much searching has not yielded an appropriate solution, I was curious as to your end result. I too would like to be able to enforce the destination index on the indexer as opposed to relying on the forwarder specifying it. It seems to me that allowing the forwarder specify the destination index is a bit of a secuirty issue as one is essentially relying on user input which is generally a security no-no.Did you ever come up with a solution do enforce the destination index on the indexer? Or did you wind up relying on the forwarder to specify it?

Thanks much

0 Karma

lguinn2
Legend

Note that in itinney's answer, he says how to do this using transforms.conf

If you are using Universal Forwarders, transforms.conf must be on the indexers. This would accomplish your goal.

It will add overhead to the parsing to do it this way, though.

0 Karma

jimcroft
Explorer

Thanks itinney

So is the documentation for inputs.conf wrong or am I just reading it wrong? It says that 'index = ' is general setting and...

The following attribute/value pairs are valid for all input types (except file system change monitor,
which is described in a separate section in this file)

There's no indication that it shouldn't work with [splunktcp://XXXX] stanzas.

Do you mind if I run my use case by you and see if you have any ideas?

We're a managed service provider looking after various customers AWS estates. We collect their logs in Splunk. I'm building a new Splunk 4.3 setup and reviewing all the reports, security, etc.

What I'm trying to achieve is some kind of security and data isolation around receiving data from the forwarders. There are two main issues I'm trying resolve:

  1. We cannot guarantee hostname uniqueness across clients (so I want a separate index per client).
  2. Some clients are have admin rights to some servers meaning I cannot rely 100% on them not tampering with the forwarder config. So I want to rely on the [splunktcp://XXXX] port they (any only they) have access to.

What I wanted to do was setup a specific listener port per client and have that listener directed to the clients index. And I'd like this enforced at the indexer rather than forwarder.

Is this at all possible or do I need to rethink this?

Any thoughts you have would be welcome.

Thanks

0 Karma

sowings
Splunk Employee
Splunk Employee

If the forwarder's inputs.conf defines an index, that will override the splunktcp setting for index destination.

You may need a transforms.conf rule like the one cited above to enforce the "per host/port" destination index.

0 Karma

itinney
Path Finder

The "splunktcp" input is not a "data" input, but instead an input to listen to Splunk Forwarders.

Instead you can do one of the following:

  1. Set this in the Forwarder's inputs.conf for each stanza, i.e.:

    [monitor:///var/log]
    disabled = false
    index = blah

  2. Create a transform on the Indexer to override the _MetaData::index field at index time., i.e.

    $ cat transforms.conf
    [setIndexMeta]
    DEFAULT_VALUE = unknown
    REGEX = .
    DEST_KEY = _MetaData:Index
    FORMAT = blah

...and you can call the transforms from props.conf by specifying a host, source or source type upon which to apply the transform.

ephemeric
Contributor

"The "splunktcp" input is not a "data" input, but instead an input to listen to Splunk Forwarders."

Thank you, this clarified the problem for me as I was experiencing the same issue.

0 Karma

broomcupboardsu
Engager

Thx so much, second solution is handy but look like a workaround.
First solution reflect how Splunk UF are meant to be configured to define a specific index for a source, and I totally missed that.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...