Getting Data In

Can [splunktcp://XXX] be forced to a specific index

Explorer

Hi all

I've configured the following in inputs.conf on our indexer:

[splunktcp://9998]

index=blah

The universal forwarders are configured to send to this port (I've disabled port 9997 on the indexer to be sure) however data still ends up in the main index.

I've checked './bin/splunk cmd btool transforms list --debug' and there are no transforms redirecting to other indexes. The only reference to 'index' in our transforms is:

system [splunkindexhistory]

system CANOPTIMIZE = True

system CLEAN
KEYS = True

system DEFAULTVALUE =

system DEST
KEY = MetaData:Index

system FORMAT = history

system KEEP
EMPTYVALS = False

system LOOKAHEAD = 4096

system MV
ADD = False

system REGEX = .

system SOURCEKEY = _raw

system WRITE
META = False

... which shouldn't cause any problems.

Does anyone know why that global input stanza isn't working?

Many thanks

Jim

Tags (2)

Explorer

I know that this is a very old question. As much searching has not yielded an appropriate solution, I was curious as to your end result. I too would like to be able to enforce the destination index on the indexer as opposed to relying on the forwarder specifying it. It seems to me that allowing the forwarder specify the destination index is a bit of a secuirty issue as one is essentially relying on user input which is generally a security no-no.Did you ever come up with a solution do enforce the destination index on the indexer? Or did you wind up relying on the forwarder to specify it?

Thanks much

0 Karma

Legend

Note that in itinney's answer, he says how to do this using transforms.conf

If you are using Universal Forwarders, transforms.conf must be on the indexers. This would accomplish your goal.

It will add overhead to the parsing to do it this way, though.

0 Karma

Explorer

Thanks itinney

So is the documentation for inputs.conf wrong or am I just reading it wrong? It says that 'index = ' is general setting and...

The following attribute/value pairs are valid for all input types (except file system change monitor,
which is described in a separate section in this file)

There's no indication that it shouldn't work with [splunktcp://XXXX] stanzas.

Do you mind if I run my use case by you and see if you have any ideas?

We're a managed service provider looking after various customers AWS estates. We collect their logs in Splunk. I'm building a new Splunk 4.3 setup and reviewing all the reports, security, etc.

What I'm trying to achieve is some kind of security and data isolation around receiving data from the forwarders. There are two main issues I'm trying resolve:

  1. We cannot guarantee hostname uniqueness across clients (so I want a separate index per client).
  2. Some clients are have admin rights to some servers meaning I cannot rely 100% on them not tampering with the forwarder config. So I want to rely on the [splunktcp://XXXX] port they (any only they) have access to.

What I wanted to do was setup a specific listener port per client and have that listener directed to the clients index. And I'd like this enforced at the indexer rather than forwarder.

Is this at all possible or do I need to rethink this?

Any thoughts you have would be welcome.

Thanks

0 Karma

Splunk Employee
Splunk Employee

If the forwarder's inputs.conf defines an index, that will override the splunktcp setting for index destination.

You may need a transforms.conf rule like the one cited above to enforce the "per host/port" destination index.

0 Karma

Path Finder

The "splunktcp" input is not a "data" input, but instead an input to listen to Splunk Forwarders.

Instead you can do one of the following:

  1. Set this in the Forwarder's inputs.conf for each stanza, i.e.:

    [monitor:///var/log]
    disabled = false
    index = blah

  2. Create a transform on the Indexer to override the _MetaData::index field at index time., i.e.

    $ cat transforms.conf
    [setIndexMeta]
    DEFAULTVALUE = unknown
    REGEX = .
    DEST
    KEY = _MetaData:Index
    FORMAT = blah

...and you can call the transforms from props.conf by specifying a host, source or source type upon which to apply the transform.

Thx so much, second solution is handy but look like a workaround.
First solution reflect how Splunk UF are meant to be configured to define a specific index for a source, and I totally missed that.

0 Karma