Getting Data In

Why is data segregation by index not displaying events?

fab73
Path Finder

I'm trying to segregate data coming from a specific Heavy Forwarder using a specific index (my_index). So as per Answers and Manual:

  1. I defined also "my_index" index on the two Indexers that receive the data. No index is defined on the Search Head.
  2. In inputs.conf, I inserted on the Heavy Forwarder:

    [input]
    index = my_index

  3. I configured a specific role and its users to search on this index

Looking at the console the my_index is empty (zero events), zero current size. Any search like index=my_index give zero results,
although events are coming to the indexer (I see tcmpdump trace of the message arriving on the Indexer when events occur). Any idea? Something different in Splunk 6.5.2?

thanks in advance

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi fab73,
you have to correct some items, because you don't need to configure anything on your Heavy Forwarder:

  • create an index on your indexers (my_index);
  • create a specific role (new_role);
  • grant access to my_index only to new_role;
  • create in your Universal Forwarders inputs.conf addressing my_index for the specific events;

    [monitor://your_path/your_file.log]
    index = my_index
    sourcetype = my_sourcetype

  • if you don't want to filter or elaborate your events on Heavy Forwarder you don't need to configure anything on it, except obviously forwarding to your Indexers.

Bye.
Giuseppe

View solution in original post

fab73
Path Finder

Better solution for me has been finally (filtering and) making data segmentation at Intermediate Forwarder level this way ..... ( for instance I put all sourcetype syslog into my_index)

in props.conf

[syslog]
TRANSFORMS-index-my_index=overrideindex-syslog

in transform.conf

[overrideindex-syslog]
 DEST_KEY =_MetaData:Index
 REGEX = .
 FORMAT = my_index

this way no file is to be pointed but the sourcetype. Configuration is working for all UF deplyed. Multitenancy is served.

0 Karma

fab73
Path Finder

I solved operating at a UF level as suggested by Giuseppe and Odonio. Question about why the index use at a IF tier did not work still remains open as I asked originally, but I go further! tanks!

Maybe because index assignment (index=my_index) cannot be done before indexing and IF does not index.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi fab73,
you have to correct some items, because you don't need to configure anything on your Heavy Forwarder:

  • create an index on your indexers (my_index);
  • create a specific role (new_role);
  • grant access to my_index only to new_role;
  • create in your Universal Forwarders inputs.conf addressing my_index for the specific events;

    [monitor://your_path/your_file.log]
    index = my_index
    sourcetype = my_sourcetype

  • if you don't want to filter or elaborate your events on Heavy Forwarder you don't need to configure anything on it, except obviously forwarding to your Indexers.

Bye.
Giuseppe

fab73
Path Finder

Hi Giuseppe, I have an Intermediate Forwarder in order to concentrate all filtering at that tier.
So I have

......<-----INDEXER(S)<---------INTERMEDIATE FORWARDER<---------------UNIVERSAL FORWARDER(S)

The Intermediate Forwarder doesn't perform indexing it's just like an Heavy Forwarder. this tier should only put all data coming from UNIVERSAL FORWARDER into my_index!!

So i configured /opt/splunk/etc/system/local/inputs.conf with the proper input stanza. That is not working.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi fab73,
maybe I didn't understood the situation you described:
in your question you said that you inserted inputs.conf in Heavy Forwarder, this is wrong: inputs.conf must be on Universal Forwarders not in HF!

Many times I used an Heavy Forwarder as a log concentrator, but the only configurations I did on HF were:

  • enabled forwarding
  • enabled receiving

instead inputs.conf with the correct index addressing was configured on Universal Forwarders and not on HF.

The only exception was when I had the need to filter events before forwarding to Indexers, but it was a different need.

there is another way to override index before indexing
On indexers or HFs (not on UFs):

props.conf

[your_sourcetype]
TRANSFORMS-index = overrideindex

transforms.conf

[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index

Bye.
Giuseppe

0 Karma

fab73
Path Finder

Hi Giuseppe,

me too perform filtering at Intermediate level. Since Intermediate Forwarder is a complete Splunk instance (just configured as you said: forwarding+rerceiving, not indexing) this instance can concentrate many actions. One of this for me would like to be segregation on geographic base.

The inputs.conf is on the Intermediate Forwarder too. I want everithing that comes to this node go into a specific index, when it will arrive to indexing tier. In Splunk Manual we see "Send events to specific indexes" Indexer Manual. I was trying this strategy.

Transform for a specific source_type means defining a source_type, but I'm trying to have multitenant not segregation by source_type.

I will go further and will report. Thanks a lot Giuseppe.

0 Karma

woodcock
Esteemed Legend

Why? This is a bad idea and will not scale. Get rid of the do-nothing-good IF/HF.

0 Karma

fab73
Path Finder

Architecture scale fast horizontally and solve some "political" questions upon Customers. It depends on the needs.

0 Karma

adonio
Ultra Champion

try to search index = * and verify the data really come to indexer. it supposed to land in your default index. If it does, now check your inputs file and make sure your stanza reflects the inputs. here is the link for docs with examples: https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Inputsconf
Hope it helps

0 Karma

fab73
Path Finder

Thanks adonio: data comes to the indexer but into "main" index, not in the "my_index". Data segregation doesn't take place as configured. A search with index=* shows the events n the current time windows. It seems like the stanza is ignored.

0 Karma

adonio
Ultra Champion

just fix inputs.conf as suggested in the answer below

0 Karma

fab73
Path Finder

I think this configuration below is suitable only for Universal Fw not for an Intermediate Forwarder. Actually I need to operate at an Intermediate Forwarder tier.

If I have 100 hosts to monitor is not efficient fix 100 inputs.conf files. So I am wondering if is there any instance working for "all" data coming to the Intermedite Forwarder from a Group of host/Forwarders.

The only way to do this I found in Manual and Answers is to operate with index = my_index but unfortunately is not working .....

0 Karma

adonio
Ultra Champion

fab73, the inputs config should work on your forwarder (regular).

lets take a step back, you can easily manage thousands of forwarders and scale by using deployment server. some reading material here: http://docs.splunk.com/Documentation/Splunk/6.5.2/Updating/Aboutdeploymentserver
can you elaborate on the considerations for Heavy / intermediate forwarder?
cheers

0 Karma

fab73
Path Finder

Hi adonio, index segregation should work but it does not work for me, something should be checked in my general architecture and configuration, and I'm wondering what....

Have a look to my comment to Giuseppe below also: my Intemediate forwarder does not perform indexing, because indexAndForward=false.

So I'm wondering if the

[input]
index = my_index

in inputs.conf is working in this condition. Also if I change indexAndForward=true, I doesn't make difference.

Intermediate Forwarder has several advanages:

1-reduce security rules to send data out from Customer to Cloud
2-aggregate all filetring into one instance (optimization of licence consumption)
3-aggregate segregation rules to one instance (if it would work!!!!!! :-))

no need for a deploiment server at the moment.

0 Karma

adonio
Ultra Champion

take a look at that answer: https://answers.splunk.com/answers/211403/how-to-configure-inputsconf-and-outputsconf-on-the.html
keep inputs.conf in regular forwarders as follow:

[monitor://<your/path/to/data>]
sourcetype = <your_sourcetype>
index = <your_index>

BTW, i disagree with advantages 2 and 3 - these are just aren't true but this is for another time

0 Karma

fab73
Path Finder

thanks adonio for your comments. For what I can see I have no choice, I have to operate at Forwarder tier. I have to good into deep about deployment server for sure in the future.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...