Why is data segregation by index not displaying events?

[fab73]

I'm trying to segregate data coming from a specific Heavy Forwarder using a specific index (my_index). So as per Answers and Manual:

1. I defined also "my_index" index on the two Indexers that receive the data. No index is defined on the Search Head.
   

2. In inputs.conf, I inserted on the Heavy Forwarder:

   [input]
   index = my_index

3. I configured a specific role and its users to search on this index

Looking at the console the my_index is empty (zero events), zero current size. Any search like index=my_index give zero results,
although events are coming to the indexer (I see tcmpdump trace of the message arriving on the Indexer when events occur). Any idea? Something different in Splunk 6.5.2?

thanks in advance

[fab73]

Better solution for me has been finally (filtering and) making data segmentation at Intermediate Forwarder level this way ..... ( for instance I put all sourcetype syslog into my_index)

in props.conf

[syslog]
TRANSFORMS-index-my_index=overrideindex-syslog

in transform.conf

[overrideindex-syslog]
 DEST_KEY =_MetaData:Index
 REGEX = .
 FORMAT = my_index

this way no file is to be pointed but the sourcetype. Configuration is working for all UF deplyed. Multitenancy is served.

[fab73]

I solved operating at a UF level as suggested by Giuseppe and Odonio. Question about why the index use at a IF tier did not work still remains open as I asked originally, but I go further! tanks!

Maybe because index assignment (index=my_index) cannot be done before indexing and IF does not index.

[gcusello · Accepted solution]

Hi fab73,
you have to correct some items, because you don't need to configure anything on your Heavy Forwarder:

• create an index on your indexers (my_index);
• create a specific role (new_role);
• grant access to my_index only to new_role;

• create in your Universal Forwarders inputs.conf addressing my_index for the specific events;

  [monitor://your_path/your_file.log]
  index = my_index
  sourcetype = my_sourcetype

• if you don't want to filter or elaborate your events on Heavy Forwarder you don't need to configure anything on it, except obviously forwarding to your Indexers.

Bye.
Giuseppe

[fab73]

Hi Giuseppe, I have an Intermediate Forwarder in order to concentrate all filtering at that tier.
So I have

......<-----INDEXER(S)<---------INTERMEDIATE FORWARDER<---------------UNIVERSAL FORWARDER(S)

The Intermediate Forwarder doesn't perform indexing it's just like an Heavy Forwarder. this tier should only put all data coming from UNIVERSAL FORWARDER into my_index!!

So i configured /opt/splunk/etc/system/local/inputs.conf with the proper input stanza. That is not working.

[gcusello]

Hi fab73,
maybe I didn't understood the situation you described:
in your question you said that you inserted inputs.conf in Heavy Forwarder, this is wrong: inputs.conf must be on Universal Forwarders not in HF!

Many times I used an Heavy Forwarder as a log concentrator, but the only configurations I did on HF were:

• enabled forwarding
• enabled receiving

instead inputs.conf with the correct index addressing was configured on Universal Forwarders and not on HF.

The only exception was when I had the need to filter events before forwarding to Indexers, but it was a different need.

there is another way to override index before indexing
On indexers or HFs (not on UFs):

props.conf

[your_sourcetype]
TRANSFORMS-index = overrideindex

transforms.conf

[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index

Bye.
Giuseppe

[fab73]

Hi Giuseppe,

me too perform filtering at Intermediate level. Since Intermediate Forwarder is a complete Splunk instance (just configured as you said: forwarding+rerceiving, not indexing) this instance can concentrate many actions. One of this for me would like to be segregation on geographic base.

The inputs.conf is on the Intermediate Forwarder too. I want everithing that comes to this node go into a specific index, when it will arrive to indexing tier. In Splunk Manual we see "Send events to specific indexes" Indexer Manual. I was trying this strategy.

Transform for a specific source_type means defining a source_type, but I'm trying to have multitenant not segregation by source_type.

I will go further and will report. Thanks a lot Giuseppe.

[woodcock]

Why? This is a bad idea and will not scale. Get rid of the do-nothing-good IF/HF.

[fab73]

Architecture scale fast horizontally and solve some "political" questions upon Customers. It depends on the needs.

[adonio]

try to search index = * and verify the data really come to indexer. it supposed to land in your default index. If it does, now check your inputs file and make sure your stanza reflects the inputs. here is the link for docs with examples: https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Inputsconf
Hope it helps

[fab73]

Thanks adonio: data comes to the indexer but into "main" index, not in the "my_index". Data segregation doesn't take place as configured. A search with index=* shows the events n the current time windows. It seems like the stanza is ignored.

[adonio]

just fix inputs.conf as suggested in the answer below

[fab73]

I think this configuration below is suitable only for Universal Fw not for an Intermediate Forwarder. Actually I need to operate at an Intermediate Forwarder tier.

If I have 100 hosts to monitor is not efficient fix 100 inputs.conf files. So I am wondering if is there any instance working for "all" data coming to the Intermedite Forwarder from a Group of host/Forwarders.

The only way to do this I found in Manual and Answers is to operate with index = my_index but unfortunately is not working .....

[adonio]

fab73, the inputs config should work on your forwarder (regular).

lets take a step back, you can easily manage thousands of forwarders and scale by using deployment server. some reading material here: http://docs.splunk.com/Documentation/Splunk/6.5.2/Updating/Aboutdeploymentserver
can you elaborate on the considerations for Heavy / intermediate forwarder?
cheers

[fab73]

Hi adonio, index segregation should work but it does not work for me, something should be checked in my general architecture and configuration, and I'm wondering what....

Have a look to my comment to Giuseppe below also: my Intemediate forwarder does not perform indexing, because indexAndForward=false.

So I'm wondering if the

[input]
index = my_index

in inputs.conf is working in this condition. Also if I change indexAndForward=true, I doesn't make difference.

Intermediate Forwarder has several advanages:

1-reduce security rules to send data out from Customer to Cloud
2-aggregate all filetring into one instance (optimization of licence consumption)
3-aggregate segregation rules to one instance (if it would work!!!!!! :-))

no need for a deploiment server at the moment.

[adonio]

take a look at that answer: https://answers.splunk.com/answers/211403/how-to-configure-inputsconf-and-outputsconf-on-the.html
keep inputs.conf in regular forwarders as follow:

[monitor://<your/path/to/data>]
sourcetype = <your_sourcetype>
index = <your_index>

BTW, i disagree with advantages 2 and 3 - these are just aren't true but this is for another time

[fab73]

thanks adonio for your comments. For what I can see I have no choice, I have to operate at Forwarder tier. I have to good into deep about deployment server for sure in the future.