Index created in Cluster Master not showing in Heavy Forwarder

[Karthikeya]

Hi, 

We have configured a data input in HF and there is an option to select index there. I have created new index in Cluster master and pushed it to indexers. But that created index is not showing in HF. I believe HF is not linked in this cluster that is why it is not showing. What to do in this case? I tried to create same index in HF but our hot and cold path contains volumes which is failing to create index in HF. Please help me what can I do? If I keep default index in HF... will it pick the index in indexers? How to configure this? Please clarify my confusion here....

[gcusello]

Hi @Karthikeya ,

it's normal: in the indexes list, you see only indexes locally created.

You have two choices:

• create the index also on HF even if you don't use it, only to see it in dropdowns choice lists;
• manually add the index in the conf files on Hevy Forwarders.

Ciao.

Giuseppe

[Karthikeya]

Not able to create same index in HF it is throwing this error.

• manually add the index in the conf files on Hevy Forwarders. ---> Where to do this for particular data input? Please guide me

[gcusello]

Hi @Karthikeya ,

on the indexers you have the Volume thats not relevant on HFs: remove the volume defintion and use

$SPLUNK_DB/app_juniper_dev/db

$SPLUNK_DB/app_juniper_dev/colddb

Ciao.

Giuseppe

[Karthikeya]

But in indexers we have volumes and here it cannot be configured. Is if fine? finally data input takes which index? HF or one which is there on indexer?

[gcusello]

Hi @Karthikeya ,

as I said, indexes on HFs is a workaround to have the indexes list in dropdowns, but it isn't really used, so you can use every path you like.

It's obviously different on Indexers where Volume is relevant.

Ciao.

Giuseppe

[Karthikeya]

so finally data inputs from HF finally takes the index which is created on indexers right? Please clarify.

[gcusello · Accepted solution]

Hi @Karthikeya ,

you must create Indexes on Indexers using the Cluster Manager and it's relevant.

Then, only to see the indexes in the dropdown lists on HFs, you can create Indexes also on HFs but it's a workaround they aren't used.

You must only put attention that the names are the same.

Ciao.

Giuseppe

[PickleRick]

To add a bit to this - otherwise good and valid - answer...

Every Splunk component uses local indexes.conf instance. So if you have a search head, it uses its own indexes.conf file to see which indexes it can autofill for you, if you have a HF, only those indexes which are in indexes.conf on that HF are shown in the "create input" dialogs.

If a component does not do local indexing, those entries are - apart from the component being able to show those indexes - meaningless since... well, the component does only forward events "to outside" but it has to know about those indexes to show them. In no situation splunk "reaches out" anywhere for a list of indexes.

[Karthikeya]

@PickleRick what if I have indexes.conf in HF etc/system/local and in cluster manager etc/master-apps/<app_name>/local... Which index will splunk take finally when going to SH? 

Data will forward from HF to indexer... 

[PickleRick]

None of them.

As I said - each instance has its own indexes.conf (or set of separate index.conf files which are merged according to the precedence rules - https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles )

So the indexes.conf from CM will get pushed to indexers and indexers will use their own local copies of that file, SH has its own indexes.conf (might have been pushed from SH deployer) and HF has its own one (possibly coming from an app from DS).

But in a well-deployed Splunk installation HFs and SHs would not index anything locally so even though the indexes would be defined on them, they would be empty since all events would be pushed to outputs only.

[Karthikeya]

@PickleRick 

But in a well-deployed Splunk installation HFs and SHs would not index anything locally so even though the indexes would be defined on them, they would be empty since all events would be pushed to outputs only. ---> I didn't get you. 

What do you mean by locally? etc/system/local folder?

Our HF outputs.conf is configured to CM ip address and from there HF will forward data to indexers. 

[PickleRick]

No. I mean that in a properly deployed installation all components except indexers should have indexAndForward=false setting so they don't do any indexing on their own - everything should be forwarded to indexers (with an exception for some internal data needed by deployment server which should selectively be indexed locally but that's another story).

EDIT: Wait, what? "Our HF outputs.conf is configured to CM ip address and from there HF will forward data to indexers". You mean that you use indexer discovery and CM returns a list of indexers to which your HF forwards the data? That's ok. Still it has nothing to do with indexes themselves.

[isoutamo]

Except newer DS which must have it like this

[indexAndForward]
index = true
selectiveIndexing = true

On HF side it's actually depends on TA are those indexes needed on that server or not. At least some e.g. DBX have this field as text where you could just write those names instead of select those from popup list.

[Karthikeya]

Yes indexery discovery I mean.

So finally the index which is present in indexers will be used right?

What if we don't configure indexAndForward=false in other components? This is by default false??

[PickleRick]

I don't understand the phrase "index will be used" in this context.

OK. A bit simplified indexing and forwarding process

As an event comes in, it arrives on an input. It might have a destination index already associated with it if it comes from a HF and is already parsed or is explicitly desined for an index by HEC input. Otherwise it will get assigned a destination index either explicitly configured for an input or a default one.

That's what's happening on the input side.

Now we have two things which can happen in parallel.

1. The event is supposed to be indexed locally (we're on an indexer, indexAndForward=true) - the indexing part of Splunk searches _this instance's_ configuration for an existing index matching the one assigned to the event. If it finds one, it indexes an event into this index. If it doesn't it either puts the index into last chance index if one is configured or drops the event completely (possibly emitting a warning into splunkd.log).

2. There are outputs configured on the box - splunk sends the event to a configured output along with its metadata (destination index among them). But the metadata in this case is just a text label attached to an event. It doesn't have to correspond to anything on the receiving end.

[Karthikeya]

There are outputs configured on the box - splunk sends the event to a configured output along with its metadata (destination index among them). --> Here outputs means indexers in my case... HF to IND and events should take indexers index as destination index. But I have created identical index in HF as well but with different path.

[isoutamo]

When your hf isn’t indexing events, it doesn’t take care anything what you have defined in indexes.conf. Only place where it use that conf file is give you list of index names from you could choose index where events are stored on your indexers.

Personally I didn’t configure any index definitions on HFs unless there are any modular inputs, which wants those in GUI and I cannot configure those by .conf files.

[Karthikeya]

@isoutamo thank you so much. You guys are like pillars to this community. We freshers are so happy to be here. 

[PickleRick]

Ok. Let's try to use an overly complicated example.

Let's assume you have a setup like this (fairly unrealistic but shows many options)

(1) UF (reading a file) -> (2) intermediate UF -> (3) HF -> (4) indexer -> (5) another indexer

1. The initial UF has input defined as follows:

[monitor:///var/log/somelog.txt]
sourcetype=custom:sourcetype
index=testindex1

So the initial UF reads the file /var/log/somelog.txt and sends data from it. It's a UF so it doesn't yet break the data into single events, it sends the data in chunks (so called cooked data). Each chunk has the following metadata attached to it:

source = /var/log/somelog.txt
sourcetype = custom:sourcetype
index = testindex1
host = yourhostname

2. Another UF (an intermediate UF) receives the data on its input. Since it's a UF it doesn't do parsing, it doesn't do any indexing, it just sends the data to its output(s) as it received it (still only cooked) so the metadata stays as it was.

The data is sent to a HF.

3. A HF is just a name for a Splunk Enterprise instance which does mostly forwarding and does not do any local indexing (has indexAndForward=false). So the HF receives the data on its input.

Since it's a full Splunk Enterprise instance, the data is getting parsed which means it goes through all stages like line breaking, timestamp recognition, custom transforms.

From this point on your data is processed as "cooked and parsed" (often called "parsed" for short) so each event is processed separately and each event has its own set of metadata. In your case it's at least (we're not digging into how the timestamp is getting assigned at this point):

source = /var/log/somelog.txt
sourcetype = custom:sourcetype
index = testindex1
host = yourhostname
_time = <some timestamp assigned to an event>
_raw = <raw data of your event>
linecount = <number of lines if your original event was multiline>

(to be fully honest, I'm not 100% sure if linecount is added at this point or at indexing).

Also for some events the metadata could have been additionally altered by transforms but let's not touch that at the moment.

Since you're not doing any indexing locally, the event is only sent through an output.

The HF at this point doesn't care about the index metadata attached to an event. It just sends it away.

4. Now an indexer receives the event. Since the event is received as parsed data (notice that I'm no longer saying that you receive a chunk of data; you receive a particular event) there is no parsing performed at this point. We're dealing with already parsed data so we don't mangle it anymore (with possible exception for ingest actions but that's not what we want to talk about here). The indexer does have local indexing enabled so it tries to write the event to the index. The metadata attached to the event says

index = testindex1

So the indexer searches whether it does have an index called "testindex1" defined in its configuration. If it does, it indexes the event. If it doesn't it drops it or sends to a last chance index if one is defined.

But also, as the indexer has output defined, sends the same event to the next indexer.

5. The last step in our path - another indexer - receives an event. It's parsed data so we don't parse it again. And if we have an index called "testindex1" defined, we write the event to that index. If not - again, either drop the event or send it to last chance index.

What's important here is that only indexers at step 4 and 5 actually care about the contents of the "index" metadata field. And they do so only while indexing the data. For forwarding it's always just a label glued on to a chunk of data or an individual event (depending whether we're talking about cooked or parsed data). A forwarder has no knowledge beyond

1. The metadata that the preceeding component in the path added to the chunk/event

2. The destination output

It doesn't know nor does it care about whether the output is an indexer or another forwarder (either heavy or universal).

[splunklearner]

@PickleRick @isoutamo I have gone through all the thread.

I am configuring Akamai add-on in my environment to get akamai logs. We have installed this add-on on our HF and sending that data to indexers (CM which configured indexer discovery). I think it will come under modular inputs. I have created an index in CM and pushed it to indexers. Now in add-on if I keep main index (which is showing in drop-down in that data input) and forward the logs to indexers, how will indexers pick the desired index (which is created) for these data input (akamai) logs? Where to configure this? This data input will not have any log path right to configure it in inputs.conf? Bi.t confused on this. Can you please clarify?

This app came with inputs.conf in default and this is how it is:

[TA-AKAMAI_SIEM]

index=default

sourcetype=akamaisiem

interval=60

This app not pushed to indexers only HF it is there.