topic Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head? in Getting Data In

After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054 — Mon, 28 Sep 2020 20:20:17 GMT

I'm trying to create a new index called 'winevents_endpoint'. I've added this index to the Search Head, Indexer, and Heavy Forwarder (not sure if it's needed on all of them?). When I set an input in inputs.conf on the Universal Forwarder, I set the index to 'winevents_endpoint'.

[WinEventLog://Security]
disabled = 0 
index = winevents_endpoint

However, if I search 'index=winevents_endpoint' on my Search Head, nothing comes up. Does anyone know what could be the issue? I can see that when I changed the index from 'main' to 'winevents_endpoint', the Universal Forwarder stopped sending stuff (AKA the index of winevents_endpoint isn't making it through, but it took effect).

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

woodcock — Tue, 23 Jun 2015 13:31:21 GMT

You also need to create the new index in indexes.conf on all of your indexers:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Indexesconf

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054 — Mon, 28 Sep 2020 20:20:20 GMT

I see. I was under the impression that doing it through the Web interface was sufficient. There are paths that I have to designate, for coldPath, homePath, etc.
[main]
homePath = $SPLUNK_DB\defaultdb\db
coldPath = $SPLUNK_DB\defaultdb\colddb
thawedPath = $SPLUNK_DB\defaultdb\thaweddb

Do I just declare them, like homePath = $SPLUNK_DB\winevents_endpointdb\db , and they create themselves? Or do I need to go deeper and create these?

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

woodcock — Tue, 23 Jun 2015 14:08:53 GMT

You need to prepare each of your indexers such that the amount of space that you are telling Splunk to use and where you are telling it to use it actually exists. Then you need to deploy indexes.conf to each indexers and restart each Splunk instance. Then, yes, Splunk will create it's proprietary DB in that spot. Also, you need to change main in your stanza header to winevents_endpoint.

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054 — Tue, 23 Jun 2015 14:17:25 GMT

Okay, I've now added the indexers.conf to each etc\system\local on my two Indexers, and added the correct stanza, where main is now winevents_endpoint. The only thing I do not understand is why I need to push inputs.conf to the Indexer; isn't that only configured on the Universal Forwarder? My inputs.conf on my Indexers looks like `[default]
host = HOUMSMGTLOG301

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
`

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

woodcock — Tue, 23 Jun 2015 14:31:10 GMT

I accidentally wrote inputs.conf when I meant indexes.conf (I went back and changed it). You need to put indexes.conf on your indexers and inputs.conf on your forwarders. The indexer has the physical DB that contains the events that are in each index. Think about it; you need a bunch of disk space somewhere, right?

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054 — Tue, 23 Jun 2015 14:36:27 GMT

Right, the Indexers are where the DB's are that the Search Head looks into when it is performing a search. So I've configured indexes.conf on both of my Indexers. Now, when I have a stanza like [WinEventLog://Security] disabled = 0 index = winevents_endpoint on my inputs.conf on my Universal Forwarder, it should send it to the DB purposed for the index of winevents_endpoint. However, on my Search Head, the WinEventLog://Security events are still showing up with the index main.

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054 — Tue, 23 Jun 2015 14:46:52 GMT

It's also worth noting that I have these events going through a filter on my Heavy Forwarder. However, to my knowledge, this shouldn't be changing the index. It goes as follows:
props.conf

[source::WinEventLog...]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = events-null, events-filter

transforms.conf

[events-filter]
REGEX=(?msi)^EventCode=(7|100)
DEST_KEY = queue
FORMAT = indexQueue

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

woodcock — Tue, 23 Jun 2015 15:35:13 GMT

Are you sure you are not looking at old events? Changing indexes.conf and inputs.conf now will not change where the older events went; they will still be in main. What was the output when you restarted Splunk on one of your indexers (you did restart Splunk, right)? If there was a problem with your configuration, it should have complained about it. Also try this search on your search head:

 index=_internal err* OR warn* OR winevents_endpoint

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054 — Tue, 23 Jun 2015 15:38:45 GMT

Yes, I am looking at new events that are still going to main. That search yielded some warnings for issues other than what I'm dealing with (other hosts that don't have the index I'm working on). There was no output when I restarted Splunk. I restarted the Search Head, Indexers, and Heavy Forwarder.

Re: After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054 — Tue, 23 Jun 2015 16:05:55 GMT

Got it working!! I had forgotten to restart the Universal Forwarders. I thought that pushing the configuration files to them was sufficient. Thanks for your help, woodcock. Slowly but surely, you're getting all of my coins 🙂