Getting Data In

After creating and setting a new index in inputs.conf on the universal forwarder, why do I see no search results on the search head?

nce054
Path Finder

I'm trying to create a new index called 'winevents_endpoint'. I've added this index to the Search Head, Indexer, and Heavy Forwarder (not sure if it's needed on all of them?). When I set an input in inputs.conf on the Universal Forwarder, I set the index to 'winevents_endpoint'.

[WinEventLog://Security]
disabled = 0 
index = winevents_endpoint

However, if I search 'index=winevents_endpoint' on my Search Head, nothing comes up. Does anyone know what could be the issue? I can see that when I changed the index from 'main' to 'winevents_endpoint', the Universal Forwarder stopped sending stuff (AKA the index of winevents_endpoint isn't making it through, but it took effect).

1 Solution

woodcock
Esteemed Legend

You also need to create the new index in indexes.conf on all of your indexers:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Indexesconf

View solution in original post

woodcock
Esteemed Legend

You also need to create the new index in indexes.conf on all of your indexers:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Indexesconf

nce054
Path Finder

I see. I was under the impression that doing it through the Web interface was sufficient. There are paths that I have to designate, for coldPath, homePath, etc.
[main]
homePath = $SPLUNK_DB\defaultdb\db
coldPath = $SPLUNK_DB\defaultdb\colddb
thawedPath = $SPLUNK_DB\defaultdb\thaweddb

Do I just declare them, like homePath = $SPLUNK_DB\winevents_endpointdb\db , and they create themselves? Or do I need to go deeper and create these?

0 Karma

woodcock
Esteemed Legend

You need to prepare each of your indexers such that the amount of space that you are telling Splunk to use and where you are telling it to use it actually exists. Then you need to deploy indexes.conf to each indexers and restart each Splunk instance. Then, yes, Splunk will create it's proprietary DB in that spot. Also, you need to change main in your stanza header to winevents_endpoint.

nce054
Path Finder

Okay, I've now added the indexers.conf to each etc\system\local on my two Indexers, and added the correct stanza, where main is now winevents_endpoint. The only thing I do not understand is why I need to push inputs.conf to the Indexer; isn't that only configured on the Universal Forwarder? My inputs.conf on my Indexers looks like `[default]
host = HOUMSMGTLOG301

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
`

0 Karma

woodcock
Esteemed Legend

I accidentally wrote inputs.conf when I meant indexes.conf (I went back and changed it). You need to put indexes.conf on your indexers and inputs.conf on your forwarders. The indexer has the physical DB that contains the events that are in each index. Think about it; you need a bunch of disk space somewhere, right?

nce054
Path Finder

It's also worth noting that I have these events going through a filter on my Heavy Forwarder. However, to my knowledge, this shouldn't be changing the index. It goes as follows:
props.conf

[source::WinEventLog...]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = events-null, events-filter

transforms.conf

[events-filter]
REGEX=(?msi)^EventCode=(7|100)
DEST_KEY = queue
FORMAT = indexQueue

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

nce054
Path Finder

Right, the Indexers are where the DB's are that the Search Head looks into when it is performing a search. So I've configured indexes.conf on both of my Indexers. Now, when I have a stanza like [WinEventLog://Security]
disabled = 0
index = winevents_endpoint
on my inputs.conf on my Universal Forwarder, it should send it to the DB purposed for the index of winevents_endpoint. However, on my Search Head, the WinEventLog://Security events are still showing up with the index main.

0 Karma

woodcock
Esteemed Legend

Are you sure you are not looking at old events? Changing indexes.conf and inputs.conf now will not change where the older events went; they will still be in main. What was the output when you restarted Splunk on one of your indexers (you did restart Splunk, right)? If there was a problem with your configuration, it should have complained about it. Also try this search on your search head:

 index=_internal err* OR warn* OR winevents_endpoint

nce054
Path Finder

Got it working!! I had forgotten to restart the Universal Forwarders. I thought that pushing the configuration files to them was sufficient. Thanks for your help, woodcock. Slowly but surely, you're getting all of my coins 🙂

0 Karma

nce054
Path Finder

Yes, I am looking at new events that are still going to main. That search yielded some warnings for issues other than what I'm dealing with (other hosts that don't have the index I'm working on). There was no output when I restarted Splunk. I restarted the Search Head, Indexers, and Heavy Forwarder.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...