Looking for best way to set up props.conf: Many Universal Forwarders, two indexers, one search head.

[Ricapar]

My environment looks like this:

[Datacenter A]
> Forwarder (many)
> Splunk Indexer & Search Head
[Datacenter B]
> Forwarder (many)
> Splunk Indexer (search peer for Datacenter A Search Head)

Two datacenters, each with many servers running a Universal Forwarder. I have an indexer set up in each datacenter. Forwarders are set to send data to the indexer in their local datacenter. Server in Datacenter A sends to the indexer in Datacenter A, etc.

We designated the Search Head to be running from the same machine as the Datacenter A indexer. As of right now, we don't require a dedicated machine to be the search head. That machine is also the deployment server too.

One thing I haven't quite figured out is this though: When I'm configuring a new sourcetype, where should the props.conf be going?

1. Do the Forwarders need to know about props.conf? I'm guessing no, since they're Universal Forwarders and don't process anything anyways. Right now the Forwarders are just getting inputs.conf, telling it to monitor:// a path, and setting sourcetype=blah.
2. If data of a certain sourcetype is being sent to both indexers, do both of them need to have props.conf configured for that sourcetype? Or is that something for the machine that is designated as the Search Head?
3. If both indexers need to have props.conf, what's the best way to keep props.conf synced between all the indexers?

[sowings · Accepted solution]

Props.conf can contain rules which apply both at parsing time and at search time. Parsing rules include things like TIME_FORMAT, SHOULD_LINEMERGE, TRANSFORMS, etc. Search-time rules are things like your REPORT and EXTRACT rules.

Parsing time properties go to the systems doing the parsing: Heavy Forwarder, Indexer.

Search time properties go to the systems running searches: Search head.

The same props.conf can go to your indexers as well as your search head; the various internals of Splunk will only look for the pieces that apply to their work (i.e., parsing systems will look for parsing rules, but ignore search-time, until or unless that same system is used to search, too).

In your situation if both indexers are receiving data for the same sourcetype, they should both receive the same props.conf. Typically, I see situations where props rules are shipped to all indexers as well as all search head(s), and to any heavy forwarders which might be in play. I'm not aware of any extra overhead if you ship props.conf rules for a sourcetype that is never seen by an indexer / search. Those configs will just conveniently be ignored by that Splunk instance.

[sowings]

Anything that's in system/local (from a props / transforms standpoint) will have to be picked up and placed in a separate app to be served by the deployment server. You're probably aware that system/local configs act as a trump card over anything from an app. To manage these settings going forward, it's best to relocate them out of system/local if you can. It sounds your environment is small enough to make that easily feasible.

[Ricapar]

Thanks! That does help.

What's the best way to go about getting that props.conf to all of the indexers though?

In the case of the props.conf being for a specific app, that's easy to do using the deployment server. What about for stuff in $SPLUNK_HOME/etc/system/local though?