Should my Heavy Forwarder also get a props.conf copy that is currently at Splunk Cloud Search head?

[iyersudh]

We have recently migrated from On-prem to Splunk cloud. 

Current setup is : UFs ( several of them) --> 2x HFs --> Splunk Cloud

On one of the UF hosts, my outputs.conf is currently configured to send events to both the ON-prem Indexer ( for business reasons) and the HF. This is so that, the events are seen on both On-prem indexer and Cloud indexer for a short period of time until On-prem is retired completed.

When I query on the index on  on-prem and cloud for the same time interval, I see my events breaking is interpreted differently on Cloud compared to on-prem even though both search heads have the same props.conf for my source type. For example, for a particular search string on both search heads on the same index, i get different event counts. Also, on the cloud, my events are not breaking as per the regex defined for line breaking . And some events are completely missing on Cloud even though I can see them  on on-prem search head.

Props.conf snippet:

DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 12
NO_BINARY_CHECK = true
TIME_FORMAT = %H:%M:%S.%3N
category = Custom
pulldown_type = 1

Do I need to be putting a copy of props.conf on HF also? If yes, must I put every props.conf of every source type in HF? Or only under certain specific conditions?

[iyersudh]

Thanks @PickleRick  and @venkatasri .

I have multiple custom apps on Splunk Cloud and each have their own props.conf for various sourcetypes.

To copy the props.conf into HF, 

1) should I have a single props.conf in /etc/system/local/ combining all the props.conf from the indexer OR 2) should I have the different props.conf ( one per app ) deployed into the HF under their respective apps?

I realized @PickleRick  mentioned "...its not about the files but the settings contained within..."  so I interpret that to mean that I can have a single props.conf containing a union of all different props.conf that current sit on my indexer.

[PickleRick]

In case of "global" (i.e. input) settings, you don't have the app context and permissions to take into account so indeed purely theoretically speaking, you could combine settings from all props.conf occurences and put them into a single file. And do the same with transforms.conf and so on.

Technicaly - it should work. But it would be annoyingly hard to maintain.

With search-time settings it gets more complicated because effective settings are calculated with regard to app permisions for a given user and then proper inheritance order. So you can't just make one global props.conf or transforms.conf unless you only use admin role in your environment.

[venkatasri · Accepted solution]

Hi @iyersudh 

the props conf snippet you have provided works at parsing time/ just before indexing pipeline so they meant's to go to HF.

Regarding 1) & 2) , if you have props separated into apps just keep them in apps and deploy to HF you don't need to combine them into single file. There is no harm in merging them into one file it's just don't necessary/mandatory.

If you copy the apps that would be easy so you won't miss anything for example if props  referring to transforms etc..

--

An upvote would be appreciated if this reply helps!

[PickleRick]

It's not about files as such but about the settings that are contained within. The props.conf and transforms.conf can contain settings relevant to index-time or search-time. The settings pertinent to search-time are needed on search head(s). The settings applied at index-time are needed... yes, you guessed it, on components performing the ingestion.

The docs for an app typically specify where you should install it (search heads or hfs/indexers). You can usually (there might be exceptions especially if the app is badly written and contains - for example - scripted inputs enabled by default)  install the app even if part of its settings is not applicable on given component - those settings will simply be ignored.

As has been already said - the first "heavy" splunk component in the path (HF or indexer) performs the event breaking, date parsing, and so on so it needs to know how to do it.

So if you send events directly to indexer, you need your index-time parsing settings there. If you send it to hf, you need it there.

[venkatasri]

Hi @iyersudh 

Technically yes the line_breaking happens on first Splunk Enterprise instance in your architecture, if UF is pointed to Cloud indexers directly then you shall deploy props conf there. 

if Cloud indexers fronted HF  where UF connects to you shall deploy props conf on HF.

--

An upvote would be appreciated if this reply helps!