Why does props.conf for eventline break sometimes and works sometimes?

[Rajiv_splunk]

HI All,

I am stuck with one issue for a event line breaking. I have an environment in which UF sending logs from HF to IDX. I created props.conf in HF and Indexer both. and it worked. The stanzas i provided in the props.cong is working exactly the way i wanted, but after sometime, some of the events appears in unstructured format again...and after sometime it worked exactly the way i want.

Its kind of a bizarre scenario where when i am testing it works fine just at the time of presenting it to the client it stopped working again...and after few minutes it started working fine again :P.

Can anyone help me in finding the root cause for inconsistent performance of props.conf

Note- The stanzas are fine and its breaking the events the way i want but something is stopping it to work consistently

[Rajiv_splunk]

@woodcock Thanks for your reply. The issue is fixed. My team has recently created one new HF which i was not aware of(could be because was on leave). So i added my props in that newly added HF and issue is fixed

[VatsalJagani]

@Rajiv_splunk - If the description of your question is accurate then I think the props.conf is not complete, meaning for those events it's not working it has be modified.

Can you please share your props.conf configuration, events that breaks fine, and events that do not break properly?

(You can mask the sensitive information from the events)

[woodcock · Accepted solution]

Here are some things to check.
1: props.conf is not on all indexers.
2: props.conf is not on HFs (some events go indexer-direct, others go through HF).
3: Some events have different <spec> values that your props.conf stanza header (e.g. if you are using host-based, some events are from a host that you are not expecting).
4: You have timestamp problems such that previous messed up events were thrown into the future and are trickling back to now and inter-mingling with newer correctly-linbroken events.  To check for this, do an All-Time search but use "_index_earliest=-1h _index_latest=now" with your search.

[Rajiv_splunk]

HI Thanks for the reply. I checked all the points you mentioned.

props.conf is in all indexer and in all HF

I am using sourcetype based configuration and all the logs are coming from the same sourcetype and same host.

Can you help me with the 4th point you mentioned-  when i am running this query across all time its returning nothing. I am using this query like below

_index_earliest=-1h _index_latest=now sourcetype=mysourcetype.

Is there any other way i can check if earlier messed up events are combining

[woodcock]

index="YourIndexHere" AND sourcetype="YourSourcetypeHere" _index_earliest=-1h _index_latest=now  earliest=0 latest=+5y