topic Can some one explain what are aggregation issues? in Getting Data In

Can some one explain what are aggregation issues?

snallam123 — Sat, 14 Mar 2020 04:15:00 GMT

we on-boarded an application recently, Now we are seeing there are 100K aggregation issues(Log level= WARN) and 30k timestamp issues(Log Level=WARN) yesterday from one source, we are monitoring that source from last 10 days. we have similar events and formatting.
The maximum number of events coming from that source is not more than 5k per day
Do i need to ignore these Warnings? What will cause these issues? will it affect our environment? I don't know where to start looking from.. Can some one help!

Thank you for support Splunkers!!!

Re: Can some one explain what are aggregation issues?

richgalloway — Sat, 14 Mar 2020 15:12:27 GMT

Please share the log messages.

Re: Can some one explain what are aggregation issues?

snallam123 — Wed, 30 Sep 2020 04:34:13 GMT

Sample log:
Aggregation sample logs:
03-14-2020 20:07:27.829 -0400 WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded - data_source="SOURCE_NAME", data_host="Host_name", data_sourcetype="sourcetype_name"

Timestamp parsing issues:
03-14-2020 20:09:24.832 -0400 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event (Wed Mar 4 19:09:40 2020).

All Aggregation/timestamp parsing issues are giving us same logs as above.

Re: Can some one explain what are aggregation issues?

richgalloway — Sun, 15 Mar 2020 13:04:53 GMT

The aggregator message is caused by an event that is longer than 256 lines.
The timestamp message is caused by events that do not contain a timestamp in the first 128 characters.
Both messages are caused by incorrect (or missing) props.conf settings for the sourcetype. Verify there is a [sourcetype_name] stanza in one of your props.conf files and that the settings are correct for the data in that sourcetype.
If you would like help with the settings, please share some log data.

Re: Can some one explain what are aggregation issues?

snallam123 — Mon, 16 Mar 2020 07:21:17 GMT

Thanks a lot @richgalloway for the answer, For these logs there is no props, We need to define based on log, Can you please clarify these
Does these issues really affect indexing?
Does it impacts any performance or increase queues?
What might be the impact from these issues.

Re: Can some one explain what are aggregation issues?

richgalloway — Mon, 16 Mar 2020 12:14:43 GMT

Yes, these issues really affect indexing. Events may be combined or split or truncated unpredictably. Events without a timestamp will be assigned a timestamp that may not accurately reflect when the event occurred. Your search results may be unreliable.
There is a minor performance degradation. Specifying props for a sourcetype means Splunk does not have to guess about what props to use, which improves performance. Every sourcetype you ingest should have props specified.