We are using the add-on, and it was working fine until suddenly, one of the cloudtrail inputs was no longer logging data. splunkd.log gives us this: 02-06-2019 16:25:55.018 +0000 WARN TcpOutputProc - Pipeline data does not have indexKey. [_path] = /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudwatch.py\n[_raw] = \n[_meta] = punct::\n[_stmid] = fJyw3RpcQ9LtT7\n[MetaData:Source] = source::aws_cloudwatch\n[MetaData:Host] = host::REDACTED\n[MetaData:Sourcetype] = sourcetype::aws_cloudwatch\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_conf] = source::aws_cloudwatch|host::REDACTED|aws_cloudwatch|\n The index does exist, and has data, it just isn't writing anything new. We have another cloudtrail input, going to a different index, that works fine. This is probably related: 02-06-2019 16:22:26.214 +0000 INFO IndexWriter - Creating hot bucket=hot_v1_27, idx=aws_REDACTED, event timestamp=1549466074, reason="suitable bucket not found, number of hot buckets=0, max=3" 02-06-2019 16:22:26.215 +0000 INFO DatabaseDirectoryManager - idx=aws_REDACTED Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/aws_REDACTED/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=aws_REDACTED~27~8090F922-ECEC-4FF0-A4D8-99F6464206E1' ... View more

We have our dnsdebuglog turned on and I want to create a summary search of # of events in descending order. Results would look like this: server 1 700,000 server 2 600,000 server 3 444,000 etc Currently, we run a scheduled search every 24 hours (at midnight), but I would like to replace that with the summary search. My attempts so far to make this work have resulted in there being approximately half as many events in the summary search as the scheduled search. Here is what I have: Scheduled search (we only have 75 servers this applies to): index="dns" sourcetype="dnsdebuglog" | top host limit=500 Summary search (run hourly): index=dns sourcetype=dnsdebuglog | sistats count by host Scheduled summary search: index=summary search_name=summary_dnsdebuglog | stats count by orig_host | sort -count What am I doing wrong? ... View more

I need to change the TRUNCATE= value to a higher one as I'm getting truncate warnings in my events. However, we have numerous props.conf files - several for the different apps, the default, and the one that is pushed to all of our forwarders and indexers from our deployment server. I could manually go in and change them all, but I want to figure out which one takes precedence. The log entry, as far as I can tell, doesn't tell me which app (if it is, in fact, an app) that is causing the truncation issue. 08-14-2012 08:22:51.849 -0700 WARN LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded: 10975 Is it the event that occurred directly before this error? We were told that the props.conf files are cumulative, but if you've got four different TRUNCATE= values across 10 props.conf files, how do you know which is being used? ... View more