- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Re: Why is the monitoring of 350GB of gzip files v...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Why is the monitoring of 350GB of gzip files via a batch job running slow, with the forwarder sending data at only 1MB/s?
I am trying to monitor via a batch job, approx 300 gzip files and each file uncompressed is about 4GB. and it was about 350GB gzipped. The forwarder all this is occurring on is only running at 10% CPU on a single core of an 8 core box, 25 IOPS, 10% memory and only forwarding the data at about 1MB/s
This is substantially slow and I am questioning why it is only using 1 core on this box, and not using more resources to perform these batch jobs. I have already changed the [thruput] to = 0 but nothing has seemed to work.
Please let me know what if anything I can do to speed up this process. It has been running for 28 hours and only processed about 120GB of those files.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hard to now what the exact problem is, but I would start by checking the splunkd logs for the forwarder and indexer to see if you have hit your limits.
Also when you say you have [thruput]=0, do you mean you're limits.conf is
[thruput]
maxKBps = 0
?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sorry, Yes. maxKBps=0 is there.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Cool. So check those splunkd logs (hopefully you have them available in index=_internal). Also I don't suppose you have any licensing issues processing that much data?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No licensing issue, the license is for pretty large. But this is the issue i just noticed in the _internal index:
IndexConfig - Max bucket size is larger than destination path size limit. Please check your index configuration. idx=MyIndex; bucket size in (from maxDataSize) 10240 MB, homePath.maxDataSizeMB=10000, coldPath.maxDataSizeMB=0
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Have a look at the config options for maxDataSize (http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Indexesconf). You should have it set to auto_high_volume. Not 100% sure if thats the heart of the issue but that is probably something you should change.
Have a read of this too: http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since my environment is SplunkCloud, i would need to open a ticket to check and change that configuration, would't I?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm not sure of the differences with SplunkCloud. I would definitely get in touch with Splunk.
Splunk Smartness with Brandon Sternfield | Episode 3
Monitoring Postgres with OpenTelemetry
Mastering Synthetic Browser Testing: Pro Tips to Keep Your Web App Running Smoothly
