Monitoring Splunk

How to whitelist or blacklist a files inside .zip file in Splunk monitoring

Jagadeesh2022
Path Finder

Hi Friends,

My current situation is: 

I'm monitoring the files from this path:   source="/opt/redprairie/prod/prodwms/les/log/SplunkMonitoring/*"

In this path we receive 2 different .zip files.

1.support-prodwms--<date & time>.zip

2. commandUsage_<date & time>.csv

I want to monitor the first file (support-prodwms--<date & time>.zip). In side the zip file we have 15 different files. 

1.probes.csv

2. tasks.csv

3.jobs.csv

4.log-files.csv

so on......

Exactly I want to monitor only (2. tasks.csv & 3.jobs.csv) files from zip. remaining files I should not monitor. 

Currently I'm using in input.conf:

[monitor:///opt/redprairie/*/*/les/log/SplunkMonitoring/support-prodwms--*]
index = pg_idx_whse_prod_events
sourcetype= SPLUNKMONITORINGNEW
whitelist = /tasks\.csv$
crcSalt = <string>
recursive = true
disabled = false
_meta = entity_type::NIX service_name::WHSE environment::PROD

 

Kindly help me friends. I'm struggling for last 2 days on this topic.  Thanks in advance. 

@gcusello @richgalloway @splunk 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

By "expand the ZIP file into a monitored directory" I mean to create a script/cron job that unzips the zip file into a directory that Splunk is monitoring.  Ideally, the script would extract only the needed files  (tasks.csv and jobs.csv).

The directory would be monitored using these settings

 

 

[batch:///opt/redprairie/*/*/les/log/SplunkMonitoring/support-prodwms--unzip]
move_policy = sinkhole
index = pg_idx_whse_prod_events
sourcetype= SPLUNKMONITORINGNEW
whitelist = (tasks|jobs)\.csv$
crcSalt = <string>
recursive = false
disabled = false
_meta = entity_type::NIX service_name::WHSE environment::PROD

 

 

A batch input automatically deletes the source file after it has been read.

My challenge is in that particular zip file we have 10+ files. If we monitor that particular zip file Splunk not index the whole data from that zip file.

This is why I suggest expanding the zip file into a separate directory.  The script or utility that does the extraction can pull out only the desired files.

I believe splunk index do based on alphabetic order. it taking too much time to load all the data from particular zip file.

only few files ingest to splunk index when file received in the path. Remaining files ingest very slowly and its started visible after 24 hours only. 

Is the any best way to load all the data from zip file into splunk index on time. 

Ex: we receive zip file every 5 min. before we receive new file we should ingest all the files data into splunk index. How to achieve this one.

How large are the ZIP files?  Speed of ingestion is directly related to the size of the file.  Also, the maxKBps setting in limits.conf controls how fast data is read from disk.  When input is slow, that setting should be checked.

---
If this reply helps you, Karma would be appreciated.

Jagadeesh2022
Path Finder

Hi @richgalloway 

Thanks for your reply. 

I have verified in my Deployment server and find the below configurations:

/etc/deployment-apps/PG_WHSE_PROD_Send_To_SEA_HF/local$ cat limits.conf

[thruput]
maxKBps = 0

I'm not sure each Universal forwarder have this setting but I found in DS.

PG_WHSE_PROD_Send_To_SEA_HF - One of the Heavy forwarder. 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Allow and deny lists apply to files being monitored and not to any files within those files.

As a workaround, consider expanding the ZIP file into a monitored directory or write a scripted input that extracts the desired files and writes their contents to stdout.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Jagadeesh2022
Path Finder

Hi @richgalloway 

Thanks for the reply. 

Could you please explain in details how to achieve your first workaround ? 

consider expanding the ZIP file into a monitored directory.

 

My challenge is in that particular zip file we have 10+ files.  If we monitor that particular zip file Splunk not index the whole data from that zip file. 

I believe splunk index do based on alphabetic order. it taking too much time to load all the data from particular zip file.

only few files ingest to splunk index when file received in the path. Remaining files  ingest very slowly and its started visible after 24 hours only. 

Is the any best way to load all the data from zip file into splunk index on time. 

Ex: we receive zip file every 5 min. before we receive new file we should ingest all the files data into splunk index. How to achieve this one. 

Thanks in advance. 

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...