How to whitelist or blacklist a files inside .zip file in Splunk monitoring

[Jagadeesh2022]

Hi Friends,

My current situation is: 

I'm monitoring the files from this path:   source="/opt/redprairie/prod/prodwms/les/log/SplunkMonitoring/*"

In this path we receive 2 different .zip files.

1.support-prodwms--<date & time>.zip

2. commandUsage_<date & time>.csv

I want to monitor the first file (support-prodwms--<date & time>.zip). In side the zip file we have 15 different files. 

1.probes.csv

2. tasks.csv

3.jobs.csv

4.log-files.csv

so on......

Exactly I want to monitor only (2. tasks.csv & 3.jobs.csv) files from zip. remaining files I should not monitor. 

Currently I'm using in input.conf:

[monitor:///opt/redprairie/*/*/les/log/SplunkMonitoring/support-prodwms--*]
index = pg_idx_whse_prod_events
sourcetype= SPLUNKMONITORINGNEW
whitelist = /tasks\.csv$
crcSalt = <string>
recursive = true
disabled = false
_meta = entity_type::NIX service_name::WHSE environment::PROD

Kindly help me friends. I'm struggling for last 2 days on this topic.  Thanks in advance. 

@gcusello @richgalloway @splunk 

[richgalloway]

By "expand the ZIP file into a monitored directory" I mean to create a script/cron job that unzips the zip file into a directory that Splunk is monitoring.  Ideally, the script would extract only the needed files  (tasks.csv and jobs.csv).

The directory would be monitored using these settings

[batch:///opt/redprairie/*/*/les/log/SplunkMonitoring/support-prodwms--unzip]
move_policy = sinkhole
index = pg_idx_whse_prod_events
sourcetype= SPLUNKMONITORINGNEW
whitelist = (tasks|jobs)\.csv$
crcSalt = <string>
recursive = false
disabled = false
_meta = entity_type::NIX service_name::WHSE environment::PROD

A batch input automatically deletes the source file after it has been read.

My challenge is in that particular zip file we have 10+ files. If we monitor that particular zip file Splunk not index the whole data from that zip file.

This is why I suggest expanding the zip file into a separate directory.  The script or utility that does the extraction can pull out only the desired files.

I believe splunk index do based on alphabetic order. it taking too much time to load all the data from particular zip file.

only few files ingest to splunk index when file received in the path. Remaining files ingest very slowly and its started visible after 24 hours only. 

Is the any best way to load all the data from zip file into splunk index on time. 

Ex: we receive zip file every 5 min. before we receive new file we should ingest all the files data into splunk index. How to achieve this one.

How large are the ZIP files?  Speed of ingestion is directly related to the size of the file.  Also, the maxKBps setting in limits.conf controls how fast data is read from disk.  When input is slow, that setting should be checked.

[Jagadeesh2022]

Hi @richgalloway 

Thanks for your reply. 

I have verified in my Deployment server and find the below configurations:

/etc/deployment-apps/PG_WHSE_PROD_Send_To_SEA_HF/local$ cat limits.conf

[thruput]
maxKBps = 0

I'm not sure each Universal forwarder have this setting but I found in DS.

PG_WHSE_PROD_Send_To_SEA_HF - One of the Heavy forwarder. 

[richgalloway]

Allow and deny lists apply to files being monitored and not to any files within those files.

As a workaround, consider expanding the ZIP file into a monitored directory or write a scripted input that extracts the desired files and writes their contents to stdout.

[Jagadeesh2022]

Hi @richgalloway 

Thanks for the reply. 

Could you please explain in details how to achieve your first workaround ? 

consider expanding the ZIP file into a monitored directory.

My challenge is in that particular zip file we have 10+ files.  If we monitor that particular zip file Splunk not index the whole data from that zip file. 

I believe splunk index do based on alphabetic order. it taking too much time to load all the data from particular zip file.

only few files ingest to splunk index when file received in the path. Remaining files  ingest very slowly and its started visible after 24 hours only. 

Is the any best way to load all the data from zip file into splunk index on time. 

Ex: we receive zip file every 5 min. before we receive new file we should ingest all the files data into splunk index. How to achieve this one. 

Thanks in advance.