thanks lycollicott. I had the above query and got the output and I also added to find the total amount of data that are being injected last 24 hours host=splunk01* sourcetype=splunkd index=_internal "syslog" | stats sum(kb) by series | addcoltotals labelfield=series for server 01 I am getting total value 17805533.451029 kb almost 17.8GB of data and server 02 total value 17546902.822219 its gain 17.5 GB of data. I think the load are being equally distributed. ... View more

thanks martin, I tried to execute the below query with time frame set as All-time. But I could see that data are present only for two months (From July 16 to current date). Even executed index=_internal* I could see only data available from July 2016 to till now. index=_internal component=bucketmover idx=net_fw splunk_server=splunk03. 1:37:27.302 AM 08-10-2016 01:37:27.302 -0400 INFO BucketMover - idx=net_fw Moving bucket='db_1467094714_1467081929_6749' because maximum number of warm databases exceeded, starting warm_to_cold: from='/splogs/net_fw/db' to='/splogs/net_fw/colddb' host = splunk03 function_splunkindexer production tier_production type_nix webops source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 8/10/16 1:37:27.278 AM 08-10-2016 01:37:27.278 -0400 INFO BucketMover - idx=net_fw Moving bucket='db_1467081941_1467072478_6748' because maximum number of warm databases exceeded, starting warm_to_cold: from='/splogs/net_fw/db' to='/splogs/net_fw/colddb' host = splunk03 function_splunkindexer production tier_production type_nix webops source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 8/10/16 1:37:27.255 AM 08-10-2016 01:37:27.255 -0400 INFO BucketMover - idx=net_fw Moving bucket='db_1467072469_1467058580_6747' because maximum number of warm databases exceeded, starting warm_to_cold: from='/splogs/net_fw/db' to='/splogs/net_fw/colddb' host = splunk03 function_splunkindexer production tier_production type_nix webops source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 8/9/16 10:39:23.905 PM 08-09-2016 22:39:23.905 -0400 INFO BucketMover - idx=net_fw Moving bucket='db_1467058579_1467047364_6746' because maximum number of warm databases exceeded, starting warm_to_cold: from='/splogs/net_fw/db' to='/splogs/net_fw/colddb' host = splunk03 function_splunkindexer production tier_production type_nix webops source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 8/9/16 7:28:41.901 PM 08-09-2016 19:28:41.901 -0400 INFO BucketMover - idx=net_fw Moving bucket='db_1467047363_1467031159_6745' because maximum number of warm databases exceeded, starting warm_to_cold: from='/splogs/net_fw/db' to='/splogs/net_fw/colddb' host = splunk03 function_splunkindexer production tier_production type_nix webops source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd We can see only last two months data's are available when we run the above query. kindly let me know it will show like this if we run the above query. thank in advance. ... View more

yes, you are right, I got this output |dbinspect index=net_fw | search state=cold splunk_server=splunk03 | stats count sum(sizeOnDiskMB) count 2974 1344829.046891 |dbinspect index=* | search state=cold splunk_server=splunk03 | stats count sum(sizeOnDiskMB) count 8644 3437339.870991 ... View more

Martin, after executing the query with time period for 2 year but I am getting no result found. Even tried to remove the search command but still no luck. Regarding permission I hope I am having the admin privilege. |dbinspect index = net_fw | search state=cold splunk_server=splunk03 | stats count sum(sizeOnDiskMB). Regarding the old app "/opt/splunk/etc/apps/all_indexer_base/local/indexes.conf.2013.06.03" should I need to uncomment the entire stanza. thanks in advance. ... View more

thanks Martin, I had run the query which you had shared in the comment, but when tried to execute the same with time frame as All Time, it was throwing this error. "Error in 'dbinspect' command: This command is not supported in a real-time search" So tried to execute by setting the time frame to two years and got "no result found " | dbinspect index=_internal | search state=cold splunk_server=splunk03 | stats count sum(sizeOnDiskMB) You should eventually clean up the four different locations all defining Indexes.conf ? I could not understand this question, so can you tell me what exactly I need to do here. You mean I need to check in the below location [net_fw] homePath = volume:Hot/net_fw/db coldPath = volume:Cold/net_fw/colddb tstatsHomePath = volume:Hot/net_fw/datamodel_summary thawedPath = $SPLUNK_DB/net_fw/thaweddb maxTotalDataSizeMB = 250000 thanks in advance. ... View more

thank Skoelpin, for sharing your experience. I might be doing the same after getting the required permission as its in prod environment. ... View more

thanks Martin. Yes I have checked all the indexes instances individually and found that this particular indexer instance data's are not being deleted "under /splogs/net_fw/colddb/" after reaching the frozen time stamp. when compared with rest four indexer instance we found only data's from May 16 to Jun 16. Are your indexers replicating buckets as part of an indexer cluster? Ours is not cluster environment, In all the Indexes.conf we did not find this stanza RepFactor more over when we ran the command and found that in index setting RepFactor = 0 and also ran this command to ./splunk cmd btool server list --debug shclustering and it returned only default values from /etc/system/default/server.conf for all the indexers servers. $SPLUNK_HOME/bin/splunk cmd btool --debug indexes list net_fw /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf [net_fw] /opt/splunk/etc/system/default/indexes.conf assureUTF8 = false /opt/splunk/etc/system/default/indexes.conf blockSignSize = 0 /opt/splunk/etc/system/default/indexes.conf blockSignatureDatabase = _blocksignature /opt/splunk/etc/system/default/indexes.conf bucketRebuildMemoryHint = auto /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf coldPath = volume:Cold/net_fw/colddb /opt/splunk/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf coldToFrozenDir = /opt/splunk/etc/system/default/indexes.conf coldToFrozenScript = /opt/splunk/etc/system/default/indexes.conf compressRawdata = true /opt/splunk/etc/system/default/indexes.conf defaultDatabase = main /opt/splunk/etc/system/default/indexes.conf enableOnlineBucketRepair = true /opt/splunk/etc/system/default/indexes.conf enableRealtimeSearch = true /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf frozenTimePeriodInSecs = 31536000 /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf homePath = volume:Hot/net_fw/db /opt/splunk/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10 /opt/splunk/etc/system/default/indexes.conf indexThreads = auto /opt/splunk/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d /opt/splunk/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0 /opt/splunk/etc/system/default/indexes.conf maxConcurrentOptimizes = 6 /opt/splunk/etc/system/default/indexes.conf maxDataSize = auto /opt/splunk/etc/system/default/indexes.conf maxHotBuckets = 3 /opt/splunk/etc/system/default/indexes.conf maxHotIdleSecs = 0 /opt/splunk/etc/system/default/indexes.conf maxHotSpanSecs = 7776000 /opt/splunk/etc/system/default/indexes.conf maxMemMB = 5 /opt/splunk/etc/system/default/indexes.conf maxMetaEntries = 1000000 /opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroups = 8 /opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1 /opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300 /opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60 /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf maxTotalDataSizeMB = 250000 /opt/splunk/etc/system/default/indexes.conf maxWarmDBCount = 300 /opt/splunk/etc/system/default/indexes.conf memPoolMB = auto /opt/splunk/etc/system/default/indexes.conf minRawFileSyncSecs = disable /opt/splunk/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000 /opt/splunk/etc/system/default/indexes.conf partialServiceMetaPeriod = 0 /opt/splunk/etc/system/default/indexes.conf processTrackerServiceInterval = 1 /opt/splunk/etc/system/default/indexes.conf quarantineFutureSecs = 2592000 /opt/splunk/etc/system/default/indexes.conf quarantinePastSecs = 77760000 /opt/splunk/etc/system/default/indexes.conf rawChunkSizeBytes = 131072 /opt/splunk/etc/system/default/indexes.conf repFactor = 0 /opt/splunk/etc/system/default/indexes.conf rotatePeriodInSecs = 60 /opt/splunk/etc/system/default/indexes.conf serviceMetaPeriod = 25 /opt/splunk/etc/system/default/indexes.conf serviceOnlyAsNeeded = true /opt/splunk/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30 /opt/splunk/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000 /opt/splunk/etc/system/default/indexes.conf suppressBannerList = /opt/splunk/etc/system/default/indexes.conf sync = 0 /opt/splunk/etc/system/default/indexes.conf syncMeta = true /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf thawedPath = $SPLUNK_DB/net_fw/thaweddb /opt/splunk/etc/system/default/indexes.conf throttleCheckPeriod = 15 /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf tstatsHomePath = volume:Hot/net_fw/datamodel_summary /opt/splunk/etc/system/default/indexes.conf warmToColdScript = What do you see when you go to Settings -> Distributed Management Console -> Indexing -> Indexes and Volumes -> Index Detail: Instance and select the indexer and one of the two indexes? I had checked in the DMC, was unable to find the option --> Indexing -> Indexes and Volume ->Index Details in one of the search head. We are using 6.2.1 version. When you ran the search I gave you, what results did you see? Is the remaining free space shrinking? index=_internal component=bucketmover idx=net_fw (interval 60 min) 08-09-2016 11:09:53.114 -0400 INFO BucketMover - idx=net_fw Moving bucket='db_1466869169_1466854110_2713' because maximum number of warm databases exceeded, starting warm_to_cold: from='/splogs/net_fw/db' to='/splogs/net_fw/colddb' There are 46 Indexes.conf file configured in each of the indexer instance and out of this four Indexes.conf containing this parameter? Not sure why they had created four Indexes.conf for same index. /opt/splunk/etc/apps/ADMIN-all_indexers/default/Indexes.conf [volume:Hot] path = /splogs [volume:Cold] path = /splogs [volume:Base] path = /splogs [default] frozenTimePeriodInSecs = 31536000 [net_fw] homePath = volume:Hot/net_fw/db coldPath = volume:Cold/net_fw/colddb tstatsHomePath = volume:Hot/net_fw/datamodel_summary thawedPath = $SPLUNK_DB/net_fw/thaweddb 2) /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf [volume:Hot] path = /splogs [volume:Cold] path = /splogs [volume:Base] path = /splogs [default] frozenTimePeriodInSecs = 31536000 [net_fw] homePath = volume:Hot/net_fw/db coldPath = volume:Cold/net_fw/colddb tstatsHomePath = volume:Hot/net_fw/datamodel_summary thawedPath = $SPLUNK_DB/net_fw/thaweddb maxTotalDataSizeMB = 250000 -- > Included this stanza /opt/splunk/etc/apps/all_indexer_base/local/Indexes.conf [volume:Hot] path = /splogs [volume:Cold] path = /splogs [volume:Base] path = /splogs [net_fw] homePath = volume:Hot/net_fw/db coldPath = volume:Cold/net_fw/colddb thawedPath = $SPLUNK_DB/net_fw/thaweddb There is no frozentime period is mentioned in this stanza. /opt/splunk/etc/apps/all_indexer_base/local/indexes.conf.2013.06.03 [volume:Hot] path = /splogs [volume:Cold] path = /splogs [volume:Base] path = /splogs [net_fw] homePath = volume:Hot/net_fw/db coldPath = volume:Cold/net_fw/colddb thawedPath = $SPLUNK_DB/net_fw/thaweddb No frozentime period is mentioned in this stanza. I am sure that, going through this much detail will be pain full, Sorry for that. But I wanted to share entire to know what exactly is broken in my environment why splunk is not deleting the buckets. thanks in advance. ... View more

thanks lycollicott, we got this output after executing above query and we hope syslog data are being monitored from both the servers 01 and 02. 8/9/16 10:29:31.392 AM 08-09-2016 10:29:31.392 -0400 INFO Metrics - group=per_sourcetype_thruput, series="bluecoat_syslog", kbps=18.937373, eps=61.547462, kb=587.067383, ev=1908, avg_age=2.618973, max_age=6 host = splunk01.xxxx.comsource = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd 8/9/16 10:29:31.392 AM 08-09-2016 10:29:31.392 -0400 INFO Metrics - group=per_source_thruput, series="/opt/syslogs/webops_security/hostname/ssg.log", kbps=4.225995, eps=21.741609, kb=131.007812, ev=674, avg_age=1.587537, max_age=3 host = splunk01.xxxx.comsource = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd How to monitor the amount of being injected under the path /opt/syslog/* in both the servers 01 & 02. To have proactive monitoring on disk size /opt. ... View more

thanks skoelpin, There are two frozenTimePeriodInSecs = 31536000 are set one is under the default stanza and another one is under summary stanza frozenTimePeriodInSecs = 188697600 as show in Indexes.conf . So which one will be taken in consideration for deleting the data. ... View more

thank Martin for throwing some lights on this issue. We have 5 individual physical server for splunk indexer instance with space of 5.6TB configured in each indexer instance, which in turn pointed to separate mount point /splogs in each indexer instances. Now the problem is only with one of indexer instance which is running out of space. I had executed above query as mentioned in comment, but not sure how to check whether the old data are being removed or not? maxTotalDataSizeMB 250 GB is mentioned only for few indexes in the Indexes.conf file, but how to check whether data are being deleted after reaching the 250GB mark? I mean how and where to check the frozen bucket and there is no archive policy being set in the Indexes.conf file to archive the data. There are two frozenTimePeriodInSecs = 31536000 are set one is under the default stanza and another one is under summary stanza frozenTimePeriodInSecs = 188697600. So which one will be taken in consideration for deleting the data. Can rm -rf used to delete the data present under the coldb bucket. thanks in advance. ... View more

thanks lycollicott, I think data are being monitored from this server 01. Tried this query and got this result. host =splunk01* sourcetype=splunkd series="*" index=_internal 08-06-2016 10:20:42.391 -0400 INFO Metrics - group=per_index_thruput, series="webops_security", kbps=0.067068, eps=0.419353, kb=2.079102, ev=13, avg_age=1.153846, max_age=3 host = splunk01.xxxx.com source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd 8/6/16 10:20:42.391 AM 08-06-2016 10:20:42.391 -0400 INFO Metrics - group=per_index_thruput, series="unix_svrs", kbps=62.673401, eps=371.063295, kb=1942.881836, ev=11503, avg_age=-9247.172477, max_age=5000 host = splunk01.xxxx.com source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd 8/6/16 10:20:42.391 AM 08-06-2016 10:20:42.391 -0400 INFO Metrics - group=per_index_thruput, series="sos", kbps=0.052734, eps=0.387096, kb=1.634766, ev=12, avg_age=0.000000, max_age=0 host = splunk01.xxxx.com source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd 8/6/16 10:20:42.391 AM 08-06-2016 10:20:42.391 -0400 INFO Metrics - group=per_index_thruput, series="sec_sym_wg", kbps=0.359720, eps=2.096767, kb=11.151367, ev=65, avg_age=54499.676923, max_age=604801 host = splunk01.xxxx.com source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd 8/6/16 10:20:42.391 AM 08-06-2016 10:20:42.391 -0400 INFO Metrics - group=per_index_thruput, series="net_proxy", kbps=8.613284, eps=25.516045, kb=267.012695, ev=791, avg_age=0.716814, max_age=3 host = splunk01.xxxx.com source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd In filed series=* all the index details are listed and I am not sure what is the purpose of series filed, tried to research in google about the series filed in SPL but could not get information, do you have any idea why this filed is used. How to monitor the amount of being injected under the path /opt/syslog/* in both the servers 01 & 02. To have proactive monitoring on disk size /opt. thanks in advance. ... View more

thanks for putting your effort on this. I had tried to execute the query as you said and I could see the below data in the splunk portal. SPL query host= splunkhvy01* index=_internal sourcetype=splunkd Search Result 8/5/16 2:37:16.455 PM 08-05-2016 14:37:16.455 -0400 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/syslogs/generic/txxxx25bap/8804.log'. host = splunkhvy01.xxxx.com source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 8/5/16 2:37:16.218 PM 08-05-2016 14:37:16.218 -0400 INFO TailReader - File descriptor cache is full (100), trimming... But when I tried this query sourcetype=syslog source="/opt/syslogs/webops_security//.log" hostname = splunkhvy01 - I am not getting any value. What I am trying to do here, I want to know whether the syslog data are being captured in this server (splunkhvy01). Please let me know how to check that whether syslog data are being captured in this HF instances. And also how to measure the data load between two HF server 01 & 02. thanks in advance. ... View more

thanks lycollicott, yes you are right, we could see On server splunkhvy01 sysglog-ng IS NOT running and also it was not listening to the port. After restarting the syslog-ng, the syslog-ng status is running and its listening to the port. Restarted the syslog-ng service by executing the below script and verified the syslog-ng status /etc/init.d/syslog-ng restart [root@splunk01 init.d]# service syslog-ng status syslog-ng (pid 8892) is running... [root@splunk01 ~]# netstat -a | grep syslog-ng unix 2 [ ACC ] STREAM LISTENING 98712418 /var/lib/syslog-ng/syslog-ng.ctl But I am not sure what this number 98712418 mean, it like a mobile number. Now we have two challenges : 1) how to check whether the syslogs logs are being captured from this server01 in splunk portal --- Is there any query which we can execute and filter out the data that being captured from this server 01. As said this the inputs.conf files that is configured to monitor the syslog data from both the servers. /opt/splunk/etc/apps/HVY-ADMIN-hvy_forwarders/default/inputs.conf [monitor:///opt/syslogs/web_access/.../Common/.log] [monitor:///opt/syslogs/symantec/SymantecServer/...] [monitor:///opt/syslogs/symantec/semp/...] [monitor:///opt/syslogs/symantec/.../ID.log] [monitor:///opt/syslogs/proxy/...] sourcetype = bluecoat_syslog [monitor:///opt/syslogs/dns/.../.log] sourcetype = syslog [monitor:///opt/syslogs/webops_security/.../.log] sourcetype = syslog [monitor:///opt/syslogs/firewall/.../.log] sourcetype = syslog [monitor:///opt/syslogs/esx/.../*.log] sourcetype = syslog 2) How to check the amount of data being injected into both server 01 & 02 under the path /opt/syslogs/ per hour. So that we can monitor and clear the logs once the /opt mount point reached 80 % thanks in advance .. ... View more

thanks for your help , as you told I had looked into the inputs.conf file and outputs.conf file in Heavy forwarder and found in our environment we got 30 inputs.conf / 10 outputs.conf file. In all of these configuration files I could not see the stanza as mentioned in your comments [UDP = 514], instead I could see the stanza like this path /opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://9998] connection_host = ip [splunktcp://9999] connection_host = ip /opt/splunk/etc/system/local/inputs.conf [splunktcp://9998] connection_host = ip [splunktcp://9999] connection_host = ip Instead of find command, used locate command and filtered out “udp | 514 | syslog” using the below script for i in locate inputs.conf ;do echo $i;echo "================================";egrep -i "udp|514|syslog" $i;done >>/var/tmp/aaa Similarly in outputs.conf, I could see the below stanza in both the HF. /opt/splunk/etc/apps/HVY-ADMIN-all_fwd_outputs/default/outputs.conf [tcpout] defaultGroup = all_indexers maxQueueSize = 1GB [tcpout:all_indexers] server = splunkp01.XXXX.com:9997,splunkp02.XXXX.com:9997,splunkp03.XXXX.com:9997,splunkp04.XXXX.com:9997,splunkp05.XXXX.com:9997 autoLB = true /opt/splunk/etc/apps/HVY-ADMIN-hvy_forwarders/default/outputs.conf [tcpout] indexAndForward = false forwardedindex.filter.disable = true forceTimebasedAutoLB = true /opt/splunk/etc/apps/all_fwd_outputs/local/outputs.conf [tcpout] defaultGroup = all_indexers [tcpout:all_indexers] server = splunkp01.XXXX.com:9997,splunkp02.XXXX.com:9997,splunkp03.XXXX.com:9997,splunkp04.XXXX.com:9997,splunkp05.XXXX.com:9997 autoLB = true /opt/splunk/etc/system/local/outputs.conf [tcpout] indexAndForward = false forwardedindex.filter.disable = true [tcpout:all_indexers] server = splunkp01.XXXX.com:9999,splunkp02.XXXX.com:9999,splunkp03.XXXX.com:9999,splunkp04.XXXX.com:9999,splunkp05.XXXX.com:9999 autoLB = true As said earlier both syslog /splunk HF instance are running in the same server and noticed that syslog data itself not getting updated in the server from F5 load balancer, I doubt whether this might be the cause for the issue. I verified by executing this command and found no data. [root@splunkhvy01 ~]# netstat -a | grep syslog Where us when executed from another active HF which is capturing the logs. [root@splunkhvy02 ~]# netstat -a | grep syslog tcp 0 0 splunkhvy02.xxxx:shell vmzscaler01-syslog.he:ppsms ESTABLISHED tcp 0 0 splunkhvy02.xxxx:shell vmzscalerpoc01-syslog:23999 ESTABLISHED udp 0 0 :syslog *: unix 2 [ ACC ] STREAM LISTENING 10795 /var/lib/syslog-ng/syslog-ng.ctl Kindly guide me on how to fix this issue and also how to open the port for syslog-ng to listen data from F5 load balancer. thanks in advance ... View more

yes your right it seems UDP port is not configured to listen from this server. When check for another server which is currently listening using the above command and got this output. [root@splunk ~]# netstat -an | grep 514 BLISHED udp 3664 0 0.0.0.0:514 0.0.0.0:* unix 2 [ ] STREAM CONNECTED 364451410 unix 3 [ ] STREAM CONNECTED 364451406 unix 3 [ ] STREAM CONNECTED 364451405 unix 2 [ ] DGRAM 364451402 Is there way to configure it through command prompt, as we do not have GUI access to enable this port. thanks in advance. ... View more

its configured with the UDP port 514 in F5 load balancer. I have executed the below command and got this result. [root@splunk ~]# lsof | grep -i 514 pbx_excha 1662 root 10u IPv4 12585 0t0 TCP localhost:42365->localhost:51432 (ESTABLISHED) pbx_excha 1662 root 11u IPv4 12604 0t0 TCP localhost:42365->localhost:51434 (ESTABLISHED) vnetd 2139 root 6u IPv4 12576 0t0 TCP localhost:51432->localhost:42365 (ESTABLISHED) bpcd 2142 root 7u IPv4 12595 0t0 TCP localhost:51434->localhost:42365 (ESTABLISHED) certmonge 2372 root 195r FIFO 0,8 0t0 69464514 pipe certmonge 2372 root 201r FIFO 0,8 0t0 72351433 pipe vxdclid 25125 root mem REG 253,2 551496 9696648 /opt/VRTSsfmh/lib/lib32/libvxdcli.so I think its not listening to the port, so is there any other command to find out whether udp port is listening or not? then if not listening then what we should to do make it to listen ? ... View more

Thanks Giuseppe, We have tested the above command under path from where the splunk reads the file and forwards to HF. As I told you earlier there is no data under this path from the source system (host machine). [root@splunkhvy hostname]# pwd /opt/syslogs/symantec/SymantecServer/hostname [root@splunkhvy hostname]# ls -la total 8 drwx------ 2 root root 4096 Apr 22 05:27 . drwx------ 5 root root 4096 Apr 22 05:27 .. As you know , we are getting the data from the same host under this path [root@splunkhvy syslogs]# cd generic/hostname/ [root@splunkhvy hostname]# pwd /opt/syslogs/generic/hostname [root@splunkhvy hostname]# ls -ltr total 15683036 -rw------- 1 root root 16059374206 Jul 25 06:25 SymantecServer.log [root@splunkhvy hostname]# Now we have doubt whether the hostname.log is getting generated from the source system or not ? As we are unable to see the data from the Symantec source system with the file name hostname.log. Suppose if we want to re-install the universal and configure new agent, then what are the steps we should follow to get the data from UF to HF then to Indexer. thanks in Advance. ... View more

thanks Cusello, I had tried to search the same way as you had suggested in the search portal but no result. query details - source= "/opt/syslogs/Symantec/Symantecserver/hostname/*.log" similarly when searched same thing in CLI using the command ls -la there were no hidden files present in the path /opt/syslogs/Symantec/symantecserver/hostname/.... kindly guide me on this ... View more

thanks cusello. But I have question, how can I create a stanza to monitor the path when there is no data getting into this path /opt/syslogs/Symantec/Symantecserver/hostname/.log ---> zero logs. As I told you, that logs are getting into /opt/syslogs/generic/hostname/SymantecServer.log and its pointed to this index=unix_srvs and sourcetype as syslogs. which is not correct index and sourcetype. It should be index=Symantec and sourcetype = sym. kindly guide me on this. ... View more

thanks Giuseppe. yes we have found that the data are getting into the HF but its pointing to some other indexer and I am not sure how this got changed. Currently the data's from the Symantec servers are getting in to this path /opt/syslogs/generic/hostname/SymantecServer.log in HF and but pointing to different index name =unix_srvs and source type=syslog. Inputs.conf Stanza details [monitor:///opt/syslogs/generic/.../*.log] sourcetype = syslog host_segment = 4 blacklist = dxxx*ltm* index=unix_srvs now its clear that the data are getting into HF but in different location, different indexer, source type, so how to fix this issue. By changing the inputs.conf stanza alone will fix the issue or before doing this we should make sure that data are getting in to the correct path in HF before splunk monitors the logs. correct path = opt/syslogs/symantec/Symantecserver/xxxx/.log index =Symantec sourcetype =sym kindly guide us how to proceed to fix the issue. thanks in advance. ... View more

thanks cpetterborg, Ours is a Distributed Splunk Environment, where we have 4 search head two are in clustered and other are independent, one file sharing pool, 5 indexer, License/deployment manager and two heavy forwarder with version 6.2.1. As per the architecture diagram, data's from UF are forwarded to Indexers directly and only the syslogs are forwarded to the HF using the TCP/UDP port 514. But when searched in the portal by executing this query host=XXXX index=Symantec, I could see the data and the source is pointed to this path /opt/syslogs/Symantec/Symantecserver/server name/servername.log and the same path is configured in the HF inputs.conf. I am getting confused with lots of inputs.conf and outputs.conf in HF/UF/Indexers. Kindly let know how to figure out which configuration files are used to send the data. thanks in advance.. ... View more

thanks Giuseppe, Actually I am new to this environment, the person who had built the entire splunk environment had left the organization and there is no document on how they have configured it. Ours is Distributed Splunk Environment, where we have 4 search head two are in clustered and other are independent, one file sharing pool, 5 indexer, License/deployment manager and two heavy forwarder with version 6.2.1. As per the architecture diagram, data's from UF are forwarded to Indexers directly and only the syslogs are forwarded to the HF using the TCP/UDP port 514. From search portal, by executing this query host=XXXX index=Symantec, I could see the data and the source is pointed to this path /opt/syslogs/Symantec/Symantecserver/server name/servername.log and the same path is configured in the HF inputs.conf. HF input stanza [monitor:///opt/syslogs/symantec/SymantecServer/...] whitelist = \.log index = Symantec sourcetype = sep host_segment = 5 But currently under this path opt/syslogs/Symantec/Symantecserver/server name / there is no logs getting in. I am not sure how it got broken and currently the user has complained that he is not getting the data to analysis. So kindly tell me how to trouble shoot this issue. ... View more

thank Lucas, for your inputs on this issue. I have pasted the outputs.conf stanza in above comment. Kindly guide me to get this fix. thanks Advance. ... View more