About cwacha

cwacha · ‎08-05-2013

Hi, I have built an app that aggregates data into a summary index. The app also provides a query that searches for the data in the summary index and displays it. When running the app on one single search head everything works fine. The problem is that I have two search heads and the app should run on the other search head as well. Both search heads also have the "other" search head configured as a search peer in distributed search. So here is the problem: When the query runs from the app on the first search head it searches for data from index=my_summary_index. Since the app also runs on the other search head we also have such a summary index on that other indxer too. Both indexes are returning data and thus duplicate the results. How can I prevent results from the "other" search head to pollute my query? How can I find out the hostname or splunk_server of the search head "this" query is running on?

cwacha · ‎01-18-2013

We are monitoring many files with the UF using the [monitor] stanza. For housekeeping reasons we need to delete the files every now and then. Now deleting a file that was not already fully transferred to the indexer would lead to data loss. How can we find out which files have already been transferred by the UF? Of course the UF itself does not know that a file is already complete or closed. It only sees that there was not more data added since some time. So what we need is some kind of list that shows the filenames being monitored as well as the file pointer up to what amount was already transferred. example output: /var/adm/messages 1233122 /var/adm/authlog 1233 Based on that our cleanup mechanism could then compare the actual size of a closed file with the size reported in the output above. If they are equal it knows that the UF has transferred everything and we can delete the file. An alternative would also be to provide an "auto-close" function in the UF. This function would trigger the execution of a shell command (specified in inputs.conf) as soon as the file's size has not changed for "auto-close-seconds" (also specified in intputs.conf).

cwacha · ‎09-20-2012

Hi all, I found the solution. The events I am looking for are: 09-20-2012 15:38:08.583 +0200 INFO VolumeManager - The size of volume 'cold' exceeds the limit, will have to acquiesce it (size=2055243225109, max_size=2055208960000, path='/opt/splunk_cold/data/indexer/cold/indexer') 09-20-2012 15:38:08.583 +0200 INFO VolumeManager - Getting a list of candidate buckets for moving (chilling or freezing) 09-20-2012 15:38:08.704 +0200 INFO VolumeManager - Will move bucket with latest=1332892827, path='/opt/splunk_cold/data/indexer/cold/indexer/_internaldb/colddb/db_1332892827_1332892827_31' 09-20-2012 15:38:08.705 +0200 INFO VolumeManager - Bucket moved successfully (current size=2055243221817, max=2055208960000) 09-20-2012 15:38:08.705 +0200 INFO VolumeManager - Will move bucket with latest=1332894456, path='/opt/splunk_cold/data/indexer/cold/indexer/main/colddb/db_1332894456_1332808057_143' 09-20-2012 15:38:08.705 +0200 INFO VolumeManager - Bucket moved successfully (current size=2054983523562, max=2055208960000) I was able to query the relevant data (Will move bucket...) and calculate the actual retention as follows: index=_internal sourcetype="splunkd" component="VolumeManager" move /opt/splunk_cold | eval eff_retention_days=((_time-latest)/3600/24) | timechart span=1d min(eff_retention_days) as "actual retention (d)" Thank you all for your help!

cwacha · ‎09-20-2012

Thanks but this query does not tell me when buckets get rolled from cold to frozen (which is what I want to see). Can you tell me what I have to look for?

cwacha · ‎09-19-2012

We have configured our Splunk 4.2 to remove log data as follows: remove data if it is older than 200 days remove data if it uses more diskspace than X The actual disk and diskspace settings are scaled, so that always the time rule should apply. They are used only as a failsafe mechanism. How can we monitor if data really is deleted because it is old and not because the disk is full? We would like to verify that the diskspace numbers we calculated are correct. Is there a way to see why certain buckets were rolled, so that we can search for "rolled because disk full"?

cwacha · ‎09-06-2012

We have one univeral forwarder running on a solaris system and it is monitoring around 30000 files where 1% of them is being constantly written to. Its working using the 4.3.3 code but we get delays of several minutes until events show up in the search GUI. If we only monitor one single file on this box the transport delay lies within seconds.

cwacha · ‎05-09-2012

Great! This did the trick. Thanks!

cwacha · ‎05-09-2012

For security reasons we would like to disable the management port but unfortunately very neat debugging commands such as ./bin/splunk list monitor stop working after that. It would be ok if we have the port enabled but accessible only for localhost communication. Altough the configuration explicitly sets the localhost IP address it is still possible to open telnet connections from other ports. web.conf [settings] mgmtHostPort = 127.0.0.1:8089 Even netstat shows that the port is open on every available network interface. # netstat -an | grep 8089 tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN Question: how can we force the Universal Forwarder (or even the Heavy Forwarder) to bind to the localhost interface only?

cwacha · ‎02-06-2012

I don't know exactly what the problem is here but we had a similar issue. Basically it turned out that forwardedindex.filter.disable = true was not working at all. We had to leave it on default = false and add everything to the white and blacklists. Maybe defaultGroup = grou1, group2 might do load balancing between the two...???

cwacha · ‎10-10-2011

Actual Situation: A Heavy Forwarder with the [batch://] stanza configured using default values is reading files from a directory and forwards them to a second indexer. Mark that there is no index= mapping defined in the inputs.conf. A call to splunk cmd btool inputs list --debug shows that the index for this input is set to default . On the indexer side all log records from this source will end up in index=default (as we would expect). Unfortunately no matter what you configure on the indexers transforms.conf they always end up in the default index. It is not possible to re-direct the events to another index. Expected Situation: A proper transforms.conf on the indexer should make it possible to redirect the records to an index of choice. Additional Findings: If we add the exact statement index=test1 to the batch stanza on the Heavy Forwarder (so that the records would go to index test1 ) the same settings in transforms.conf on the indexer suddenly begin to work! It seems that redirecting the data on the indexer to an index of choice is only possible if any (even non-existent) index is configured on the Heavy Forwarder side. I consider this a bug. Please fix. 🙂

cwacha · ‎08-11-2011

Our splunk universal forwarders may be configured by our customers in terms of logs they want to collect and transfer. An application that is logging very exessively could therefore drastically increase the amount of log data to be indexed. Is there some sort of flood protection available (on indexer or heavy forwarders) to detect universal forwarders that are transferring lots of data? Actions could be blacklisting the agent for a limited time period (and redirecting its log data to the nullqueue) or generating an alert or logging an error message.

cwacha · ‎08-11-2011

We are currently creating a Disaster Recovery guide for our Splunk installation that must contain all steps necessary to fail over splunk in our environment. One task is to roll all hot buckets to warm. Obviously we need to be "logged in" on the CLI before we can execute this step. The problem is that we need to use a username and password that exists in splunk to get this done and we would have to add that username and password to the DR guide as well to make sure in case of a disaster anyone with basic system knowledge would be able to execute the steps. What is the best way to do this? Can we disable the password functionality on the CLI? should we put the admin users credentials into the DR guide?

cwacha · ‎06-07-2011

As fas as I know props/transforms cannot be used with the universal forwarder...

cwacha · ‎06-06-2011

I use the universal forwarders ability to enrich the transported files with _meta keywords as follows: ./etc/apps/myapp/local/inputs.conf [monitor:///myfile] disabled = false _meta = key1::value key2::value I also have global key/value pairs for _meta that I would like to add automatically to all monitor stanzas. They are defined in ./system/local/inputs.conf [default] _meta = globalkey::value The globalkey keyword gets added to all monitor stanzas that do not define a specific _meta keyword. I would like to have the globalkey keyword as well as the additionally defined key/value pairs in the apps inputs.conf defined. Unfortunately the _meta field in apps/../inputs.conf overwrites the system/local/inputs.conf _meta entry. Is it possible to append the global keywords (defined in ./system/local/) to the defined _meta tag (defined in ./apps/local/inputs.conf) ?. For example with a configuration as follows using $_meta: ./system/local/inputs.conf [default] _meta = globalkey::value ./etc/apps/myapp/local/inputs.conf [monitor:///myfile] disabled = false _meta = $_meta key1::value key2::value

Posts	14
Solutions	1
Karma Given	1
Karma Received	5
Member Since	‎06-06-2011

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

How to build a query that returns the hostname of ...

How to query a universal forwarder about the trans...

How to verify that data is retained for 200 days a...

How to limit mgmt port access to localhost only on...

Heavy Forwarder batch stanza bug

Data flood protection on forwarder or indexers

Executing CLI commands without password

How can I merge _meta from several inputs.conf fil...

How to build a query that returns the hostname of ...

How to query a universal forwarder about the trans...

Re: How to verify that data is retained for 200 da...

Re: How to verify that data is retained for 200 da...

How to verify that data is retained for 200 days a...

Re: Is there a limit on the number of files a forw...

Re: How to limit mgmt port access to localhost onl...

How to limit mgmt port access to localhost only on...

Re: forwardedindex.filter.disable = false ?

Heavy Forwarder batch stanza bug

Data flood protection on forwarder or indexers

Executing CLI commands without password

Re: How can I merge _meta from several inputs.conf...

How can I merge _meta from several inputs.conf fil...