We've been chugging along fine with our 4 unreplicated indexers. I'd like to add a new index now, but have gotten stuck. This app is successfully deployed from the deployment server: /opt/splunkforwarder/etc/apps/throwaway_app/ cat /opt/splunkforwarder/etc/apps/throwaway_app/bin/inputs.conf [script:///opt/splunkforwarder/etc/apps/throwaway_app/bin/topn1.sh] disabled = false index = throwaway # Run every 15 minutes interval = 900 source = throwaway_top sourcetype = script:///opt/splunkforwarder/etc/apps/throwaway_app/bin/topn1.sh [monitor:///opt/logfile.log] index = throwaway disabled = false sourcetype = throwaway.logfile cat /opt/splunkforwarder/etc/apps/throwaway_app/bin/topn1.sh #!/bin/bash #top -n 1 | grep splun[k] | awk '{print $3" "$6" "$7}' ps -ef This is added to the end of a well used indexes.conf file and is successfully deployed to the indexers: [throwaway] homePath = volume:primary/throwaway coldPath = volume:primary/throwaway/colddb thawedPath = $SPLUNK_DB/throwaway/thaweddb tstatsHomePath = volume:primary/throwaway/datamodel_summary summaryHomePath = volume:primary/throwaway/summary maxMemMB = 20 maxHotBuckets = 10 maxConcurrentOptimizes = 6 maxTotalDataSizeMB = 4294967295 maxWarmDBCount = 9999999 maxDataSize = auto_high_volume The throwaway index is recognized and listed in this search with the settings I have put in indexes.conf | eventcount summarize=false index=* | dedup index | fields index As mentioned above, data is not aggregating in the new index either when I search or when I look fora folder. I thought that new data would force the creation of the index folder structure, but nothing is getting created. I may be under the false impression that since we are not replicating data we are not using a master. I've been reading through the docs, but everything seems to point to clustered (replicated?) indexers, which I don't have. Can someone help? ... View more

I'm troubleshooting a deployment client and I've gotten stuck; Deploy server $ /splunk/bin/splunk --version Splunk 6.1.4 (build 233537) Note: This server deploys apps successfully to 125+ clients. Deploy client $ /opt/splunkforwarder/bin/splunk --version Splunk Universal Forwarder 6.1.5 (build 239630) The problem; no deployments. $ /opt/splunkforwarder/bin/splunk display deploy-client Deployment Client is disabled. This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down. $ service splunk status Splunk status: splunkd is running (PID: 4922). splunk helpers are running (PIDs: 4923). Other than the obvious need to upgrade, does this indicate a glaring mistake I'm missing or is this more subtle? Currently setting up log debugging to see if I can learn more. Thanks. ... View more

I have a simple scripted input that runs a powermt command periodically. <apps>/powermt/bin/powermt.sh #!/bin/bash sudo powermt display <apps>/powermt/local/inputs.conf ##### Powermount scripted Inputs ######1 [script:///opt/splunkforwarder/etc/apps/powermt/bin/powermt.sh] ## Run every 15 minutes disabled = false interval = 15 source = powermnt sourcetype = script:///opt/splunkforwarder/etc/apps/powermt/bin/powermt.sh] <apps>/local/props.conf [powermt] DATETIME_CONFIG = CURRENT The app deploys and runs as it should on 5 out of the 6 target hosts otherwise , however on the 6th host it attributes the data to an incorrect host name (a test server that has been retired). The good news is that there's lots documentation about this. The bad news is that there's lots of documentation about this. While I go back and convert several hours of my day into unraveling the root issue, I'm wondering if there's a way I can force the script to force/set the local hostname. This would relieve quite a bit of my immediate pressure. ... View more

We have a service account that populates /var/log/messages on many systems with 3 lines of text every 5 minutes. I'd like to filter that out. These entries come in on a single source: /var/log/messages ...and a single source type: syslog We have 4 indexers that are fed by 12 universal forwarders (in turn fed by many other Windows and linux hosts). I'm using a deployment server to successfully deploy props.conf and transforms.conf to each of the indexers. This is my entry in props.conf; [syslog] TRANSFORMS-set = setnull This is my entry in transfro [setnull] REGEX = svc_scomlinux DEST_KEY = queue FORMAT = nullQueue I expect any entries coming in on the source/sourcetype and containing the text svc_scomlinux to be discarded, but it's still coming through. I spent a good part of the day reading answers here, docs and other sources on the web but I'm not getting any traction. What am I missing? Thanks ... View more

Something's just not clicking here. Colleagues have EC2 instances in AWS and want to index logs in our internal Splunk environment. I see that they have CloudTrail configured, but I am a complete noob to AWS and my experience with Splunk is not deep. I see these two apps; Splunk App for AWS Splunk Add-on for Amazon Web Services Where exactly do these apps get installed? On the instance? on the Searchhead? How can we “bake” splunk in to our instances? How will we tell which instance the logs are from? ... View more

I'm using this search to retrieve indexing data by month; index="_internal" source="*metrics.log" group="per_host_thruput"| chart sum(kb) by series date_month | sort + series It returns in this format, which works well for me. HOST January February March ... hosta 1234567 1234567 1234567 ... hostb 2345678 1234567 1234567 ... hostc 3456789 1234567 1234567 . I've created a lookup table that gives every indication of success when I look at the search output in event mode, I see them listed, E.g.; host_owner Infrastructure host_role Splunk indexer I expect including these fields will reduce downstream questions but I'm having problems getting a good search when including them. I think part of my problem might be that the search is actually getting the host name from the series and I'm using the data inappropriately. Actually now that I look at it I'm certain. The host for each of these servers is an indexer. Does this mean that I need to create a different lookup table? Ultimately I need to add some additional information beyond the hostname. ... View more

I'm using this simple search to get indexing volume by host. index="_internal" source="*metrics.log" group="per_host_thruput" | chart sum(kb) by series | sort + series This sorts by hostname which is what I want; HOST sum(kb) hosta 1234567 hostb 2345678 hostc 3456789 ... ...but my ultimate goal is actually to break the total down by month: HOST January February March ... hosta 1234567 1234567 1234567 ... hostb 2345678 1234567 1234567 ... hostc 3456789 1234567 1234567 ... ... I've looked at timechart and span options but I feel I may be running into a more fundamental problem with my approach. ... View more