I'm indexing a field with DBConnect that contains the backslash character, eg \, in order to escape quotation marks and hyphens within the data. This has a side effect of breaking the field extraction after the first \ character. Has anyone encountered this problem, and if so, how do you work around it? ... View more

I have a DB Connect query for which the results in one column read as ***** BINARY ***** Is there a way to get DBConnect V1 to parse that column into Splunk-friendly ASCII? ... View more

I'm looking at behavior of a service which consumes messages about products, the unifying factor being a field called UPC. In theory there should just be a single message sent across per UPC, but I know this is often not the case and regularly see up to 20 messages per UPC come across. I'm not concerned with individual UPCs here, but with aggregate data: what's the average number of messages per UPC over the 50000 UPCs we saw come through? The median? The maximum? I can get these with the transaction command: **sourcetype=blahblah | transaction UPC_CODE | timechart span=1m max(eventcount),median(eventcount),avg(eventcount)** But this is really slow and I can't help but think there is a more efficient way to do this. ... View more

We have a text file to log row counts for CSV files used in an ETL job. The format of the file is like this: 08/30/15 16:08:51 acme_file.csv: 0 coyote_file.csv: 0 anvil_file.csv: 88835 roadrunner_file.csv: 330333 I want to break this into one line per filename from line 2 onward, into fields {filename}.csv: {rowcount} Note that the first line in the .txt file is the datetime for all of the events. I think the linebreaking is fairly straightforward via props.conf on my indexers, but that first line with the datetime is in a different format; I want to use that and apply it to all subsequent lines. Little help! ... View more

We have a distributed Splunk environment with two search heads, four indexers in two clusters, and a deployment server. I set up the Splunk App for Web Analytics on the search heads, configured some web sites, and began running the data model builds described in the Setup steps. As this ran, we started seeing bundle replication warnings, and the search head running the data model build started to fill with very large bundle files, @ 1 GB a piece. From Splunk logs: 07-31-2015 13:32:36.583 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error 07-31-2015 13:32:36.583 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named server1 with uri=https://11.22.33.44:8089. ... View more

I'm working on building the lookups and data model for the Splunk App for Web Analytics. I'm supporting an environment with multiple websites (eg roadrunner.acme.com, coyote.acme.com, anvil.acme.com) and lots of web servers (server1, server2, server3, and so on). I note that the Setup New Website dialog asks for site, host, and source information, and will accept a wildcard for the host field. Two things: in the real world, I might have hundreds of hosts that match a wildcard search (server*), and yet only a few that host roadrunner.acme.com. It would be better if I could name multiple specific hosts for a given website, rather than have to do this through a wildcard; yet the form actually allows the user to input multiple hosts for a given website, but then will not work to build lookups. And if my host naming convention (prior to the *) and log file path are the same for coyote.acme.com, I suspect that will also cause a problem in the app, though I've not gotten that far yet. Finally, the initial lookup build for "Generate User Sessions," and "Generate pages," is excruciatingly slow, perhaps due to the wildcard search on the host name. Tips for speeding this up in the documentation, or at least a clearer idea of what "a long time," means, would be helpful. ... View more

We're indexing /var/log/secure, as one does, and I have a request to list users who've logged in in a comma-delimted list per host. I'm using the Splunk search source=/var/log/secure "pam_unix(sshd:session): session opened" | stats values(user) as "user" by host | mvcombine delim="," user This works great, but for some reason the results are not honoring the delim="," and so they're coming out as space-delimited instead. A minor annoyance, but pointers for getting this to do exactly as I want would be great. ... View more

I'm working with a customer to run rowcount comparisons between two tables that are replicating data in one direction, from A>B, and alert if the delta between the two is more than x%. If the results of select count * from DB_A_Table_1 are more than +/- 5% different from the results of select count * from DB_B_Table_1, then we trigger an alert. The POC was against a single pair of tables, and worked so well the customer now wants about 170 pairs of tables compared. I have to imagine there's a more elegant way to do this than to set up 340 DBConnect queries to index and then 170 alerts, though I do want to index rowcount results each time so we can see trends when troubleshooting. If someone has done this in the past, your guidance is appreciated. If no one pipes up, I will post the solution when I arrive at one, hopefully only slightly balder and greyer then I am at the time of this writing. ... View more

I'm troubleshooting a possible configuration change to splunk-launch.conf and wonder if there is a search I can run in Enterprise that will tell me what paths the Splunk index data was written to, preferably by index. ... View more

I have a search like this: index=wilee sourcetype=foo OR sourcetype=foo2 | transaction fields="acme,coyote" maxevents=2 keeporphans=f endswith="beepbeep" startswith="genius" | rangemap field=duration "0-1m"=0-60 "1-2m"=60.0001-120 "2-5m"=120.00001-300 "5-10m"=300.00001-600 ">10m"=600.00001-9999999 | timechart span=1h count by range This makes for a nice stacked column showing me distributions of durations by hour. I'd like to overlay a line on a secondary Y axis to display median durations by hour, eg index=wilee sourcetype=foo OR sourcetype=foo2 | transaction fields="acme,coyote" maxevents=2 keeporphans=f endswith="beepbeep" startswith="genius" | timechart span=1h perc50(duration) ...but I'm struggling. Splunk's examples use a large dataset for axis y1, and then a subset of that for y2. ... View more

From what I can tell, the vmstat.sh script in the Nix app runs the vmstat 1 1 command to collect memory information. However, this brings back averages since the machine was last rebooted rather than current information. Has anyone tweaked the scripts to run something like vmstat 2 2 and ignore the first line of data returned, in order to get current stats? ... View more

We have an image processing service, and from the service logs I can calculate the duration in seconds of processing time for a given image. When we make hardware or software changes I want to be able to visualize performance changes. It's easy to do a time series of counts, but I've been asked to do so for a percentage instead. My search to get overall percentages, again fairly easy: sourcetype="imagelogs" | eval duration=ceil(duration+(_time-(strptime(lastmodifiedtime,"%Y-%m-%d %H:%M:%S")))) | where duration < 86400 | rangemap field=duration "0-5 min"=0-299 "5-10 min"=300-599 "10-15 min"=600-899 "15-60 min"=900-3599 "60+ min"=3600-9999999 | top limit=5 range showcount=false percentfield=Percentage The top search command seems to be incompatible with timechart. Do I need to construct a whole series of eval command within the timechart command to replace the rangemap function? ... View more

We've seen inconsistent problems where one log file on a host will stop indexing in Splunk once the log file rotates after a JVM restart, possibly due to a timestamp change. We're working with Splunk on the root cause. Typically other logs on the same host will continue indexing as normal. I'm trying to think of an efficient way to alert when this happens while we work to resolve it, and from what I can tell SOS does not contain a readymade solution. The basics are simple - alert when we have 0 events in the last x minutes. What makes this complicated is that we have a large number of sources that I care about, in the hundreds. In our taxonomy the sourcetype and source attributes will be shared across multiple logs on multiple hosts, eg the sourcetype=jvm_log and source=/www/logs/server.log attributes will be common to somewhere between 2-12 individual files across as many hosts. Ideally I'd want an alert for any unique host+source combination where the event count=0 for the last x minutes, without writing hundreds of nearly identical searches. ... View more