Activity Feed
- Got Karma for Re: DB Connect configuration: Unable to save database inputs and slow search query. 06-05-2020 12:47 AM
- Got Karma for Re: DB Connect configuration: Unable to save database inputs and slow search query. 06-05-2020 12:47 AM
- Karma Re: How can I use Google Apps OpenID authentication with Splunk single sign-on? for cnk. 06-05-2020 12:46 AM
- Karma Re: HALP! Consulting the summary dashboard of the search app causes my system to run out of memory! for jrodman. 06-05-2020 12:46 AM
- Karma Sizing Splunk 4.3 for snevarezh. 06-05-2020 12:46 AM
- Karma where could I find a list of params that are supported in the new JSChart? for imrago. 06-05-2020 12:46 AM
- Karma Re: Determining Logging Lag (and Device Feed Monitoring) for mataharry. 06-05-2020 12:46 AM
- Karma Re: Dotted Line Chart for bbingham. 06-05-2020 12:46 AM
- Karma Re: Find days with no events for johnnymc. 06-05-2020 12:46 AM
- Karma Re: Find days with no events for sideview. 06-05-2020 12:46 AM
- Karma Re: how to use scripted input for refreshing lookup data? for melting. 06-05-2020 12:46 AM
- Karma Re: Anyone else seeing clients not restarting after a deployment server push? for yannK. 06-05-2020 12:46 AM
- Karma Splunk implements Graphite? for tpaulsen. 06-05-2020 12:46 AM
- Karma Splunk rest api change column order. for voffko. 06-05-2020 12:46 AM
- Karma Re: Reducing Time taken for Search in splunk query for sideview. 06-05-2020 12:46 AM
- Karma Re: How to create an "OK" alert - after an alert has been issued? for Damien_Dallimor. 06-05-2020 12:46 AM
- Karma Re: Difference between stats and chart for Ayn. 06-05-2020 12:46 AM
- Karma Key-Value extraction using rex function for josknigh. 06-05-2020 12:46 AM
- Karma Re: splunk 5.0 for ChrisG. 06-05-2020 12:46 AM
- Karma Re: Why is FSChange (file system change monitor) a deprecated feature in Splunk 5.0? for cervelli. 06-05-2020 12:46 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 1 | |||
| 3 | |||
| 0 | |||
| 9 | |||
| 11 |
08-14-2014
08:50 PM
2 Karma
FYI - you can reload the jbridge server by updating the JVM Command Line Options under the "Splunk DB Connect Configuration" dialog (http://your.splunk:8000/en-US/manager/dbx/apps/local/dbx/setup?action=edit). I typically just change the heap size between two equivalent sets of units (eg "-Xmx2048m" and "-Xmx2G") and save to restart the java bridge.
You can also reload the java bridge by hitting the debug/refresh endpoint as follows:
http://your.splunk:8000/en-US/debug/refresh?entity=admin/dbx-dblookups&entity=admin/dbx-dboutput
Verify that this worked by looking at the Java Bridge Status UI's uptime info.
... View more
04-30-2013
11:09 AM
Aha, I just found the optional fields= parameter for splunk.Intersplunk.outputResults() -- will give that a go to see if I can use it to specify column order.
outputResults(results, messages=None, fields=None, mvdelim='\n', outputfile=<open file '<stdout>', mode 'w'>)
... View more
04-30-2013
10:57 AM
Thanks sideview, that definitely is the issue. The chart module only works properly when _time is the first field returned. The problem is that the custom command is using the splunk API to return the results as a list of dictionaries (a dict for each row of results), and dictionaries do not allow fieldorder to be set.
Do you (or anybody) know of any way to specify the fieldorder via the API as results are returned, aside from switching to csv output?
... View more
04-29-2013
06:49 PM
1 Karma
I've written a custom (generating) Splunk command that retrieves data from Carbon/Graphite, a numerical data-logging tool similar to RRD (but better), for the time period specified in the time-picker. The command works as I had hoped, quickly retrieving the data and providing it in columnar format, with _time , _span , and each requesed column of data provided as outputs.
The problem is that when I click the "Results chart" button above the search results, three out of four times the chart doesn't use _time as the x-axis, instead choosing one of the other columns for this, and tries to use _time as the value being plotted. This problem shows up consistently for the same results over a fixed time-period (same number of points, same columns, field values, and _time and _span fields). To validate my work, I wrote another command that dumps the raw field data to a log file, and can't see anything changing in the output of my command between each execution, yet the chart handler behaves inconsistently. I also compared my results to those of timechart with the same span, and they look identical in the raw data dump.
Does anyone of of any magic that timechart does to prepare data for the Chart module, outside of the result fields themselves?
EDIT: The answer was to use the fields= parameter with splunk.Intersplunk.outputResults() to specify column order. I just needed to create a list of the columns, starting with _time and _span , and append the names of each generated column to the list, then provide that as the second argument:
outputResults(results, fields=column_list)
... View more
03-15-2013
06:08 PM
1 Karma
I'm not sure about the dashboard part of the request, but you'd find it easier if you set KV_MODE=JSON in a props.conf entry referencing your sourcetype (on the indexer/search heads). I like to set the sourcetype=json for the json input, and then I only need to set up one props.conf entry for all data that I'll be ingesting later.
The KV_MODE automatically runs spath-like field extractions over the entire event, so you can run a search like:
sourcetype=json source=xyz timings | stats count by timings.cat
or
sourcetype=json source=xyz timings.cpu>1000000
Also, if you run your search in "smart" or "verbose" mode, you'll see all your contextual fields on the left with statistics (eg. timings.subtimings.find.subtimings.cat ). With that in place, it will be easier to glob on fields for charting purposes.
I think your actual data structure is somewhat cumbersome, though. You may want to have an entity like timings.subtimings.operation = find , rather than find being a key underneath subtimings , that way you can run stats over the values of timings.subtimings.operation , which you could do using your structure. The same principle applies further in the nest. Hope that helps a little bit.
... View more
01-07-2013
02:05 PM
In your case the time (in seconds) field comes three positions after the request details. Since the request is always enclosed in quotes in apache logs, I like to use them as an anchor point for the regex, rather than the start of the log message (it allows for a simpler regex). Also, you have three numeric fields following a double-quote and preceding another double-quote, so I would write a regex like this:
\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)
Note that this forum won't let me use the correct syntax above, so if you use this regex with either rex or the field extraction UI, remove the spaces on either side of response_time. I am not sure if the %T field may include decimal points or not, so I made the regex support them optionally.
You can use the regex with either rex (inline pipe to rex), or you can save this field extraction for your apache sourcetype (Manager > Fields > Field Extractions).
Here is what a search might look like with Field Extraction configured:
index=xxx sourcetype=access_combined response_time>10
And with inline rex:
index=xxx sourcetype=access_combined | rex "\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)" | where response_time>10
I should note that you also have the %D parameter in your apache configuration, which is the response time in microseconds. I would personally use this field as it has better resolution and its at the end of the log, which makes it easier to anchor within the string:
(?P< response_time >\d+)$
Then just adjust your where/search syntax for the precision, response_time>10000000 .
As I mentioned before, make sure to remove the spaces around 'response_time' above, when you use it in any of the splunk UI's.
... View more
12-19-2012
05:13 PM
Thanks dart, this seems to work well for me. I'll try to make it into a macro so that the span time can be supplied as an argument and the macro used as a replacement for bucket .
... View more
12-04-2012
07:18 PM
Thats true, sideview, but timechart handle output columns differently than stats (a column will exist for each permutation of the field being split upon rather than a single column with a row for each permutation), and also doesn't let you segment by multiple fields, like stats does (count by a,b,c). Since most of my work involves nested calls to stats, eventstats, and streamstats, using timechart anywhere in the search flattens the data too much to be useful. However, it could be used as the generating command I was talking about, since it could create the _time fields.
... View more
12-04-2012
12:00 PM
I also ran into this problem yesterday. I think the best way would be to create a generating command (creates synthetic events) that outputs a fake event for every day (or configurable timespan) in the timerange for which it is called, then use a join type=outer _time [search your_search] to join your timechart or bucket'ed stats results by time. Then you can use eval to find events where the field(s) from the join is/are null ( eval zeroresults=if(isnull(myfield),"true","false") ). If writing generating commands is too difficult, you could create a csv file with a column for _time, and set the timestamp for each row to the day boundary (in seconds since 1970), then use the inputlookup command to generate the synthetic events and perform the join mentioned earlier.
Thinking of this, I'm surprised Splunk wouldn't already have a command to do this. May need to dig deeper in the commands documentation.
... View more
12-03-2012
03:27 PM
3 Karma
I am writing a search that looks at weighted moving averages of data points summarized and logged at 2 minute intervals. I need to bucket the data into two minute spans, in a window of ten minutes. The search will run every minute, and look at the past ten minutes worth of data, thus, there should always be five buckets of 2 minutes each. You'd think this would be as easy as:
earliest=-10m@m latest=@m *base_search* | bucket _time span=2m | stats xxx by _time
However, the bucket command (and timechart, etc) always make bucket boundaries snap to even numbered time boundaries, rather than being relative to the search time boundaries. To elaborate, if the search is made at 10:10:23, there are five buckets, for 10:00, 10:02, 10:04, 10:06, and 10:00, and if the search is run at 10:11:xx, there are six buckets: 10:00, 10:02 ... 10:10, with the first and last bucket containing one minutes' worth of data each (half the data).
What I think should happen with the 10:11 search is five buckets, the first being 10:01, then 10:03, etc. Has anyone found a way to do this that still lets them sleep at night? Maybe a call to eval that segments time similarly to the bucket command (could be a macro).
... View more
11-02-2012
04:56 PM
1 Karma
Almost, you have to search the _audit index for the audit logs:
index=_audit action=search search=*
... View more
11-02-2012
01:21 PM
In the scripted input, you could pipe the output of the curl command through a command like hxnormalize, which is part of the html-xml-utils package in Linux.
... View more
11-02-2012
10:46 AM
No problem, can you paste in a sample log record or your apache LogFormat directive?Apache log files typically don't include field=value pairs (just values), so regular expressions must be position-sensitive (eg populate this field with the fifth element in the message).
... View more
11-02-2012
12:45 AM
1 Karma
This can be difficult, since so many top level domains are possible. If you have a fixed list of tld's that are common in your result set, you could use a regex like this to extract the website name (eg, google) from an existing field (eg, referer😞
(?P< website>[^. ]*)\.(?:co[m]?|net|cn|ca)
(Note that the space in front of the field name, website, is only there because the forum code required them for parsing, remove them in your rex command and/or transforms). Example usage (with the extra space):
index=weblogs referer=* | rex field=referer "(?P< referer_website>[^. ]*)\.(?:co[m]?|net|cn|ca)" | top referer_website
... View more
11-02-2012
12:31 AM
5 Karma
As hexx recommended, posting this as an answer:
The following commands instantly fixed the CPU usage by the universal forwarder:
/etc/init.d/ntp stop
(date +"%H:%M:%S" |perl -pe 'chomp';echo `date +"%N"` / 999999999|bc -l) | sudo perl -ne 'chomp;system ("date","-s",$_);'
/etc/init.d/ntp start
Note that a simpler date command has been referenced in other articles on the web, but we found it to be less accurate than the above (which includes microseconds).
... View more
11-02-2012
12:06 AM
You can find out whether or not it is Splunk listening on udp:162 with the following command on any splunk server:
$SPLUNK_HOME/splunk list udp
It is hard to answer further without knowing what your setup is, aka do you use forwarders, do you have an snmptrapd instance logging to files that are ingested by Splunk on the indexer or forwarders, etc. If you use forwarders, it is possible that they have an inputs.conf file somewhere in $SPLUNK_HOME/etc/apps that specifies listening on udp:162, which won't show up in your web UI on the indexer and/or search head(s).
... View more
11-01-2012
11:28 PM
I have no AIX systems to test with, hope you found a fix, jnhth!
... View more
11-01-2012
11:22 PM
I did write my own 'printf' splunk command that let me inject newlines at will, by as Ayn noted, I would then need css modifcations or some other magic to make it work in the HTML UI, which seemed cumbersome. Rather than bending Splunk to my will, but I found that I could get what I was looking for by altering the search to split by permutations (one event returned per permutation) instead of trying to list out all the permutations with line breaks inside of a single event.
... View more
11-01-2012
10:13 PM
1 Karma
Do you already have a Field Extraction set up for the response_time field? If you do, you can simply use greater and less than signs in the search:
index=weblogs response_time>10
... View more
11-01-2012
10:07 PM
1 Karma
I can verify that this is happening with our instances after the 5.0 upgrade, as well. Could be a change to the $SPLUNK_HOME/etc/log.cfg file, but I don't have an old copy handy to compare against.
... View more
11-01-2012
10:02 PM
I recently got this working on my heavy forwarders by creating $SPLUNK_HOME/etc/system/local/outputs.conf and adding the following lines:
[tcpout]
forwardedindex.filter.disable = true
You can read more in the outputs.conf documentation.
UPDATE
If data is placed in main rather than _internal, the inputs.conf definitions must have either been changed or duplicated, with either no index, or the main index specified (because main is the fallback). Double check $SPLUNK_HOME/etc/system/default/inputs.conf -- it should look like this:
[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal
If you're on Linux, find all input stanzas for $SPLUNK_HOME/var/log/splunk with:
sudo find /opt/splunk/etc -name 'inputs.conf' -exec grep -H "/var/log/splunk" {} \;
Do this on the indexer and forwarders and examine the files to ensure that index = _internal
... View more
10-25-2012
03:47 PM
Anyone get Scripted Auth working in 4.3 (or any other version) where the user's email address is populated? I can't find any way to do this in the documentation, but seems necessary because so much of what can be done with Splunk assumes that you have an email address. What makes this much worse is that users who have been populated with Scripted Auth cannot be edited from within the Manager UI, so I am unable to set the users' email addresses manually, as well.
I am currently using PAM auth (by necessity -- don't ask), and I can easily use an algorithm to set the user's email address, if only I knew where. In that past I've used Splunk with AD/LDAP auth and had no trouble with email address population.
... View more
08-03-2012
01:42 PM
Us as well. We are using Splunk for graphing website performance and analytics, and I would LOVE to put annotations on the graph to show whenever we push new code to the website, so that we can try to better visualize the impacts of those pushes in near-real time (which may occur several times per day).
... View more
07-08-2012
10:03 PM
1 Karma
This sounds like the same thing that happened to us on debian squeeze. Check out this post.
To fix, on the ubuntu server having issues, try these commands:
/etc/init.d/ntp stop
(date +"%H:%M:%S" |perl -pe 'chomp';echo `date +"%N"` / 999999999|bc -l) | sudo perl -ne 'chomp;system ("date","-s",$_);'
/etc/init.d/ntp start
If you don't have perl installed for some reason, you can replace the middle date command with a simpler one referenced in the post above.
... View more
