About Lowell

Lowell · ‎09-28-2011

I'd suggest that you also checkout this question and answer: http://splunk-base.splunk.com/answers/7479/what-is-the-best-way-to-retrieve-data-from-a-remote-site-splunk-index There's a comment about the isReadOnly setting in indexes.conf which you may want to look into as well. The idea is to prevent accidental writing to a replicated index. It's possible that read-only file permissions may also be a good idea in this case. (BTW, I assuming that you aren't going to try to let both sites index data into both each others directories, because that will not work.) It may be a good idea to work though a plan with splunk support for your specific setup.

Lowell · ‎09-28-2011

Oh, you should also be aware of a bug in early 4.1.x releases: http://splunk-base.splunk.com/answers/3238/can-a-custom-search-command-launch-a-splunk-search

Lowell · ‎09-28-2011

It looks like push-version.txt is the file you are talking about. I've added an answer with a simple script to do increment this value. Thanks for leading me in the right direction. (BTW, I think I often forget to replace the "favicon.ico" until after I've started splunk after an upgrade, so I'll add this logic to my "post-install" script and that should make sure the right icon gets seen by the users.)

Lowell · ‎09-28-2011

For anyone following along at home. Here's a shell script I wrote to do increment the version number from the command line. Here's the contents of do_bump.sh : #!/bin/bash # Default splunk home if not already set SPLUNK_HOME=${SPLUNK_HOME-/opt/splunk} bump=`cat $SPLUNK_HOME/var/run/splunk/push-version.txt` echo "Current version: $bump" let bump++ echo -n $bump > $SPLUNK_HOME/var/run/splunk/push-version.txt echo "New version: $bump (Restart splunkweb?)"

Lowell · ‎09-27-2011

Just FYI, if you use a custom search command instead of a external "lookup" you can pass in authentication and not have to hard-code your login credentials; which raises security concerns. (I created an app that provides an example of that http://splunk-base.splunk.com/apps/22370/runsavedsearch-alert-action)

Lowell · ‎09-27-2011

I think I can clear this up for you. Standard error (stderr) gets logged to the _internal index ONLY for scripted inputs--which are used to feed some kind of custom data into Splunk. It sounds like you are creating a custom search script, which is the mechanism used to inject custom code/logic into a search. (There are also external lookup commands which can also be python scripts, and alert triggered scripts which can also be written in Python.) These are all used for different situations, but yet do share some fairly common themes. But I'll not go into all of that now.... If you are trying to create a custom search command, then the easiest approach I'd suggest adding the following code to the top of your python script: import logging as logger logger.basicConfig(level=logger.INFO, format='%(asctime)s %(levelname)s %(message)s', filename=os.path.join(os.environ['SPLUNK_HOME'],'var','log','splunk','NameOfMyCustomSearchScript.log'), filemode='a') (You can also use "python.log" if you want, there's nothing stopping you; but you may want to make sure that you use the same format string as the other processes that write to that log.) Then simply change any sys.stderr.write() calls to logger.info() or logger.error() or whatever severity you want. Hope that helps.

Lowell · ‎09-27-2011

I'd like to script a _bump call after replacing the favicon.ico . (This is something I do after each splunk install or upgrade.) Is there an quick way to do this? Preferably, is there are way to do this without requiring splunkweb to be running?

Lowell · ‎09-19-2011

If I'm understanding what your trying to do correctly, it sounds like you want to launch a sub-search for each and every event returned by your base search ( sourcetype=foo ). It also looks like you only want the sub-search to run conditionally if you don't have a valid clientIP value. Does that sound correct? First off, I don't think there is any way to conditionally launch a subsearch. Each subsearch is only run once, and is evaluated and expanded into the main search, and then the main search runs. Now, you can launch a search per-event using the map command, so you may need to do something like that, however in that case, the output from the individual "map" run search is our final output, so you lose all the events from your base search in the output. So, you'll need to do all of this inside of a sub-search and then recombine the two sets of results using a join command, or something... For the record, this is a complete guess. This would take lots of examples and probably a few hours of messing around to get something that actually works properly... sourcetype=foo | join type=outer user [ search sourcetype=foo clientIP=="x.x.x.x" | map search="search earliest=$earliest$ latest=$latest$ sourcetype=proxy user=$username$" | fields _time, clientIP, user ] I'm not sure if you can get join to do the correct time correlation that you need. It's possible that you'll need to do some sort of funky transaction instead (since it's the only search command that lets you pull events together based on a range of time, but there are also a number of down-sides to that approach.) Good luck 😉 Alternate approaches: Don't forget that you can always programatically call splunk searches. It sounds like you have something complex enough that it may warrant that kind of effort, and that would give you FULL control. It's also possible that generating lookups periodically could be a much better way to handle this.

Lowell · ‎09-19-2011

You may be able to use a field search: NOT username="A" ... Sometimes it's necessarily to filter out a single user at the very end of a search, in which case you can tack a "search" command on to the end, like so: ... | search NOT username="A" If this doesn't work, please edit your question and provide the search you are starting with.

Lowell · ‎09-19-2011

Whoops! I found the real issue, the problem is not the indexing throttle (which only appears to be turned on every once in a while for a relatively short period of time.) The real issue is that I was monitoring splunk performance with a dashboard that was doing a realtime search across all of my indexes. These events in splunkd.log show the real culprit: 09-19-2011 11:22:51.382 INFO IndexProcessor - rtsearch connection established, filter = '[ OR index::_internal [ AND index::* [ OR index::history index::main index::os index::sample index::sos index::splunkit_idxtest index::summary ] ] ]', _activeStreams = 1, queue_size = 10000, blocking = FALSE, max_block_secs = 0 09-19-2011 11:24:36.450 INFO IndexProcessor - rtsearch connection terminated, filter = '[ OR index::_internal [ AND index::* [ OR index::history index::main index::os index::sample index::sos index::splunkit_idxtest index::summary ] ] ]', _actionStreams = 0 As soon as I stopped this search, the indexing performance jumps right back up to where it should be. Mystery solved. BTW, it looks like the index throttle was never applied on my system for more than 3 seconds at a time, and normally it was right around 1 second. For anyone interested, here's a search that gives some additional insight into the length of time the indexing throttle was applied: index=_internal sourcetype=splunkd databasePartitionPolicy indexing throttle | rex "throttle for (?<path>\S+)" | transaction path startswith="applying" endswith="released" | timechart max(duration)

Lowell · ‎09-19-2011

You may want to consider using "fschange" instead of monitor. This will (1) give you a specific event when a new directory is created, and (2) this could be a more natural fit for one-time generated crash dump files. This means that splunk will perodically check for changed or new files, but it will not be polling as agressivly as it would with a standard "monitor" input. You can also have control over weather or not you want to actually index the entire file, or if you just want to see that one was created (all depending on your desired use.) I would suggest you start by reading the Monitor Changes to your file system docs. It would certainly be possible to look for the creation of new folders based on newly appearing sources in your index, but this would require (1) storing of state between your searches to know when something new appears, and (2) it would require some assumptions about the timestamps of your events and any indexing delays.... So yeah, it could be done, but I suspect that fschange will be a more straightforward solution.

Lowell · ‎09-16-2011

I'm trying to do some basic performance benchmarking using SplunkIT on a replacement splunk instance I'm building. Specifically, I'm trying to compare different Linux file systems to see how they preform in comparison with each other. (I'm considering ext3 , xfs , and ext4 ). The problem that I'm running into is that the indexing performance seems to be strongly impacted by indexing throttling and therefore doesn't seem to accurately reflect the differences in file system performance, rather I'm seeing just the difference between how long easy test was stuck in "thottled" mode. Here are some example splunkd messages I'm seeing: 09-16-2011 23:04:32.733 WARN databasePartitionPolicy - applying indexing throttle for /opt/splunk/var/lib/splunk/splunkit_idxtest/db because bucket has too many tsidx files, is your splunk-optimize working? 09-16-2011 23:04:33.462 WARN databasePartitionPolicy - released indexing throttle for /opt/splunk/var/lib/splunk/splunkit_idxtest/db Does anyone have any experience with the impact that index throttling has on the SplunkIT indexing test?

Lowell · ‎09-13-2011

With scripted inputs, you could simply write out a status code via the standard error output stream. (Most scripting languages and unix utilities will do this by default when the encounter a problem or programmatic exception). Splunk will index any such "error" messages in the internal index automatically so you can track down problems with your scripted inputs. If you are using python, you can write to standard error using the command like: sys.stderr.write("Job complete: status=SUCCESS\n") I use a search like this to periodically report any errors from my various scripted inputs: (I also filter out a bunch of Splunk's built-in scripted input errors that happen frequently, but I don't really care about.) index=_internal sourcetype=splunkd component=ExecProcessor "message from" NOT (splunk-regmon OR splunk-wmi) | rex " - message from \"(?<inputscript>[^\"]+)\"" | rename inputscript as script | rex mode=sed "s/^.* - message from \"[^\"]+\" (.*)$/\1/" | transaction fields="host,source,script" maxpause=45s One this that isn't clear: Are you using a scripted inputs simply for it's scheduling capabilities? You made it sound as though you aren't sending any data to the indexes by default. If this is correct, then I'd really suggest using a different scheduling mechanism and simply writing out your status information to a log file and then monitor that file with splunk. CRON or the windows task scheduler work well.

Lowell · ‎09-12-2011

I'm planning on migrating Splunk instance to new hardware and will be upgrading the OS at the same time, and I'd like to move to "ext4" due to the extent-based file allocation, online fragmentation capabilities, and so on. There's been a couple questions about ext4 on this site and per splunk support there is still no official "ext4" support yet as of Splunk 4.2.3, so I figured I'd see if anyone in the community has any experience on this. If you're using ext4 I'd love to know about it. Please include the following: (Additional details welcomed) Linux distro: Kernel version: Splunk version: Comments: (Works well / has problems / is it notably faster or slower?)

Lowell · ‎08-30-2011

Nope. Splunk is doing that itself, and I can't make it NOT wrap the process in a shell. Hence I posted this question.

Lowell · ‎08-30-2011

Out of curiosity, does your /bin/sh point to dash on your Ubuntu box?

Lowell · ‎08-30-2011

I've tried this approach, but the problem I had was that the "splunkd" process was killing the shell script that was wrapping the python script; and the signal never got passed down from that shell to the python process. It could be a version/OS thing. What OS are you using, and which version of Splunk?

Lowell · ‎08-29-2011

Doh! I needed a splunkweb restart. (http://splunk-base.splunk.com/answers/24407/applicationjs-not-loading) Now "application.js" is loading, but it doesn't appear to be doing anything. I actually found different workaround that suites my needs; so I'm just going to assume that I'm doing something wrong and just walk away from this for now. (All the warnings with the custom behavior docs have scared me off.) Thanks for your help.

Lowell · ‎08-29-2011

I'm having some trouble getting this to work; because "application.js" isn't being loaded by my view. Do I have to do something special to get that script to load?

Lowell · ‎08-25-2011

Do you actually have 3 different IDs that you are trying to link together? I'm slightly confused by this. Are you renaming "PBTS_ORIG_SIP_CALL_ID" twice? Do the different fields come in from different sourcetypes/indexes? Or all they all mixed around. I suspect a few example events would help clear things up.

Lowell · ‎08-25-2011

Good to know. This was very helpful, thanks!

Lowell · ‎08-25-2011

Do you have the deployment client enabled in deploymentclient.conf ? [deployment-client] disabled = false Not 100% sure what the default is for "disabled" in that stanza, but it's worth a try.

Lowell · ‎08-25-2011

This is now possible with the Pager module in Sideview Utils (which was written by nick). From the Splunk.Module.Pager docs: Provides page links. ie "Prev 1 2 3 4 5 Next". This module behaves much like Splunk's Paginator module, except that Pager accounts for postProcess searches when they are present and automatically renders the correct number of page links. Make sure to read all documentation within the Sideview Utils app for many more key details.

Lowell · ‎08-25-2011

Is the usage of the ViewstateAdapter module needed or recommended for view that use the Sideview Utils modules? I did a quick search and didn't find any uses of ViewstateAdapter in any of the examples: fgrep ViewstateAdapter ~/etc/apps/sideview_utils/default/data/ui/views/*.xml No matches. I'm guessing that means that we don't need this module?

Lowell · ‎08-25-2011

I've been converting several of my simple XML forms views into Advanced XML views using Sideview Utils with fairly good success, but I'm not sure if I'm doing everything properly. I've run into at least one view that doesn't work on IE7. (Now the Sideview util version of my view is way more complex (and 10x more helpful) than the basic simple XML "form" that I started with, so that could be part of the issue, but it's also possible that I'm not wiring all the modules properly.) Are there any helpful docs or guidelines for "converting" between a simple XML "form" to using the advanced XML with the Sideview utils modules? For the purpose of background info, I'll briefly describe the process I've been using: Basically, I tack on ?showsource=true to the URL of my form view--which gives me the advanced XML representation of my simple XML form. Then I copy the XML (to a temporary view, so that I don't break the original) and convert it to a Sideview Util based view, which is loosly done by: (1) replace all the user input modules for the Sideview util equivalents (e.g. swap out ExtendedFieldSearch with TextField ). (2) I strip out any references to intentions, add the SideviewUtils and URLLoader modules at the top, (3) I swap out any HiddenSearch modules for the Search modules. Often I can also combine several searches and use PostProcess which makes the view faster and less resource intensive. (4) I strip out all modules that don't appear to be needed, or that I don't want, such as: HidenFieldPicker modules that don't have any parameters, and EnablePreview , and ViewstateAdapter . My apologies if I've missed something in the docs or a how-to or best-practices doc out there.

Posts	1343
Solutions	217
Karma Given	1014
Karma Received	1581
Member Since	‎03-26-2010

Online Status	Offline
Date Last Visited	‎05-26-2021 05:52 PM

Is passAuth still supported for scripted Inputs?

Does inputcsv honor dispatch=true?

Can a search detect it's own sample ratio?

How do you update lookups on a SHC while running S...

How do create indexed fields in a summary index?

What is a managed app in Splunk Enterprise Securit...

How to use a custom token filter within Simple XML

Can you use appendpipe and map together?

Is there a way to pick a point in time (not a time...

How do I run a programmatic search against a Searc...

Re: Move Index Files from one system to another

Re: dynamically rewrite field value by a eval + s...

Re: Can I issue a "_bump" from the command line?

Re: Can I issue a "_bump" from the command line?

Re: dynamically rewrite field value by a eval + s...

Re: sys.stderr Not logging to splunkd

Can I issue a "_bump" from the command line?

Re: dynamically rewrite field value by a eval + s...

Re: Removing certain results from reports

Re: How do I disable index throttling when running...

Re: Alert on The Creation Of New Folders

How do I disable index throttling when running Spl...

Re: Best Practice - Logging Script Runtime Results

Quick poll: Anyone using ext4? Have you run into...

Re: Scripted input without a shell?

Re: Scripted input without a shell?

Re: Scripted input without a shell?

Re: How can I hide content on my view until a sear...

Re: How can I hide content on my view until a sear...

Re: Deduping when working with multiple sourcetype...

Re: Is ViewstateAdapter needed with Sideview Utils...

Re: Deployment Client not finding deploymentclient...

Re: HiddenPostProcess pagination issue

Is ViewstateAdapter needed with Sideview Utils?

What's the best way to convert a simple XML form t...

Join the Conversation