About Lowell

Lowell · ‎08-24-2011

@nick: Nevermind, I didn't have the !important in there properly. It works on IE now. Doh!

Lowell · ‎08-24-2011

Nick, I'm having issues doing this with IE (tried 7 and 8). There seems to be a style="width: 50%" attribute added to the div tag (possibly via javascript?) that is overriding my CSS. Any thoughts?

Lowell · ‎08-24-2011

Using the search language to search for a literal * is currently unsupported. One workaround is to disable "*" as wildcard and then you can search for it as a literal, but but then you no longer have any wildcard capabilities; which was unacceptable to me to so I did peruse that option with Splunk support. This is listed on the "Known Issues" page as follows: There is no way to escape an asterisk (*) in the search language. (SPL-30079) Just to be clear, you can filter for a " * " using a post-processing search command (e.g. by using rex like in the answer above), but you can't actually search for one in your actual search.

Lowell · ‎08-23-2011

Note about Ubuntu. On my 8.04 install, the "pam_limits" module wasn't enabled by default for the "su" command, and therefore the "splunk" init.d (startup/shutdown) script which prevented the appropriate limits from being applied. Here's what I had to do: First, edit /etc/pam.d/su : You will need to add (or uncomment) the following line: session required pam_limits.so Second, edit /etc/security/limits.conf : Add the following "nofiles" limit for the "splunk" user (or whatever user your splunkd process runs as) splunk soft nofile 4096 splunk hard nofile 8196 Note, you may also want to enable "pam_limits.so" for "sudo" as well, if you ever use that tool too login interactively restart splunk services. Update: This appears to still be true for default Ubuntu 12.04 installs.

Lowell · ‎08-19-2011

Paul, note that the backslashes before the "d"s were removed. The text formatting is a bit messed up unfortunately. 😞 The suggestion here is right on!

Lowell · ‎08-19-2011

I'm not sure this approach will work fundamentally. If you are consistently producing more logs than you can index, simply slowing down the transfer rate (or placing a cap on it) will only cause the forwarder in question to fall further and further behind realtime and will not ultimately change your total indexed volume once the indexing catches up. You'd have to actually push the indexing time across days for this to make a difference to your daily licensing level; which means that your events are going to be rather old and out of data by the time they get into splunk. There may be other approaches like filtering out unwanted/unhelpful events. I've seen truncating stack traces to 2k make an enormous volume difference for some of our java-based application servers (the stack trace weren't very helpful to begin with.) In other cases we've had run-away (endless loop) produce massive logs; in which case I setup a splunk alert to email our support team and get the client app to shutdown. (Normally, splunk indexes so quickly that deleting the offending log file can't be done quick enough--that's one place were limiting the volume could help.) The bottom line is that we actually reduce the amount of data being indexed, rather than simply slowing it down. BTW, are you sure your "grep" is getting you the right information? I don't see you search for a specific forwarder host. In case you didn't know, you can look for this same info inside splunk with a search like this: index=_internal source=*metrics* "group=tcpin_connections" | rename _tcp_KBps as tcp_KBps | table sourceHost, kb, tcp_KBps

Lowell · ‎08-19-2011

The built-in email alerting functionality is fairly limited to the degree at which you can customize or control the content of the email itself. There's lots of stuff I'd like to be able to do that I can't right now, for example: Issue a "CC" (or BCC) instead of just "TO". (I'd also like the ability to note splunk user names, and then let splunk convert that to their listed email address.) Automatically show "_time" as something humanly readable. (I know there are timezone issues here.) Customize or control the layout or content of the email body. I'd like to be able to hide (or push to the bottom the actual search string.) Because (1) non-technical or non-splunk users just find that confusing, and (2) for me, I have some long complicated searches with just a small table; so I have to scroll around for awhile on my blackberry just to get to the results table. *. I'd like to be able to add some simple template text to the output. (Or perhaps, just include the searches "description". We have several alerts that need additional explication.) I'd like to be able to alter the subject line independently of the search name. It would also be nice to include some value from the results within the subject. (Something like " Disk Error on host $results[0].host$ "). Modify the from address per search. Add links to splunk views. Either within the body of the email, or within the table results (for a per-row hyperlink). Including more than one search would be helpful from time to time. (Sure you can email PDF views, but they are pretty limited, and they load much slower than HTML content; especially on mobile devices.) That's just off the top of my head. I know I can go hack the sendemail.py but that's a long-term maintenance nightmare. Has anybody build some kind of app that augments/hijacks/replaces the out-of-the-box email alerting functionality?

Lowell · ‎08-15-2011

How do I manually check the stored "hash" in a splunk fs_notification event and compare it against the a file on the file system? The docs say that the fschange input generates a hash value using the SHA256 algorithm. I have the sha256sum utility installed, but it's output looks completely different. Anyone know how to verify the hash of a file manually? Example: Here is my fs_notification event: Thu Aug 4 12:08:31 2011 action=update, path="/home/myuser/SignOn.vb", isdir=0, size=604392, gid=65534, uid=1002, modtime="Tue Jun 7 12:46:32 2011", mode="rw-r--r--", hash=zXd0LxXGDoNzeewkquKeksEV/PhnhtAakFmfMjkpmtU=, chgs="modtime hash " Looking at the file on my file system: ls -ln /home/myuser/SignOn.vb -rw-r--r-- 1 1002 65534 604392 2011-06-07 12:46 /home/myuser/SignOn.vb sha256sum /home/myuser/SignOn.vb 1cf9c9267c0183d2e248d6318a289796302dd12ddde4712b010a4fadfc008fad /home/myuser/SignOn.vb I've compared the file size and modification time, but those are easy to fake; I want to be sure these files are the same. BTW, I've tried simply converting the hex to base64, but that doesn't seem to be enough. It seems like there is some kind of salt or prefix added before the hash is calculated... I'm not sure, any thoughts?

Lowell · ‎08-12-2011

Does anyone know if Splunk plans on building a "non-simple" version of the SimpleResultsTable module? Or has anyone else packaged up an app with a updated or replacement module? I've seen a bunch of questions out there about ways to hack this module in a number of different ways (just search for "SimpleResultsTable" and you'll see what I mean). I know that Sideview Utils add some functionality based on some posts I've read, but there still seems to be a general lack of ability to some basic formatting without adding custom hacks. For example: Colorizing row/cells, or per-column settings like setting the cell with or text justification. I'm really just digging into the whole view development thing, and I'm just wondering what direction things are moving. If just hacking at the existing module is the only (or "best") option, then I'll go that direction too, but before I go to all that effort, I figured I'd ask around 😉

Lowell · ‎08-11-2011

I noticed that some stuff doesn't show up on your view until after the search has been started. For example, the ViewRedirectorLink module doesn't show the "View results" link until the search has been kicked off. Is it possible to do this with other modules too? I'm specifically trying to hide some text in an HTML module (which is part of the Sideview Utils app). I'd like it to only show up after the search starts. I'm looking for something fairly simple, if at all possible.

Lowell · ‎08-11-2011

I'd suggest that you check out the Sideview Utils app. It comes with a URLLoader module which lets you do this quite easily. You can download and install the app, then go to the application's home page for a tutorial of the various features. http://splunk-base.splunk.com/apps/22279/sideview-utils I've really just started to using this app within the last few weeks, and already it's proven to be invaluable! I'm never going back to plain old Advanced XML!

Lowell · ‎08-09-2011

I know your the splunk guru here, but I don't think you're completely right on this one. Splunk does appear to kill the wrapping shell process when the scripted input is disabled. I've added some additional info to the original question.

Lowell · ‎08-09-2011

I got it to work in Linux. (I had forgotten to use the absolute path to my executable. Whoops!) However, this STILL caused a shell ( /bin/sh -c ) process to be launched. See http://splunk-base.splunk.com/answers/28733/scripted-input-without-a-shell

Lowell · ‎08-09-2011

Is it possible to create a scripted input that is launched directly from splunkd and not from a shell? I've tried shell script, a python script, and a .path file, and they all seem to be wrapped by a " /bin/sh -c <MY_COMMAND> " wrapper shell. I have long-running scripted input, and the process is not being shutdown when splunkd restarts, which I think is due to the extra sh process not passing down the kill signal to my process. Update / Additional info: I know that splunkd is stopping the shell whenever splunkd is shutdown or whenever the scripted-input is disabled. (Note: To save time during testing, I've been enabling and disabling my scripted input stanza in inputs.conf , then issuing the the following refresh link: http://mysplunkserver:8000/en-US/debug/refresh?entity=admin%2Fscript , which has been working like a charm.) Whenever the input is disabled (or splunkd shutdown), the shell wrapper process goes away, but the child process (aka, my scripted input program) continues to run. But instead of my process being a grandchild of splunkd , now it's directly under process 1 (init). Here are two examples showing the processes running on my system (output generated by pstree -A -p ). Example 1: This shows my scripted input when it's enabled. My scripted input process is pid 4177, with several threads. init(1)-+ |-splunkd(2642)-+-splunkd(2643)-+-sh(4176)---java(4177)-+-{java}(4185) | | | |-{java}(4186) | | | |-{java}(4187) | | | |-{java}(4188) | | | |-{java}(4189) | | | |-{java}(4191) | | | |-{java}(4194) | | | |-{java}(4195) | | | |-{java}(4201) | | | |-{java}(4202) | | | |-{java}(4203) | | | |-{java}(4204) | | | |-{java}(4205) | | | |-{java}(4207) | | | `-{java}(4211) Example 2: I then disabled by input ( disabled=1 in inputs.conf ), then refreshed the "admin/script" entities, and now my process tree looks as follows: (Note that the "java" process is now owned by init, and the wrapper shell (4176) is now gone. init(1)-+ |-java(4177)-+-{java}(4185) | |-{java}(4186) | |-{java}(4187) | |-{java}(4188) | |-{java}(4189) | |-{java}(4191) | |-{java}(4194) | |-{java}(4195) | |-{java}(4201) | |-{java}(4202) | |-{java}(4203) | |-{java}(4204) | |-{java}(4205) | |-{java}(4207) | `-{java}(4211) |-splunkd(2642)-+-splunkd(2643) The problem seems to be that the wrapper shell ( /bin/sh ) is simply not passing on the kill request. Note: As shown above, I'm using my own wrapper script to setup the environment and launching the java executable using exec to prevent an additional shell layer in the mix. I've messed around with using traps and such (when I wasn't using exec , of course), but ultimately if the parent process (aka the /bin/sh wrapper shell) doesn't pass down the signal, there's nothing to trap. My only other option is implementing some kind of polling mechanism to see if my parent process is dead. So once again, I'm back to: How do I turn off that annoying wrapper shell and keep things simple? I'm running Splunk 4.1.8 on Ubuntu 8.04 (32 bit) and /bin/sh is currently using dash (which is the Ubuntu default)

Lowell · ‎08-09-2011

Does this work for platforms other than Windows? (For example, on Linux)

Lowell · ‎08-08-2011

You can add a comment/note to the docs (online), or email [email protected]. Otherwise this comment may go unnoticed.

Lowell · ‎08-05-2011

Your "_raw" field wouldn't exist anymore after your | stats values(req_time) . Not sure if that's relevant to your entire search or not. Have you tried using a field name other than _raw (since it has a special purpose) while debugging your search? Sometimes renaming the field and then naming it back lets you get around some internal assumptions built in to some of the search commands. For example, you could try: ... | eval raw=split(_raw, "\n") | mvexpand raw | rename raw as _raw | ...

Lowell · ‎08-05-2011

That's true. It depends on what you are looking for. rachelneal, if you would provided additional details in your question (use the "edit" link below you question), then a more suitable regex could be suggested. Specifically, note what strings you would like to extract from the samples given.

Lowell · ‎08-05-2011

Can anyone help explain what the "objectMode" attribute is used for when defining an advanced XML view? I've seen a few different options, like "SimpleForm", and "viewconf", but I can't tell what difference this makes? How do you know which one to pick? Example: <view autoCancelInterval="90" isVisible="false" objectMode="SimpleForm" onunloadCancelJobs="true" template="dashboard.html">

Lowell · ‎08-04-2011

Is there a space between the last word and the "(". If so, your regex isn't going to match. Try: | rex "(? \w+)\s+("

Lowell · ‎08-03-2011

I run into a number of issues like this before, and this is my theory: Splunk use the dynamic input header to set the source/souretype/host/index of the events being processed (as you'd expect), but it doesn't do the normal props.conf processing based on the newly assigned source/source/host; it simply processes it based on whatever props rules the input would have used if no dynamic input header existed. I'm not sure if I've explained this well. Here's another example. You can use a transformer to rewrite the "source", and splunk will index your event with whatever new source you've assigned in the transformer. When you assign a new "source", splunk does NOT go back through all of your props.conf files looking to see if the new source value matches any stanzas--it simply uses the original props.conf rules. I think the dynamic input header has a similar limitation. One thing I would try is to give your spool files a unique file pattern. (Like say, they all contain "MY_SOURCETYPE" in the middle of the filename.) Then setup a source pattern matting rule to match that source pattern in your spool directory, and use that to assign "mysource". If you are trying to do this with multiple sourcetypes, then this get's more tricky. For example: props.conf [source::...MY_SOURCETYPE.*] sourcetype = mysourcetpe BTW, if you are trying to pass generic (multiline) messages into splunk, then you may find the following helpful. I've attempted to so something similar, and this is the best solution I've come up with: (This example talks about sending in events over TCP, but I use the same sourcetype for file-based input as well, and it works well there too.) http://splunk-base.splunk.com/answers/7494/issues-creating-a-gateway-to-create-splunk-events-from-an-integration-platform/28493

Lowell · ‎08-03-2011

Here's the solution I came up with. I created by own "dynamic" header (which simply uses a bunch of transformers) because I couldn't get the built-in dynamic header to work for me in this case. Each message is send to splunk on TCP port 9797. And the format of each message is like this: SPLUNK-MESSAGE: sourcetype="mysourcetype" source="mysource" host="myhost.domain.com" index="myindex" My multiline message goes here... ---END-OF-MESSAGE--- Splunk configuration Entry in inputs.conf [tcp://9797] source = tcp-9797 sourcetype = _generic_message Entries in props.conf : [_generic_message] # Messages can optionally start with a header: # SPLUNK-MESSAGE: sourcetype="st" source="s" host="host" index="index" # All messages must be followed by a tailer: # ---END-OF-MESSAGE--- SHOULD_LINEMERGE = False LINE_BREAKER = ([\r\n]+---END-OF-MESSAGE---[\r\n]+) MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 # Handle the generic header (notice that order is important here: drop-header must be last) TRANSFORMS-aa_genmsg-header = genmsg-set-index, genmsg-set-host, genmsg-set-source, genmsg-set-sourcetype, genmsg-drop-header TRUNCATE=1000000 Entries in transforms.conf : ### Generic message header line (for setting host/source/sourcetype/index) [genmsg-set-sourcetype] REGEX = ^SPLUNK-MESSAGE:[^\r\n]*\bsourcetype="([^"\r\n]+)" DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::$1 [genmsg-set-source] REGEX = ^SPLUNK-MESSAGE:[^\r\n]*\bsource="([^"\r\n]+)" DEST_KEY = MetaData:Source FORMAT = source::$1 [genmsg-set-host] REGEX = ^SPLUNK-MESSAGE:[^\r\n]*\bhost="([^"\r\n]+)" DEST_KEY = MetaData:Host FORMAT = host::$1 [genmsg-set-index] REGEX = ^SPLUNK-MESSAGE:[^\r\n]*\bindex="([^"\r\n]+)" DEST_KEY = _MetaData:Index FORMAT = $1 [genmsg-drop-header] REGEX = (?s)^SPLUNK-MESSAGE:[^\r\n]+[\r\n]{1,2}(.*)$ DEST_KEY = _raw FORMAT = $1

Lowell · ‎08-03-2011

Yes, the issue was caused by multiple inputs all using the same "source" name, and simply overriding the "source" was all it takes to prevent these inputs from getting intermingled: My working inputs.conf file: [script://.//bin/query_performance.sh --settlement-queue --block-info] disabled = false index = osperf interval = 30 source = query_performance-1 [script://./bin/query_performance.sh --cache-info --perf-info] disabled = false index = osperf interval = 300 source = query_performance-2

Lowell · ‎08-03-2011

According to V_at_Splunk, in 4.2, you should be able to use a new parameter called -rename-source , and therefore this work around shouldn't be necessary. (This should still work in 4.2. That said, I'm not sure if the new HEADER_MODE props.conf setting will have any impact on this or not.)

Lowell · ‎08-02-2011

Do you really want Splunk choosing (own its own) to just drop certain events out of your log files?

Posts	1343
Solutions	217
Karma Given	1014
Karma Received	1581
Member Since	‎03-26-2010

Online Status	Offline
Date Last Visited	‎05-26-2021 05:52 PM

Is passAuth still supported for scripted Inputs?

Does inputcsv honor dispatch=true?

Can a search detect it's own sample ratio?

How do you update lookups on a SHC while running S...

How do create indexed fields in a summary index?

What is a managed app in Splunk Enterprise Securit...

How to use a custom token filter within Simple XML

Can you use appendpipe and map together?

Is there a way to pick a point in time (not a time...

How do I run a programmatic search against a Searc...

Re: Is it possible to specify the width of a panel...

Re: Is it possible to specify the width of a panel...

Re: literal * in search?

Re: how to tune ulimit on my server ?

Re: Line_Breaker question

Re: Limiting Forwarding Throughput

Has anyone built an enhanced email alerting app?

How do I manually check a fschange hash value?

Has anyone packaged a replacement for the SimpleRe...

How can I hide content on my view until a search r...

Re: How to Pass value from one view to another vie...

Re: Scripted input without a shell?

Re: I see splunk has some ".path" files in windows...

Scripted input without a shell?

Re: I see splunk has some ".path" files in windows...

Re: Getting a field value from the previous event

Re: After transaction: "Field '_raw' does not exis...

Re: Problem with the simplest rex

What does objectMode mean for a view?

Re: Problem with the simplest rex

Re: Batch input doesn't honor LINE_BREAKER setting...

Re: Issues creating a gateway to create splunk eve...

Re: Problems with a dynamic input header in a scri...

Re: How do you override "source" on a oneshot?

Re: how to ignore the titile line of the csv file ...

Join the Conversation