About kkalmbach

kkalmbach · ‎11-17-2011

I have some aix boxes and the fields being printed out by the df scipt are not correct. (the fields get off). The problem is that the first call to df (/usr/sysv/bin/df -n), does not return every filesystem that the second call to df (/usr/bin/df -kP) returns. So when this part of the format line runs: $2=fsTypes[mountedOn] $2 becomes empty (blank). Then when it's printed out there are fewer columns than the header states and the values are shifted. Here is the my new FORMAT line (I just added in the if at the end): FORMAT='{size=$2; used=$3; avail=$4; usePct=$5; mountedOn=$6; $2=fsTypes[mountedOn]; $3=size; $4=used; $5=avail; $6=usePct; $7=mountedOn; if ($2=="") {$2="?"}}' Does that look good? If so, can you add it into the next version of *nix? Thanks -Kevin

kkalmbach · ‎11-11-2011

Thanks, That's close to what I was thinking, but since we deploy the same app to several forwarders, I was thinking about something like: if (NF == 8) FORMAT='{cpu=$(NF-5); pctUser=$(NF-4); pctNice=$(NF-3); pctSystem=$(NF-2); pctIowait=$(NF-1); pctIdle=$NF}' else (what is currently there) The other thing I though about was to just grab the header from the output (change % to pct) and use that for a header. For now I'll probably go with the first option, but do you see a problem with the second?

kkalmbach · ‎11-10-2011

I have several linux boxes that are being monitored by splunk and a few of them are having problems running the "cpu.sh" script. The problem is that on some of the boxes, there is not "steal" column in the sar output: uname -a Linux hostname 2.6.9-89.35.1.ELsmp #1 SMP Tue Jan 4 22:30:58 EST 2011 i686 i686 i386 GNU/Linux $ sar -P ALL 1 1 Linux 2.6.9-89.35.1.ELsmp (fralnxnmsapp5) 11/10/2011 02:21:05 PM CPU %user %nice %system %iowait %idle 02:21:06 PM all 0.00 0.00 0.00 0.00 100.00 02:21:06 PM 0 0.00 0.00 0.00 0.00 100.00 02:21:06 PM 1 0.00 0.00 0.00 0.00 100.00 02:21:06 PM 2 0.00 0.00 0.00 0.00 100.00 02:21:06 PM 3 0.00 0.00 0.00 0.00 100.00 Average: CPU %user %nice %system %iowait %idle Average: all 0.00 0.00 0.00 0.00 100.00 Average: 0 0.00 0.00 0.00 0.00 100.00 Average: 1 0.00 0.00 0.00 0.00 100.00 Average: 2 0.00 0.00 0.00 0.00 100.00 Average: 3 0.00 0.00 0.00 0.00 100.00 One more piece of info, the version of sar on the box's are pretty old: $ sar -V sysstat version 5.0.5 (C) Sebastien Godard Usage: sar [ options... ] [ [ ] ] Splunk's cpu script assumes the "steal" column is there. As a result, all the values end up in the wrong column. Is there a fix for this, or should I change the cpu script myself? Thanks -Kevin

kkalmbach · ‎08-29-2011

Thanks for the idea, I'm not really following what you meant in the second part (this field is a subpart of the entire event). I do have this part extracted into a field. Are you talking about a new section in the transforms.prop file?

kkalmbach · ‎08-25-2011

I have a field that looks like this: key1=value1*key2=value2*key3=value3 I put in a stanza in transforms that looks like this: [star_equals] SOURCE_KEY=my_field DELIMS="*" "=" Then I run a search like this: index=something | extract star_equals That works great for everything except the first key/value pair. When splunk first does it's auto extract, it thinks key1 has a value of value1*key2=value2*... When I run the extract command, the value for key1 does not get overwritten. I can't turn off splunk auto extractions (too much other stuff would break). If I put in a "field - key1" before the extract, eveything works great, but I won't know what the first key will be, so that's not an option. Any other ideas?

kkalmbach · ‎08-24-2011

I have a log file that gets rotated (foo.log becomes foo.log.1, foo.log.1 becomes foo.log.2, etc). Every time the file gets rotated, the new foo.log gets a common header (which is more that 256 bytes long) Splunk believes that this file has already been indexed (since the first 256 bytes are identical). I can't add "crcSalt = " since the source filename is always the same. Are there any other ways to get this file read every time it gets rotated? Thanks -Kevin

kkalmbach · ‎06-23-2011

In a dashboard, I have a SimpleResultsTable with a drilldown that uses: <module name="SimpleResultsTable"> <param name="drilldown">row</param> <module name="Redirector"> <param name="url">next_page</param> <param name="arg.key">$click.fields.key$</param> </module> ... In this case "key" is a unique key that will determine the next_page's search. The problem is that I would like key to not display in the results table (just be available for the drilldown). The value of "key" is not anything a user should ever see. Is there anyway to add that field to the table but hide the column, or anything else that will make this possible. Thanks -Kevin

kkalmbach · ‎06-07-2011

I am using the light forwarder on AIX and running into high CPU usage (80-90% of a CPU). We tracked it down to using ellipse in the monitor line in inputs.conf. (even a "*" uses too much CPU). We are using the ellipse so that we can deploy the same inputs.conf to several machines, each with a different directory path. If we tighten it down to a single directory/file, things run great. My question is: Is there any way to have splunk traverse the directory tree less often (once a day or even less frequent), but still read files it already knows about in near real time? I did not see anything in the doc about a setting like that.

kkalmbach · ‎05-27-2011

Nick, Thanks, we are already using sideview, so this will work out great for us. Thanks, -Kevin

kkalmbach · ‎05-26-2011

Is there any way (in a view or otherwise) to determine the current application that you are in? We are going to have basically the same app deployed several times (using links), each app will have a different name and we would like to change the index that we are searching on based upon the app name. If we could get the current url, we could also parse that. I can't really use the default index function because we are searching multiple indexes (a real time index and a summary index) within the same app, but each index will always have the form of "XXX_ " and "XXX_summary_ ". Thanks -Kevin

kkalmbach · ‎04-11-2011

I think that makes sense. Thanks for you answer. I will probably go the macro route. Also, is there a big difference (performance wise) in doing index="..." | search service="abc" compared to putting it all in the same inital search? Thanks again,

kkalmbach · ‎04-08-2011

I seem to be having some problems with extracting fields from the "source" In by props.conf, I have: [my_source] SHOULD_LINEMERGE = false LINE_BREAKER = (---------*) REPORT-get_service = get_service in transforms.conf, I have: [get_service] SOURCE_KEY=source REGEX = ([^/]*)_sndmsg.out FORMAT = service::"$1" This only seems to be half working. In the search, if I do index=myindex sourcetype=my_source service=abc nothing comes back. If I split it out into a separate search: index=myindex sourcetype=my_source | search service=abc then everything works fine. Also if I prepend a wildcard, everything works fine, so this also works.. index=myindex sourcetype=my_source service=*abc If I do an eval l=len(service) it always come back with what I expect. Everything looks fine when I print out the service field, I just can't put it in the inital search. Any ideas would be greatly appreciated. Thanks -Kevin

kkalmbach · ‎04-04-2011

This does not seem to work reliably. The Cookie.py stops processing (and raises the error) when it hits the first error. So if the "Cookie" header contains: good=value; b:ad=value Things are fine with that line commented out. If the value of "Cookie" is: b:ad=value; good=value Then Cookie.py stops processing and the good=value is never read in. I think we need to comment out the raise in Cookie.py

kkalmbach · ‎01-20-2011

Lowell, Thanks for the info. I know it's been awhile, but any progress on making this an app (or cleaning up the code so you could release it)? Thanks -Kevin

kkalmbach · ‎11-02-2010

I am looking to monitor configuration files (not splunk's config files, but a app's config files). I can use fschange to monitor when a configuration file has changed (using fullevent, so I have the contents of the file). I also want to compare a file on box1 vs the same file on box2. I want a list of all files that are different between the two boxes and if they are different, I want to see the diffs (using the diff command?). What is the easiest way to do this. Thanks -Kevin

kkalmbach · ‎10-21-2010

Looking at it closer, it appears there are 2 issues: The first is that splunk fails with cookie names with colons in them (like the referenced issue) The second is that splunk is reporting the wrong cookie name when complaining about a cookie Not sure what we are going to do, but it would be best if splunk could handle the illegal cookie names.

kkalmbach · ‎10-21-2010

I think this is similar to http://answers.splunk.com/questions/3602/can-splunk-accept-cookies-with-colon-embedded but not exactly. It seems that when I have a domain wide cookie set, I can never get to splunk (if I use a fully qualified domain name to access splunk). Even if the cookie names appear valid, I still get an error from splunk. Here is one example. 400 Bad Request Illegal cookie name AtworkEnv Traceback (most recent call last): File "/opt/splunk/splunk/lib/python2.6/site-packages/cherrypy/_cprequest.py", line 581, in respond self.process_headers() File "/opt/splunk/splunk/lib/python2.6/site-packages/cherrypy/_cprequest.py", line 653, in process_headers raise cherrypy.HTTPError(400, msg) HTTPError: (400, 'Illegal cookie name AtworkEnv') The domain wide cookies are set by an ap that we have no control over (and must goto daily). Also, I must use a fully qualified domain name to access splunk (we have different domain at my work). Has anyone found a workaround for this? Thanks Kevin

kkalmbach · ‎07-02-2010

I too am attempting to view a report and have the times show up in a different time zone. Your solution simply changes the time, then displays this incorrect time in the "local" timezone so it appears correct. This does not seem correct to me. (Using an American example) Let's say something happened at noon eastern time (-4:00). This event correctly gets logged as happening at noon eastern, then I display it using my splunk server in the mountain time zone, it shows up as happening at 10:00 (which is correct). If I use your approach, I change the time so that the report thinks it happened at 14:00 eastern, then when that time is displayed for the mountain time zone, it shows up as 12:00. Although the string "12:00" is what I want, this seems a wrong way to do it. This breaks if I decide to print out the timezone as well as the time. It would then show up as "12:00 MST". This also would be fragile if I started to correlate the times with anything else. What I would like is a way to say I want this report to show up in eastern time (even though the splunk server is in the mountain time zone), then all times would show up in the correct format no matter what. My other question is when I view a report on splunk (using the web interface), how does splunk decide what timezone to use for displaying the data, does it use the TZ environment variable of the splunk user on the server? Does it use something in the browser (locale)? Something else? Can this be overwritten per user? -Kevin

Posts	18
Solutions	0
Karma Given	1
Karma Received	11
Member Since	‎07-02-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

df script in *nix for aix not always printing corr...

Splunk for linux not grabbing right fields

make extract command overwrite fields

rotating log files with large headers.

drilldown from a hidden column

CPU usage using the lightforwarder

Determine the current app

using source for a field extractions

configuration monitoring across multiple boxes

Illegal cookie name

df script in *nix for aix not always printing corr...

Re: Splunk for linux not grabbing right fields

Splunk for linux not grabbing right fields

Re: make extract command overwrite fields

make extract command overwrite fields

rotating log files with large headers.

drilldown from a hidden column

CPU usage using the lightforwarder

Re: Determine the current app

Determine the current app

Re: using source for a field extractions

using source for a field extractions

Re: Illegal cookie name

Re: What is the most effective ways to poll databa...

configuration monitoring across multiple boxes

Re: Illegal cookie name

Illegal cookie name

Re: Timezone and Timestamp modification at search/...