Ok, I am working to trim back some of our indexed data. I initially tried to drill down using a basic sum(len(_raw) for all index broken down by various other fields.
The problem is that the sum counts dont match the counts when compared to Splunk license usage for the index.
In this specific test case, I am comparing the Splunk license usage for ONE index for ONE day. I compare it to the byte sum of all of the _raw records for that SAME index for the SAME ONE day. . .
I expected the counts to at least be similar. . .
My query from a Splunk source to get license info. . .
index=_internal sourcetype=splunkd source=*license_usage.log [| rest splunk_server_group=dmc_group_indexer /services/server/info | rename guid AS i | fields i ] | eval gb=b/1024/1024/1024 | join i [|rest splunk_server_group=dmc_group_indexer /services/server/info | rename guid AS i | fields serverName i] | search serverName=*rtp* idx=xyzzy_logs | stats sum(gb) by serverName idx
...yields between 50gb to 53gb per indexer for that ONE index for that ONE day.
index=xyzzy_logs splunk_server=*rtp* | eval leng=len(_raw)/1024/1024/1024 | stats sum(leng) as totalgb by splunk_server | table splunk_server, totalgb
...which yields only 14.7gb to 15.66gb per indexer for the SAME index for the SAME day.
Again, i expected them not to be exactly the same but thought they should be closer than 300%+.
What is splunk licensing counting that does not seem to show up in my indexes?
I tried looking for answers for this. . . i found other posts using similar accepted answers with sum(len(_raw) as a "brute force" way to drill down on sizes. . See Splunk Answer: How to get license usage data for a particular index with a breakdown of usage by a field?
... View more
Thanks. . .
Here, ALL SERVERS, ALL USERS, ALL APPLICATIONS are set to US/Eastern time regardless of location/time zone.
So this server is set to US/Eastern. Problem is the application that is writing this log record want to spit out GMT along with the time in eastern. . .
So I am just trying to ignore the GMT.
I had tried this prefix/format as well with no luck. . . see next answer comments below for more info.
... View more
Next, Thanks Somesoni2,
1. Had not tried to skip past the Fri, with the line break. Unfortunately that did not work either.
What I did realize after trying these as well is that the problem is simpler.
It appears that NOTHING I try for time_format actually does anything. I can just swap the %d, %Y, %H, %M around in any order and the indexed record never changes regardless of what format I have.
Since the time is posting 4 hours off of US/Eastern, I also started trying several TZ= timezones. . . US/Eastern, US/Central, . . . Alaska, Virgin Islands, Germany. . . JUST to try to get the indexed time to change.
Regardless of time_format. . . Regardless of TZ=. . . The posted index time never changes. . . Always 4 hours off.
1. I then decided it must be using some default somewhere. . . So I changed these to go to a new/different sourcetype. . . Same issue.
2. This is strange, because in this SAME props.conf file, there is another sourcetype that I had to change the TZ to UTC to correct a DIFFERENT issue. This worked as it should have. . . Just not for this GTM stanza. . .
... View more
Ok, wanted to update with more info. . . And thanks for suggestions. . .
On first reply from Niketnilay. . .
Actually, here, ALL servers, ALL users default time zone is US/Eastern. . . regardless of where they are, we keep everything set to US/Eastern.
The server producing these records is set to US/Eastern. So the time is really US/Eastern.
The problem is that an application (that we do not own the code for) is dumping its logs and is adding the GMT. even though the time is eastern. We cant seem to get rid of the GMT so I want the time_format to ignore the GMT and just read the time as eastern.
Your prefix/format was one I had tried as well. . . I have tried several. . .
See next update for more current status.
... View more
Here is a sample log record. . .
[Fri, 25 May 2018 17:07:34GMT] [some_named_plugin.dll] [Process:4856][ERROR] : invalid token for user ZYX1\john_xyxxy
Now the time above has GMT beside it, but the actual time is US/Easter time. . .
Well, I take one record and "Add Data" with this 1 line text file just to play with the format and get the stanza to put in the props.conf file.
I create a time_format that works here. But when I put that stanza into the props.conf file, the time comes out 4 hours off. . .
For the one record above, the splunk time SHOULD be 17:07:34. Eastern. like in "Add Data" test record. I just want to ignore brackets, and GMT string.
But it comes out as 13:07:34.
Again, time_format works perfectly in "Add Data". but not in actual props.conf.
[. . . something . . .]
ANNOTATE_PUNCT = false
KV_MODE = auto
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_FORMAT=[%a, %d %b %Y %H:%M:%SGMT]
TIME_PREFIX = ^
TRUNCATE = 999999
pulldown_type = 1
Again, this stanza works fine in "Add Data" single record test. but not with the actual log records. It is 4 hours off.
The server is US/Eastern time. . . Everything on this splunk should be in US/Eastern time. . .
I am testing various time zone offsets to get it correct, but I dont think that is the CORRECT fix.
Why does it work in "Add Data" but not props.conf?
... View more
I used @SloshBurch's and with some mods got it to work.
(thanks somesoni1, but did not try yours)
Agree, don't like joins/appends, but that came closest to describe problem of my several initial attempts. . .
I did have to modify it, so for others that might come across this, I could not use the raw="*string*" . Those wildcards would not work.
SAME structure, but had to use like instead for string comparison.
This search works fine now:
| bucket _time span=1s
| stats count as totalCount, count(eval((like(_raw,"%ERROR%") OR like(_raw,"%string1%") OR like(_raw,"%string2%")) AND NOT like(_raw,"%string3%") AND NOT like(_raw,"%string4%") AND NOT like(_raw,"%string5%") AND NOT like(_raw,"%string6%"))) as totalErrors by _time, host
| eval errorRate=round(totalErrors/totalCount*100,2)
| xyseries _time,host,errorRate
... View more
Other answers I have found don't quite seem to work in my case here. Have seen similar where it can be done based on say "type=" fields and the append/join suggestions don't quite work either.
Hoping someone has a simple solution while I continue to hack/dig for the solution myself. . .
This query LOOKS like what I want as a result. . . total errors / total counts * 100. . . the eval for the error rate does not use the correct correlated bucket error count. It seems to always use the first error count for every bucket percentage. . .
every bucket error rate is 12 / (1 second total count) * 100 because 12 seems to be the First 1 second bucket error count.
each of the join querys works fine by itself to create a nice line/graph. . .
index=prod_stuff source="xyzzy" | bucket _time span=1s
| stats count as totalCount by _time, host
[search index=prod_stuff source="xyzzy" ("FATAL" OR "ERROR" OR "stringa" OR . . . )
NOT ("WARN" OR "string1" OR "string2" OR "string3" OR "string4" OR . . . )
| bucket _time span=1s | stats count AS totalErrors by _time, host ]
| eval errorRate=totalErrors/totalCount*100 | xyseries _time, host, errorRate
This produces a very nice looking graph if I did not care too much about the numbers being correct.
Don't care if this is done using a join, just most efficient way to do this is what I am looking for.
... View more
My background. . . (Heavy Unix, Shell, numerous programming languages. But new to Python and Splunk.)
The intent of this script IS to archive a csv file into a separate directory with a date/time stamp for retention.
Problem is that splunk seems to run twice. First it runs BEFORE "outputcsv" has even started creating the output csv file. Then again, after the file has been created. I can live with it in this script but for future python scripts, This is a problem. I need to understand why my script gets called twice in the following search string.
index=summary | outputcsv myfile | archcsv -c myfile -a temp # Should only run one time at end.
My python search script will look for "myfile.csv" in the /apps/splunk/var/run/splunk and move it to the ../temp folder.
IF there happens to be a myfile.csv in the .../var/run/splunk when the search string STARTS, it will move it FIRST, then the script will be called again when the new myfile.csv has been created.
I know that splunk is NOT unix, but I feel that the "pipe" command should NOT call the archcsv.py script until AFTER outputcsv as finished creating its myfile.csv file.
local commands.conf entry
type = python
filename = pydebug.py
streaming = false
retainsevents = true
UNIX Directory info with Comments:
[splunk]$ ls -ltr
[splunk]$ ls -altr /apps/splunk/var/run/splunk/csvstuff*
-rw------- 1 splunk users 12734095 Aug 4 13:08 /apps/splunk/var/run/splunk/csvstuff.csv
[splunk]$ # Now I will run the search, outputcsv and archive utility.
[splunk]$ # For some reason, it will copy the Existing csvstuff.csv and then the new one.
[splunk]$ ls -altr
drwxr-xr-x 3 splunk users 4096 Jul 31 15:58 ..
-rw-r--r-- 1 splunk users 12734095 Aug 4 13:08 csvstuff_20140804131017.csv
-rw-r--r-- 1 splunk users 10392108 Aug 4 13:10 csvstuff_20140804131021.csv
drwxr-xr-x 2 splunk users 4096 Aug 4 13:10 .
import sys, getopt, os
results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()
line = ''
archfold = 'subdir'
csvfile = 'default.csv'
options, remainder = getopt.getopt(sys.argv[1:], 'c:a:', ['csvfile=',
for opt, arg in options:
if opt in ('-c', '--csvfile'):
csvfile = arg
elif opt in ('-a', '--archfold'):
archfold = arg
adir='/apps/links/' + archfold + '/'
sfile=sdir + csvfile + '.csv'
afile=adir + csvfile + '_ date +"%Y%m%d%H%M%S" .csv'
if carg == 0 or aarg == 0:
move='mv ' + sfile + ' ' + afile
line='if [ -e ' + sfile + ' ]; then ' + move + '; fi'
line='chmod 644 ' + afile
newresults = 
oldresult = None
for result in results:
if result != oldresult:
oldresult = result
if name == "main":
[splunk]$ # now, notice the first file above is from BEFORE I ran the search command
... View more