About Lowell

Lowell · ‎10-31-2014

Check out new Windows Event Log capabilities in Splunk 6.1: http://blogs.splunk.com/2014/05/23/controlling-4662-messages-in-the-windows-security-event-log/

Lowell · ‎10-04-2014

I've run into similar issues, though not with the "splunk" group directly. I assigned a "splunklog" group I used for logs, of which, the "splunk" user is a member (along with a few other admins). The problems seems to within the user switching approach within splunk binary, the groups don't get taken into account. Same thing happens with ulimits. If you launch splunk directly (by using sudo/su) then all works as expected; but not so if you launch as root. I too reverted back to the old-style init.d scripts to avoid this issue. Oh, and I've also randomly seen that files created by splunk (indexes, config files, ...) sometimes have the group of root assigned to them. Weird.

Lowell · ‎08-07-2014

So you can't directly call your script with an argument the way you described above, but you can get pretty close with a simple wrapper script. Here is something you could get started with: run_my_script_per_host.py: import gzip import csv from subprocess import call def openany(p): if p.endswith(".gz"): return gzip.open(p) else: return open(p) results_file = sys.argv[8] # file with search results for row in csv.DictReader(openany(results_file)): # Build a command line to call based on fields from splunk output my_command = [ "myscript", row["host"], ] call(my_command) This script will execute myscript <HOST> for every result returned by your search.

Lowell · ‎08-07-2014

Great question. So roles are not assigned to indexes, each role can have a list of indexes that it's allowed to access and a list of indexes to search by default. So you'll have to add your new index to a role (not the other way around). The endpoint for doing this will be in the following form: https://<splunk_server>:8089/servicesNS/<user>/<app>/authorization/roles/<role>; Specifically take note of srchIndexesAllowed and srchIndexesDefault . You'll probably want to do this in two steps. First GET the current values for these two attributes, put them in a temporary variable, add your new index to the list, and the update the value in Splunk via a POST. Otherwise you may remove existing indexes from your roles, which would be bad. Testing this in a safe environment first is recommended. 😉

Lowell · ‎07-10-2014

Has anyone come across any good references or resource material explaining lispy ? This is visible from the search inspector and can give you some good insights into how Splunk is executing your core search, but I've not been able to find any docs, videos, or blogs that actually explain it. So far, the best references I have are answers on this site: http://answers.splunk.com/answers/30296/search-mechanics-understanding http://answers.splunk.com/answers/120767/efficient-searches-using-boolean-operators

Lowell · ‎07-10-2014

Does anyone know where I can access the "Segmentation and Index size" video by Stephen Sorkin? I know it was a bit old when I watched it like 4+ years ago, but I thought a lot of the core concepts and ideas were still relevant. I felt like it did a good job explaining some of the details of what's going on under the covers. I was hoping to review it again, but haven't been able to locate. Here's a broken link to where the video used to live: http://www.splunk.com/article/2544

Lowell · ‎07-09-2014

These are all Amazon Web Service terms. See the AWS Glossary for more details.

Lowell · ‎07-04-2014

jjensenyahoo, I assume your log files ended in with .gz , is that correct?

Lowell · ‎07-04-2014

I think the invalid_clause is set right. I even tried just reproducing the default process-gzip stuff and can't make that work. Posted as separate question here: http://answers.splunk.com/answers/143771/whats-the-trick-to-get-unarchive_cmd-to-work-for-a-custom-archive-format

Lowell · ‎07-04-2014

I've been attempting to get the unarchive_cmd to work with no success. I decided to abandon my ultimate goal for the purpose of troubleshooting and what I'm finding is that I can't get even the most simple use-case to work. My simplified use case is simple adding support for an "imaginary" gzip-2 format. (Which is really just gzip, with ".gz2" as the extension instead of ".gz"). So I copied Splunk's default props.conf settings for handling "gzip" files and just basically add a "2" to the end of everything. My configuratoin looks like so: props.conf: [preprocess-gzip2] invalid_cause = archive is_valid = False LEARN_MODEL = false [source::....gz2(.\d+)?] unarchive_cmd = gzip -cd - sourcetype = preprocess-gzip2 NO_BINARY_CHECK = true inputs.conf: [monitor:///tmp/indexme] I've tried adding unarchive_cmd to the [preprocess-gzip2] with no luck. I also added priority = 90 to [source::....gz2(.\d+)?] . Again no luck. I also tried adding sourcetype=preprocess-gzip2 to the monitor stanza -- no change. The behavior I'm seeing is this: Any normal gzip file ending with .gz is indexed successfully. Any gzip file ending with .gz2 fails to be indexed at all. No meaningful error (or debug) messages explaining why. I cranked up a few log sources for debugging purposes. Perhaps someone can shed some light on what I'm doing wrong. Again, my long-term goal is to implement some pre-processing command other than gzip -cd - but I'm just trying to boil this down to the most simple case to get things started. Splunkd log - First attempt - Loading a .gz2 file [FAILS] 07-04-2014 15:34:21.631 +0000 INFO TailingProcessor - Archive file='/tmp/indexme/test1-custom-handler-gz2.gz2' updated less than 10000ms ago, will not read it until it stops changing. 07-04-2014 15:34:31.632 +0000 INFO TailingProcessor - Archive file='/tmp/indexme/test1-custom-handler-gz2.gz2' has stopped changing, will read it now. 07-04-2014 15:34:31.632 +0000 INFO ArchiveProcessor - handling file=/tmp/indexme/test1-custom-handler-gz2.gz2 07-04-2014 15:34:31.633 +0000 INFO ArchiveProcessor - reading path=/tmp/indexme/test1-custom-handler-gz2.gz2 (seek=0 len=1337) 07-04-2014 15:34:31.633 +0000 DEBUG ArchiveContext - /tmp/indexme/test1-custom-handler-gz2.gz2 is NOT an archive file. 07-04-2014 15:34:31.719 +0000 DEBUG ArchiveProcessor - Attempting to load indexed extractions config from conf=/tmp/indexme/test1-custom-handler-gz2.gz2||preprocess-gzip2| ... 07-04-2014 15:34:31.720 +0000 DEBUG ArchiveContext - /tmp/indexme/test1-custom-handler-gz2.gz2 is NOT an archive file. 07-04-2014 15:34:31.729 +0000 DEBUG ArchiveProcessor - Basing small file initCrc on path=/tmp/indexme/test1-custom-handler-gz2. 07-04-2014 15:34:31.729 +0000 DEBUG ArchiveProcessor - This archive stream's initcrc=0x2e6fa8b616493a17. 07-04-2014 15:34:31.729 +0000 DEBUG ArchiveProcessor - Found no initcrc match for this stream, will re-read entire file. 07-04-2014 15:34:31.729 +0000 DEBUG ArchiveContext - Early termination because write has false notification 07-04-2014 15:34:31.729 +0000 DEBUG ArchiveContext - /tmp/indexme/test1-custom-handler-gz2.gz2 is NOT an archive file. 07-04-2014 15:34:31.747 +0000 DEBUG ArchiveProcessor - fname=/tmp/indexme/test1-custom-handler-gz2.gz2 fishstate=key=0xb5635587184bcff4 sptr=1337 scrc=0x65e7d9f9bdd1b805 fnamecrc=0xba14100c169eff7e modtime=0 07-04-2014 15:34:31.747 +0000 INFO ArchiveProcessor - Finished processing file '/tmp/indexme/test1-custom-handler-gz2.gz2', removing from stats Splunkd log - Second attempt - Loading a typical .gz file [WORKS] 07-04-2014 15:36:18.722 +0000 INFO TailingProcessor - Archive file='/tmp/indexme/test2-standard-gz-file.gz' updated less than 10000ms ago, will not read it until it stops changing. 07-04-2014 15:36:28.724 +0000 INFO TailingProcessor - Archive file='/tmp/indexme/test2-standard-gz-file.gz' has stopped changing, will read it now. 07-04-2014 15:36:28.725 +0000 INFO ArchiveProcessor - handling file=/tmp/indexme/test2-standard-gz-file.gz 07-04-2014 15:36:28.725 +0000 INFO ArchiveProcessor - reading path=/tmp/indexme/test2-standard-gz-file.gz (seek=0 len=1344) 07-04-2014 15:36:28.725 +0000 DEBUG ArchiveContext - /tmp/indexme/test2-standard-gz-file.gz is NOT an archive file. 07-04-2014 15:36:28.858 +0000 DEBUG ArchiveProcessor - Attempting to load indexed extractions config from conf=/tmp/indexme/test2-standard-gz-file.gz||test2-standard-gz-file-too_small| ... 07-04-2014 15:36:28.858 +0000 DEBUG ArchiveContext - /tmp/indexme/test2-standard-gz-file.gz is NOT an archive file. 07-04-2014 15:36:28.866 +0000 DEBUG ArchiveProcessor - Basing small file initCrc on path=/tmp/indexme/test2-standard-gz-file. 07-04-2014 15:36:28.866 +0000 DEBUG ArchiveProcessor - This archive stream's initcrc=0xcea312efbb705310. 07-04-2014 15:36:28.866 +0000 DEBUG ArchiveProcessor - Found no initcrc match for this stream, will re-read entire file. 07-04-2014 15:36:28.866 +0000 DEBUG ArchiveContext - Early termination because write has false notification 07-04-2014 15:36:28.867 +0000 DEBUG ArchiveContext - /tmp/indexme/test2-standard-gz-file.gz is NOT an archive file. 07-04-2014 15:36:28.878 +0000 INFO databasePartitionPolicy - idx=main Creating hot bucket=hot_v1_3, given event timestamped=1404488177 07-04-2014 15:36:28.879 +0000 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/defaultdb/db'. Reason='Adding bucket, bid=main~3~BCB6BEA7-CE1C-486C-BA49-07E40BAB59B1' 07-04-2014 15:36:28.880 +0000 DEBUG ArchiveProcessor - fname=/tmp/indexme/test2-standard-gz-file.gz fishstate=key=0x72a9c05c9c36e767 sptr=1344 scrc=0xb4910955c707a2a2 fnamecrc=0x84c51a934e8fdaab modtime=0 07-04-2014 15:36:28.880 +0000 INFO ArchiveProcessor - Finished processing file '/tmp/indexme/test2-standard-gz-file.gz', removing from stats Final thoughts It appears as though the ArchiveContext <FILE> is NOT an archive file is bogus message since it appears for all archive files; the ones that work and the ones that don't. The content of the two test files were created using a simple shell command for testing only. ( echo "`date` This is a test event ... ; dd if=/dev/urandom of=/dev/stdout count=1 | xxd ) | gzip -c - > /tmp/indexme/<SOME_FILE_NAME_ENDING_IN_GZ_OR_GZ2> I'm testing this on Splunk 6.1.1 (splunk-6.1.1-207789.x86_64 rpm) running on CentOS 6.4 using an AWS t1.micro instance. Update: I changed my extension from .gz2 to .xy (randomly chosen) just to make sure I wasn't some hitting the wrong [source:*] stanza. No change. I added the following to my local props.conf just to prove to myself that Splunk is actually calling the unarchive_cmd [source::....(?>!tar.)gz(.\d+)?] unarchive_cmd = gzip -cd - | sed -re 's/secret/HIDDEN/g' And this does work. I also tried this with an invalid command, and a typo in the sed command and all of those issues were logged in splunkd.log. 07-04-2014 16:26:15.102 +0000 ERROR ArchiveContext - From archive='/tmp/indexme/test5-standard-gz-file-with-wacky-archivecommandoverride.gz': /bin/sh: NOT_A_VALID_COMMAND: command not found 07-04-2014 16:27:11.112 +0000 ERROR ArchiveContext - From archive='/tmp/indexme/test5-standard-gz-file-with-wacky-archivecommandoverride.gz': sed: -e expression #1, char 9: unknown command: `*' I feel like this should work without me having arbitrarily add a .gz to the end of my file names.

Lowell · ‎07-03-2014

Wow, that's an interesting approach. Sounds like something I would have dreamt up ;-). So I actually thought about the unarchive_cmd option after posting the question and have played around a bit, but so far with no success. After I cranked up the DEBUG logs I'm finally seeing DEBUG ArchiveContext - /tmp/blah-debug.test.me is NOT an archive file. I get the same error even if the file IS in gzip format, so I'm puzzled. Agreed that the incremental indexing thing could be a problem, but I may be able to work around that for the use case in front of me.

Lowell · ‎07-03-2014

Johnsu, just so you know. You can edit your own answers. That's preferred over posting new ones. Thanks for taking the time to post your corrections!

Lowell · ‎07-03-2014

During the Splunk parsing phase, is there any way to hash portions of the event? I know it's possible to discard or mask (trim) portions of the event using SEDCMD or a transformer, but I don't see any options for hashing. I'm looking for a pure Splunk solution that doesn't require scripted (or modular) inputs. Calling out from Splunk would be acceptable, but I'm unaware of any custom "hooks" in the parsing phase (for performance and stability reasons, I assume). I'm pretty sure I know the answer to this, but figured I'd ask before sending in a feature request. For a bit of background: Personally identifiable information (SSN, credit cards numbers, passwords,...) ends up in log files in clear text. The core issue is often a software development one, but often the Splunk admins have now way to control this. The safest option is to remove the sensitive info, but then you loose visibility. Sometimes keeping a few characters will provide enough detail to compare different events without giving away the entire secret, but of course the risks are: (1) Some of the information is still available in clear text, potentially revealing too much information and (2) since the full value isn't known, it's not possible to accurately compare values. Using a hash function (like MD5 or SHA) the values instead would (1) fully protect the original value from being discovered, and (2) still allows for accurate grouping and/or transaction operations on the sensitive field.

Lowell · ‎07-03-2014

Don't forget to mark your question resolved by selecting the check mark next to one of the answers.

Lowell · ‎07-02-2014

Yeah, sorry missed that comment at the end the first time I read through it. I'm looking closer at the regex now. It looks inefficient, but not sure if it's actually wrong. Without a sample event it's difficult to say for sure. Have you tested it using any tools like Regexbuddy, or Kodos? Oh, keep in mind that the blog post is only relevant for the most recent versions of Splunk.

Lowell · ‎07-02-2014

By Splunk forwarder do you mean "Universal Forwarder"? If so, please note that the Universal forward does not handle data parsing, that's handled by the receiving system, like your Splunk Indexer. If you place this configuration on your indexer and restart it, this filter should take effect for new events as they arrive. This blog post may also be of interest to you: http://blogs.splunk.com/2014/05/23/controlling-4662-messages-in-the-windows-security-event-log/ Update: A possible, more efficient regex. (Test with your actual events before using it) REGEX = (?ms)^[^\r\n]+[\r\n]+LogName=Security[\r\n]+SourceName=[^\r\n]+[\r\n]+EventCode=(5145|4634|4624|5140|4625|4648|4661|4662|4672|4771|4611)[\r\n].*?[\r\n]\s+Security ID:\s+[^\r\n]+\$)[\r\n]

Lowell · ‎07-02-2014

There are a couple minor issues with the regex as writen: It doesn't account for how many times a character repeats. It doesn't look for both a pipe and a non-pipe character (it only looks for non-pipes) The value of fieldname will have a leading space based on the placement of \s I like richalloway's approach, but I'm not sure the [\S\s] is quote what you want. I believe that would be interpreted to be a character range that includes all non-spaces and all spaces (which pretty much includes everything, which could be written as simple " . ") Also, this should be anchored to the beginning of the line ( ^ ). Here's my suggestion: (Modified from richalloway's answer) ^(?:[^|]+\|){4}\s*(?<fieldname>\d+) Basically this means, from the start of the line, look for one or more character that's not a pipe, followed by a single pipe. (Repeat 4 times; thus putting us into field 5). Skip over any whitespace characters, and capture the following digits into a field named "fieldname". Of course, delimiter based field extractions are also another option using props.conf and transforms.conf.

Lowell · ‎07-02-2014

Looks like you may need to set the time field in the csv file for replay mode to work correctly. Also note that setting timeMultiple to 2 means it will come in twice as slow as they did originally. You could make them come in twice as fast with 0.5. So maybe try this: timeField = _time timeMultiple = 0.5 Alternately, you could try "sample" mode instead of "replay" mode. This will randomly grab lines from your sample file instead of trying to replay them based on the original timing within the sample file. This would give you a more consistent flow of events. It all depends on your use case as to which option is the best. mode = sample interval = 10 count = 3 This would give you 3 events every 10 seconds. Final thought. May sure you are looking at the log messages created by eventgen when it first starts up. If there's a configuration issue it could be logged.

Lowell · ‎06-12-2014

Can anyone explain the ulimit (max open files) behavior of Splunk 6.1? I've tried the traditional approach (adding entries in limits.conf for the splunk user) and it seems to have no effect if I run splunk from the service script ( /etc/init.d/splunk ) as root. The max open files ulimit is always reported as 4096. If I log in as the "splunk" user manually, and run ulimit -n I see the value I set, 10240. I know that in Splunk 6.1, splunk now handles switching from root to the designated non-privileged user internally. (The user is specified in SPLUNK_OS_USER in splunk-launch.conf , and no longer in /etc/init.d/splunk .) So I thought perhaps I'd need to increase the limits for root, so I did that as well. With no luck. Splunk still shows 4096 as the max open file limit on startup. If I switch to the splunk user interactively, and run "splunk start" manually. Then the expected, 10240 ulimit values is shown in splunkd.log. Of course starting Splunk manually isn't a great long-term option as the change doesn't persist across a simple reboot. (At startup, /etc/init.d/splunk start will be run as the root user, and not as the "splunk" user. And the ulimit will once again be set to 4096.) Any ideas?

Lowell · ‎05-29-2014

Hey David! Here's a proof-of-concept to determine which buckets are accessed for each search. The approach is rather low-level, Linux specific, and has several rough edges. (Aka, don't try this on production). Disclaimers aside, I think it may give you the kind of answers you are looking for, or at least provide a jumping-off point. In Splunk 6 (and possibly earlier), there's a settings called search_process_mode which allows "debugging" all search commands. Essentially, this allows you to run an arbitrary command instead of the normal splunkd search ... process. So basically we can drop in a debugging script to intercept the new search request and then run the standard worker process. This proof of concept uses the strace command to tracks file activity, which includes every open() call to each bucket. The file activity gets dumped to a strace.log file thrown into the dispatch folder. Here's the basic setup: inputs.conf: [monitor://$SPLUNK_HOME/var/run/splunk/dispatch] index = _internal whitelist = strace\.log$ sourcetype = splunk_search_strace limits.conf: [search] search_process_mode = debug search-perf.sh $SPLUNK_HOME/bin/scripts/search-perf.sh #!/bin/bash shift search_id=`echo "$*" | sed -re 's/.* --id=([^ ]+).*/\1/'` echo "`date` [${search_id}] START SEARCH ARGS: $*" >> $SPLUNK_HOME/var/log/splunk/search-perf.log strace -ttt -f -e trace=file $SPLUNK_HOME/bin/splunkd $* 2> $SPLUNK_HOME/var/run/splunk/dispatch/${search_id}/strace.log echo "`date` [${search_id}] END SEARCH" >> $SPLUNK_HOME/var/log/splunk/search-perf.log Here's an example search that shows a list of indexes and buckets accessed per search: index=_internal sourcetype=splunk_search_strace open (merged_lexicon OR .tsidx OR Hosts.data OR Sources.data OR Sourcetypes.data) | rex field=source "/dispatch/(?<sid>[^/]+)/" | rex "open\(\"(?<path>[^\"]+)\"," | rex field=path "/(?<idx>[^/]+)/(?<phase>[^/]+)/(?<bkt>[^/]+)/[^/]+$" | transaction sid | table sid, path, duration, idx, phase, bkt Right now you can use this to see what buckets are being accessed, but I think you could expand the example search to see which buckets are scanned (in other words, add some logic looking for open calls to 'rawdata/journal.gz' rather than just open calls to tsidx files; I'm guessing you could also look at the effectiveness of bloom filters using this technique too). I'm sure there's a way to get statistics on IO usage if you (or someone you know) has voodoo strace skills to determine times take per read and amount of data accessed. Other random notes I found out that you can just run strace without redirecting stderr to a log file and it shows up in the splunkd.log. (Search for "ProcessDispatchedSearch - PROCESS_SEARCH") I thought about using this approach, but then you can't tell which search the message came from. Additional, using the dispatch folder has the advantage of using Splunk's default job aging behavior to cleanup work for us. Also, I saw in the docs a new setting in Splunk 6.1 that looks promising, but I couldn't locate where these new metrics go. But I could have been looking in the wrong location. limits.conf: [search_metrics] debug_metrics = true This may be worth a second look.

Lowell · ‎05-29-2014

New command is splunk list licenses

Lowell · ‎05-20-2014

For whatever it's worth, in 6.1.1 the same limitation still exists. I was hoping that because 6.1 now starts as root and switches to a named (non-privileged) user during startup that this limitation may have been lifted, but testing shows otherwise.

Lowell · ‎05-01-2014

In a clustered configuration, is it possible to set different replication factors per index? This could come in handy for reducing the replication value for extremely large indexes or increasing the replication factor on super important indexes. It seems like the docs are bit contradictory: The Index.conf docs state: repFactor = <nonnegative integer>|auto * Only relevant if this instance is a clustering slave (but see note about "auto" below). * See server.conf spec for details on clustering configuration. * Value of 0 turns off replication for this index. * If set to "auto", slave will use whatever value the master is configured with * Highest legal value is 4294967295 However, Configure the peer indexes page says: The indexes.conf repFactor attribute When you add a new index stanza, you must set the repFactor attribute to auto. This causes the index's data to be replicated to other peers in the cluster. Is there some use for repFactor other than setting it to 0 or auto ?

Lowell · ‎04-22-2014

nextgen, I've updated the example searches above to resolve the regex issue.

Lowell · ‎04-18-2014

Here's an question showing a few different ways to add a timestamp to an output file: http://answers.splunk.com/answers/39974/variable-file-name-in-outputcsv

Posts	1343
Solutions	217
Karma Given	1014
Karma Received	1581
Member Since	‎03-26-2010

Online Status	Offline
Date Last Visited	‎05-26-2021 05:52 PM

Is passAuth still supported for scripted Inputs?

Does inputcsv honor dispatch=true?

Can a search detect it's own sample ratio?

How do you update lookups on a SHC while running S...

How do create indexed fields in a summary index?

What is a managed app in Splunk Enterprise Securit...

How to use a custom token filter within Simple XML

Can you use appendpipe and map together?

Is there a way to pick a point in time (not a time...

How do I run a programmatic search against a Searc...

Re: How to filter Windows event logs from a Univer...

Re: splunk enable boot-start -user in Splunk 6.1.1

Re: How to run an alert script on field values gen...

Re: Assign the correct role to the index created u...

Are there any good lispy docs out there?

Anyone know if the "Segmentation and Index size" v...

Re: Using AWS HTTPS ELB with EC2 Splunk Web on HTT...

Re: python script as unarchive_cmd in props.conf?

Re: Hashing instead of masking at index time

What's the trick to get unarchive_cmd to work for ...

Re: Hashing instead of masking at index time

Re: Masking SSN info into splunk at index time

Hashing instead of masking at index time

Re: Regex Field extraction question

Re: How to filter Windows Security Event Logs cont...

Re: How to filter Windows Security Event Logs cont...

Re: Regex Field extraction question

Re: Eventgen app is not collecting the data in rea...

How do you change the ulimits with Splunk 6.1 when...

Re: How to Know What Buckets Are Searched (hot/war...

Re: Is there a CLI command that will return licens...

Re: how to listen to port UDP 514 when splunk is n...

Can the replication factor be set per index?

Re: UDP Drops on Linux

Re: Exporting search results automatically

Join the Conversation