About Lowell

Lowell · ‎04-13-2016

This refers to Splunk prior to 4.2. (Very old) Splunk 6.4 has just introduced new features to allow searching of "frozen" buckets (aka buckets with out .tsidx files)

Lowell · ‎02-19-2016

There's no real easy way to do this. The problem here is that your file is a mix between a traditional "log" file (with timestamps existing between events) and a tabular (CSV) data format. This isn't a typical use, and therefore isn't well supported. A few thoughts/ideas listed below. Options: Modify the log-producing program to format so that there's one timestamp per row (line). Pre-process the CSV file so that the timestamp repeats for each row. This would be relatively simple to do in Python, Perl, AWK, or whatever tool you prefer. Allow Splunk ingest each chunk of tabular data into a single event; then use some search-time magic to split each event into multiple events--one per csv row. I know a wrote a custom search command a few years back that did this exact thing. (I may be able to find it lying around; I never got around to actually releasing it.) But this probably isn't the best answer and may not perform as well, depending on your use case. There could be some other clever approaches to this as well, but most of them will be either a pain to implement, hard to understand, or have a serious limitations. I'd try options 1 or 2 first.

Lowell · ‎02-12-2016

Please try running the script from the command line (shell/terminal) manually and report back what the response is. If it doesn't run correctly from the command line, it's not going to work from Splunk either. I suspect that is the problem, but need more detail. Try something like this: cd /opt/splunk/etc/apps/Splunk_TA_nix/bin/ ./rlog.sh > /dev/null echo "returncode=$?"

Lowell · ‎02-11-2016

I know I'm late to the party, just wanted to throw in one caution. It's interesting that streamstats is actually faster than a join or append in this case. I could see that working for a small amount of data, but I suspect that factors like data set size (of both the primary and secondary sources) as well as search mode (single server vs distributed) could have a significant impact on performance. The overall advice here is great, I just think it's prudent to point out that search commands like transaction , streamstats , and eventstats require that ALL events be streamed to the search head which can have some significant network bandwidth and performance implications. Compare this to stats where map-reduce allows the data to be precomputed on the indexers. Bottom line, don't forget to look a the "remote search" and the "report search" in the job inspector.

Lowell · ‎02-03-2016

I've found regex101.com to be very helpful debugging regexes, and there's a good bit of online help available on the page if you need a refresher on regex syntax. For learning regexes and advanced topics, check out www.regular-expressions.info

Lowell · ‎01-28-2016

So just found this question, and was surprised to see that this almost works: [ | stats count | eval s="index=main | head 1" | return $s ] It's totally wacky to mean that Splunk lets you pass a "|" to the search and execute it like this. I said "almost worked" because I end up with 3 results, which it turns out is because the environment I tested it on had 3 indexers. Changed it to "head 5" and got back 15 results. Interesting. Tested on a 6.2.6 environment.

Lowell · ‎01-28-2016

Hmm, it looks like a simple | append [[]] give the same error, which I suspect is simply because it's nonsensical. In particular, there's no generating SPL command given. So a search like | appendpipe [ search [ search ] ] does "work", but doesn't do anything useful. I agree that there's a subtle difference between the way that a subsearch works and the way the pipeline works. So with the example of | appendpipe [ (#1) search [ (#2) search ] ] , with search #1, that is a post-filtering operation, where as search #2 is a generating command. But I think you still have to have some SPL command in #1, because a subsearch can only be used within certain commands, like search , append , join , and so on. But I'm still not sure why this would limit the use of the map command from within the context of an appendpipeline subsearch/subpipeline.

Lowell · ‎01-28-2016

If you're event is being parsed a JSON, which normally happens automatically, then you can simply use a fieldalias instead of doing a field extraction. On second glance, is your JSON data escaped? Or is that just a side effect of posting it here? Pretty sure Splunk doesn't handle escaped JSON automatically. So you could either unescape the data on ingestion, possibly using SEDCMD in props.conf, or if that's not an option, try a field extraction with a regex like this: "event\\":\\"(?<EventType>[^"\\]+)\\"

Lowell · ‎01-28-2016

So it's interesting to me that the map works properly from an append but not from appendpipe . (This may lend itself to jplumsdaine22 note about subsearch vs pipeline) And yeah, my current workaround is using a bunch of appends and subsearches to get what I need. The key difference here is that the value of "series" isn't know ahead of time, it's determined by an earlier search, so setting the value within the append sub-search isn't really an option for me. So my search ends up looking like this: <base search returning a,b> | append [ sourcetype=c [ <base search returning b> | return b ] | manipulate ... | table c ] | stats values(*) as * by b | table a b c And of course this simplification ignores the fact that if the base search returns nothing, the parent search essentially goes wide open and returns too many results, leading the awkward solution like so: <base search returning a,b> | append [ sourcetype=c [ <base search returning b> | append [ stats count | eval b="NeverOccuringInRealLife" | fields b ] | return 2 b ] | manipulate ... | table c ] | stats values(*) as * by b May need to switch over to handling all this logic externally and use some REST APIs; which is just a bit frustrating because SPL is so close to being able to handle this natively.

Lowell · ‎01-27-2016

I have a massively complex search that's working. But now I'd like to augment the output of that search with some additional fields, which can be found by using a secondary search. For this to be efficient, I need the output of the core search to be fed as parameters of the secondary search..... (Basically, I'm looking for a "lookup", but a lookup that's based of another search not a CSV file, script, or kv-store.) I'm really only dealing with one or two results at a time, so the typical inefficiencies of launching multiple searches is not a concern here. It seems like this should be possible with the appendpipe search command in combination with the map command. Instead of trying to make this work in the context of my already complex search, I broke it down into it's simplest form. This search works, demonstrating the the "map" works as-expected: | stats count | eval series="splunkd" | map search="search index=_internal source=*metrics* group=per_sourcetype_thruput series=$series$ | head 1 | table series kb max_age" The output is: series=splunkd,kb=10.49353 max_age=1 This also works, demonstrating that the field "series" makes it from the base search into the subsearch, just as appendpipe advertises: | stats count | eval series="splunkd" | appendpipe [ eval new_field=series ] The output looks like so: count=0, series=splunkd count=0, new_field=splunkd, series=splunkd However, once combined, something goes (silently) wrong: | stats count | eval series="splunkd" | appendpipe [ map search="search index=_internal source=*metrics* group=per_sourcetype_thruput series=$series$ | head 1 | table kb series max_age" ] The output looks like: count=0, series=splunkd I was expecting the output to look like: count=0, series=splunkd series=splunkd,kb=10.49353 max_age=1 In real life, the first result would have lots of other useful fields. And I'd stick something like | stats values(*) as * by series to group all the relevant fields into a single result. Any thoughts? I've been testing this on Splunk 6.2 Update: Here's a slightly better example query and my current workaround. index=_internal source=*metrics* group=per_sourcetype_thruput | stats sum(ev) as ev by series | sort 1 - ev | appendpipe [ map search="search index=_internal source=*metrics* group=per_sourcetype_thruput series=$series$ | head 1 | table series kb" ] | selfjoin series The work around doesn't use map , because without appendpipe there's no way to "pass in" fields. So instead we have to use a subsearch, which essentially requires repeating the entire base search, which works but isn't very efficient. Here's the search: index=_internal source=*metrics* group=per_sourcetype_thruput | stats sum(ev) as ev by series | sort 1 - ev | append [ search index=_internal source=*metrics* group=per_sourcetype_thruput [ search index=_internal source=*metrics* group=per_sourcetype_thruput | stats sum(ev) as ev by series | sort 1 - ev | return series ] | head 1 | table series kb ] | selfjoin series Update 2: For whatever it's worth, the problem with the map is not field substitution, because this does not work: | stats count | appendpipe [ map search="search index=_internal source=*metrics* group=per_sourcetype_thruput series=splunkd | head 1 | table series kb" ] Weird.

Lowell · ‎01-18-2016

I was hoping that Splunk would have added support for this without requiring a REGEX, but I'm doubtful any such solution exists. My understanding is that dynamic key regex have some negative performance implications. ( $1::$2 )

Lowell · ‎01-15-2016

Hmm. Well there's way too many timestamp combinations to pre-populate a input or drop-down. The link to the conditions stuff may turn out to be helpful, thanks!

Lowell · ‎01-14-2016

I'm looking for a SimpleXML form input that allows the user to pick a point in time rather than a time range (like the default time range picker does)? Example use case: Determining who was given an IP address given DHCP lease events. For this kind of data, the event itself represents a "time range" (start/end times of the IP address lease) and I simply want to know who (which MAC address) was assigned that IP address at a specific point in time. For efficiency, I'll want to use the users' time selection to drive the Splunk search timeframe (probably +/- 6 hours) from the time selected. And then within the search, only keep events where where the time in question (user's input) is between the least start and end time. Any thoughts? I'm open to custom HTML snippets and javascript, but I'd like to avoid full-blown HTML dashboards if possible.

Lowell · ‎01-14-2016

Not supported. You can do HTTPS -> HTTPS with the ELB, as far as I know that's still the best option. Still waiting on SPL-79993. https://answers.splunk.com/answers/103674/using-aws-https-elb-with-ec2-splunk-web-on-http-port-8000.html

Lowell · ‎01-14-2016

Is there anything special about interacting with a Search Head Cluster via the REST APIs? Specifically, what endpoint should you hit? Can splunkd (port 8089) be placed behind a load balancer of some kind? SplunkWeb provides some failover and transparent result proxying to minimize user impact, but it's unclear to me how much of that happens within splunkd and how much of the magic is in SplunkWeb. I assume the easiest approach is just to point the Python SDK to a single SHC member and hope for the best, but I'm guessing there's a better way to do it. Thoughts? Anyone done this?

Lowell · ‎01-14-2016

I feel like I should know the answer to this, but just in case I missed something.... Splunk automatically handles field extractions for events like this very well: Thu Jan 14 10:46:02 EST 2016 myfakeservice[3]: successful login. user="joe" ip="10.0.0.99" This works, but it results in all field values wrapped in literal single quotes: Thu Jan 14 10:46:02 EST 2016 myfakeservice[3]: successful login. user='joe' ip='10.0.0.99' So user is now 'joe' and ip is now '10.0.0.99' (Because the single quotes become part of the field's value.) Is there a way to make this work more efficiently with Splunk's automatic KV mode ( KV_MODE=auto ) so that single quotes are treated as double quotes are traditionally handled? I realize this can be done with a REGEX, but I was hoping for a better solution.

Lowell · ‎01-14-2016

Looks like this can be done using KV_MODE=auto_escaped in props.conf.

Lowell · ‎01-08-2016

Unfortunately, that's not correct. Most of the fields that show up are in-fact indexed fields (except for splunk_server and linecount), but there's way more indexed fields than what is being shown.) For example, date_*, timestartpos, punct, ...) I just double checked on a local 6.2 instance running on my laptop. I indexed some IIS data using the structured data extraction, and confirmed that "c_ip" didn't show up on the left, but if I searched for c_ip::1.1.1.1 it did in fact return matching records. (Tested with tstats too, same result) Fast mode is "fast" because it doesn't bother extracting fields you haven't requested. But Splunk still has to look at both the raw data and the index data to fetch every event. Vs something like tstats which does a pure index-only search never needs to pull in the raw data (and therefore search-time extractions are impossible to perform). Both approaches are faster than the other search modes, but they are very different under the covers.

Lowell · ‎01-08-2016

BTW, have you consider getting your data into the Common Information Model (or CIM)? From there you could enabling acceleration on the Authentication data model and search using tstats or piviot . The results would be much faster, but you'd have to keep the data online for the full 15 months. Alternately, you could setup your own custom accelerated datamodel on your summary index and search that instead. Either options should give you really good performance compared with raw search.

Lowell · ‎01-08-2016

Yup, still waiting on the DELIMS/FIELDS UI thing in 2016. And now with Splunk Cloud that's become an even bigger pain because of the lack of access to the .conf files. ;-(

Lowell · ‎01-08-2016

I'm really wondering if Splunk has changed their stance on this at all since 2010. I base this on two thoughts, (1) structured data extractions introduced in Splunk 6 uses indexed fields automatically (although the risk of "getting it wrong" is much lower in those types of cases), and (2) it's now possible to take full advantage of indexed fields using tstats , so there's significant performance gain possibilities when adding your own indexed fields.

Lowell · ‎01-08-2016

Hey, just found out that Splunk revived their field::value search syntax (which I think was disabled) back in Splunk 5, but I guess made it's way back into Splunk 6.

Lowell · ‎01-08-2016

This may be a good use case for calculated fields. You can setup a calculated field to based on either _time or _indextime (I'll come back to a question there in a second). Take a look at the these docs: http://docs.splunk.com/Documentation/Splunk/6.3.2/Knowledge/definecalcfields Just to be sure, are the builtin timestamps correct for the events? (From your question it sounds like they are, but from a comment earlier, it was't clear.) Getting timestamp recognition correct when the data is onboard is super critical in Splunk. (I'd argue that it's the most important on boarding issue.) This is important if you want consistency in Splunk, start here: http://docs.splunk.com/Documentation/Splunk/6.3.2/Data/Configuretimestamprecognition If you're sourcetypes are setup to use structured data (IIS/JSON) take a look at TIMESTAMP_FIELDS in props.conf: http://docs.splunk.com/Documentation/Splunk/6.3.2/Data/Extractfieldsfromfileswithstructureddata Once you've got _time setup correctly, now Splunk can search on timeframes properly, and then you can make a friendly field using calculated fields: EVAL-timestamp = strftime(_time, "%Y-%m-%d %T") BTW. A field alias only "copies" the field, it doesn't "move" a field. (And you wouldn't want it too, that would break lots of stuff). It's not the same as the rename search command, it's more like eval newfield=oldfield .

Lowell · ‎01-08-2016

Can you provide additional details about your environment? Single server, distributed, are you doing any index clustering/replication? Keep in mind that events aren't truly "deleted" they are simply marked as hidden. So it's possible something is getting messed up in the file that contains the list of deleted events (per bucket). If you are deleting things regularly, you may want to look at why that is, as frequent deletes cause frequent bucket roles, which can impact performance. Sounds like this may be a good candidate for a support ticket, especially if you can reproduce it or show multiple examples of when this has happened.

Lowell · ‎01-07-2016

I just want to point out the that ELK stack can do this! So my answer is: Deploy LogStash Configure it to read in the log Configure the hashing transformation Dump the output to a new log file Ingest the new log file with Splunk (UF)

Posts	1343
Solutions	217
Karma Given	1014
Karma Received	1581
Member Since	‎03-26-2010

Online Status	Offline
Date Last Visited	‎05-26-2021 05:52 PM

Is passAuth still supported for scripted Inputs?

Does inputcsv honor dispatch=true?

Can a search detect it's own sample ratio?

How do you update lookups on a SHC while running S...

How do create indexed fields in a summary index?

What is a managed app in Splunk Enterprise Securit...

How to use a custom token filter within Simple XML

Can you use appendpipe and map together?

Is there a way to pick a point in time (not a time...

How do I run a programmatic search against a Searc...

Re: If compressing tsidx files in frozen archives,...

Re: How to extract data from a CSV file with multi...

Re: Error msg="A script exited abnormally" status=...

Re: How to compare fields over multiple sourcetype...

Re: Regex to extract a number from string

Re: Dynamic Splunk Queries

Re: Can you use appendpipe and map together?

Re: How to extract and create a new field from my ...

Re: Can you use appendpipe and map together?

Can you use appendpipe and map together?

Re: Can you tell Auto KV to honor values within si...

Re: Is there a way to pick a point in time (not a ...

Is there a way to pick a point in time (not a time...

Re: How to configure a Splunk 6.2.3 search head cl...

How do I run a programmatic search against a Searc...

Can you tell Auto KV to honor values within single...

Re: is there a way that autoKV can support both sp...

Re: Is there a way to know which fields were extra...

Re: Would creating fields at index-time improve se...

Re: search-time versus index-time field extraction...

Re: search-time vs index-time field extraction

Re: Is there a way to know which fields were extra...

Re: How to extract the _time value into a separate...

Re: Why can deleted events reappear?

Re: Hashing instead of masking at index time

Join the Conversation