About Ayn

Ayn · ‎01-29-2014

The challenge here I guess is that if you filter by "[email protected]" you will get the token, BUT you filter out the other events that have the token but not the login info. A way of solving this would be to run this filter in a subsearch that emits all corresponding tokens for a certain login, and the outer search then grabs all events with these tokens. Finally use transaction for tying sessions with the same token value together. Assuming you have tokens extracted into a field called token : * [search [email protected] | fields token] | transaction token

Ayn · ‎01-29-2014

Using eventstats might be better: index=prd_access sourcetype="access:web:iis:project" | eventstats count as totalcount | chart count,first(totalcount) as totalcount by httpstatus | eval percentage=count/totalcount

Ayn · ‎01-29-2014

http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

Ayn · ‎01-28-2014

It would be interesting to hear more about why the 10000 limit is there in the first place. I've encountered it loads of time by now (using inputlookup for loading blacklists of IP's that should be matched for instance) and it's of course frustrating to be limited like this. The question is if there's a very good reason for this limit to exist and have this specific value, like for instance that past 10000 search terms you won't increase performance with any additional terms?

Ayn · ‎01-28-2014

Without knowing the exact format of the file - if the file is in a binary format, you shouldn't be pointing Splunk to read it directly. Splunk has no magic "decoding" of binary files, it indexes what it gets as-is. All you achieve by setting NO_BINARY_CHECK is to force Splunk to read all that non-readable data even though it doesn't make sense to do so. What you need is something that will decode this data from the binary format to a textual representation of it, and then have Splunk read that output instead. Again, I don't know the exact format of this file that you're referring to, but some kind of flow reader would do this.

Ayn · ‎01-28-2014

The best (and possibly only) way would be to implement this logic in your script. Splunk doesn't have that kind of ability to compare incoming data to what's already in the index. My suggested approach would be for you to edit your script so it keeps the last version of the XML file, and when you issue the next request you compare the data you get from that with what's in the previous version.

Ayn · ‎01-28-2014

I can guarantee you that is not the case. Actually Splunk strongly suggest against index-time field extractions in the ES docs (as well as the docs for the core Splunk product).

Ayn · ‎01-28-2014

Why do you require index-time? It's almost always a bad idea.

Ayn · ‎01-28-2014

Close to 4 hours = almost exactly 4 hours? Might this be some kind of timezone issue?

Ayn · ‎01-27-2014

Are your UF's sending huge amounts of data? There's a default limit for how much bandwidth UF's will use, 256kBps. This can be altered but by default that is the limit and in environments where UF's need to send large volumes of data that can be an issue.

Ayn · ‎01-27-2014

I think you may need to explain more clearly - what are you currently getting stuck on? What do you mean by "track" here?

Ayn · ‎01-27-2014

You can route specific events to other Splunk instances, and/or index them locally on the Splunk instances you're routing the events from. See more here: http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad

Ayn · ‎01-26-2014

Sounds to me you should be reading the docs: http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Aboutforwardingandreceivingdata You need to configure inputs on your forwarder. It won't automatically read and forward any data so you need to tell it what to grab.

Ayn · ‎01-26-2014

As martin_mueller says, it's important to know here how unique the KeyID values are - that is, not only in this specific sourcetype, but across all data in the index. @dwaddle has explained very well the specifics of what goes on in a Splunk search here: http://answers.splunk.com/answers/54207/slow-search-when-evaluating-a-numeric-value?page=1&focusedAnswerId=54224#54224 It's a very good read and I think it answers your question. Short version here: KeyID="1" will be slow because "1" is very likely such a common token in your index, and as most fields aren't extracted until at at search-time, when you search for KeyID="1" Splunk will in practice find all events with the token "1" in them and THEN see if any of these tokens can be matched to the field "KeyID". In this scenario an index-time field extraction might be a good idea in order to improve performance.

Ayn · ‎01-22-2014

Oh come on don't be hurt 🙂 I'm merely stating the problem with the approach. It's still a valid approach but it's important to point out its drawbacks. Between the 3 supplied answers here I believe we've showed what various approaches the user can take. Each of them has its own advantages and disadvantages.

Ayn · ‎01-21-2014

...or if you want to use transaction for some reason, index="wireless" DHCP ACK | transaction src_mac | table src_mac _time src_ip

Ayn · ‎01-21-2014

How about index="wireless" DHCP ACK | stats list(_time) as time,list(src_ip) as src_ip by src_mac | convert ctime(time)

Ayn · ‎01-20-2014

If these are timestamps in your input, aren't they the timestamp that Splunk uses in its own timestamp recognition? I think you might be making this more complicated than it needs to be.

Ayn · ‎01-20-2014

I don't know which fields you have at your disposal, but for instance if the URL is extracted to the field url and the user is in the field user : ... url=yoururl | stats count by user

Ayn · ‎01-20-2014

Use addinfo . See the docs on both collect and addinfo . http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Collect http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Addinfo

Ayn · ‎01-18-2014

It's a good idea to test your regexes using something like for instance regexpal - http://www.regexpal.com/ - or RegExr - http://gskinner.com/RegExr/ . In your case the sample log you show supposedly starts with a timestamp which makes your extraction fail - but maybe the timestamp isn't really part of the log, just something that you included from Splunk's own output?

Ayn · ‎01-17-2014

The problem with this is that it would lead to bad performance, because no filtering on EventCode will be done at all in the base search.

Ayn · ‎01-17-2014

Do you have events newer than 7 days old that should match?

Ayn · ‎01-17-2014

The thing about transaction is that it removes the individual events, so since the concepts of the previously existing individual events is gone it's tricky to do stats "per event". You could either calculate your statistics before running transaction , or you could use eval functions like mvcount for getting counts within multivalued fields that are created as part of the transaction. It's hard to give more specific advice without knowing more about your exact scenario.

Ayn · ‎01-16-2014

No. An OR separated list like the one you've written is the way to go.

Posts	3370
Solutions	730
Karma Given	1392
Karma Received	3245
Member Since	‎02-28-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

Backslashes being doubled

How to show events per second in timechart regardl...

How to have a results table always display newest ...

Using host tags (or similar) when searching on fie...

How can i change the time format in Splunk web?

Re: dynamic lookup based upon dynamic token

Re: how to display percentage of total

Re: Time for Data ( _time )

Re: Attempt to workaround 10k subsearch limit -- h...

Re: Indexing a NetFlow logic - log file.

Re: scripted inputs and duplicate event data

Re: Index time field extraction/re-write

Re: Index time field extraction/re-write

Re: Difference between _time and -indextime

Re: Difference between _time and -indextime

Re: Fields within fields - search time extract

Re: Forward Data

Re: Forwarder - Please Clarify Usage

Re: Unique KeyValue search performance

Re: how can I search for multiple event IDs?

Re: One MAC multiple IP's Table View

Re: One MAC multiple IP's Table View

Re: timestamp conversion

Re: How to generate a report to see how many times...

Re: How to create info_time_min in collect command

Re: Using RegEx in Props.conf

Re: how can I search for multiple event IDs?

Re: Log not index entirely

Re: stats on transaction

Re: how can I search for multiple event IDs?

Join the Conversation