Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About smwilli1
smwilli1
Explorer
Member since:
08-27-2014
06-05-2020
Community Statistics
| Posts | 8 |
| Solutions | 0 |
| Karma Given | 1 |
| Karma Received | 1 |
| Member Since | 08-27-2014 |
Activity Feed
- Karma Re: How to handle a session split up between multiple events without using transaction or subsearch? for Ayn. 06-05-2020 12:47 AM
- Got Karma for How to handle a session split up between multiple events without using transaction or subsearch?. 06-05-2020 12:47 AM
- Posted Re: Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-15-2014 10:30 AM
- Posted Re: Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-12-2014 02:19 PM
- Posted Re: Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-12-2014 01:46 PM
- Posted Re: Is there an alternative search approach to using a transaction with maxspan? on Splunk Search. 09-11-2014 12:25 PM
- Posted Is there an alternative search approach to using a transaction with maxspan? on Splunk Search. 09-11-2014 12:04 PM
- Tagged Is there an alternative search approach to using a transaction with maxspan? on Splunk Search. 09-11-2014 12:04 PM
- Tagged Is there an alternative search approach to using a transaction with maxspan? on Splunk Search. 09-11-2014 12:04 PM
- Tagged Is there an alternative search approach to using a transaction with maxspan? on Splunk Search. 09-11-2014 12:04 PM
- Posted Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-11-2014 11:35 AM
- Tagged Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-11-2014 11:35 AM
- Tagged Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-11-2014 11:35 AM
- Tagged Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-11-2014 11:35 AM
- Tagged Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-11-2014 11:35 AM
- Tagged Configure separate log entries to be associated by a common field, such as a session_id on Splunk Search. 09-11-2014 11:35 AM
- Posted Re: How to handle a session split up between multiple events without using transaction or subsearch? on Splunk Search. 08-28-2014 08:26 AM
- Posted How to handle a session split up between multiple events without using transaction or subsearch? on Splunk Search. 08-27-2014 06:03 PM
- Tagged How to handle a session split up between multiple events without using transaction or subsearch? on Splunk Search. 08-27-2014 06:03 PM
- Tagged How to handle a session split up between multiple events without using transaction or subsearch? on Splunk Search. 08-27-2014 06:03 PM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 1 |
09-15-2014
10:30 AM
Can you expand on how to build a "growing lookup" or point to some material that may be able to instruct me on how to do this?
... View more
09-12-2014
02:19 PM
The problem with transaction is the same as the issue I was having with eventstats. Unless my search window is large enough to include events from the beginning of the session, it wont capture certain fields I need to be able to reference(such as user & external_ip, at session initiation). Since I am searching over a large dataset, I need to limit the window I search in, otherwise the search times will be very slow. Also, in my environment there is no limit to how long a session can be open.
... View more
09-12-2014
01:46 PM
I'm thinking this might could be accomplished with a LookupTable that I write certain fields to as I see events come in. All ive found on that is outputlookup, and I dont know how to specify certain fields when writing to a table.
Just a thought, look forward to hearing other opinions
... View more
09-11-2014
12:25 PM
Well, I have had discussions with Splunk personnel and they always say stay away from transaction if possible. I know certain situations where I was previously using transaction, and was able to accomplish the same with stats and significantly improve search time. From what I know, the transaction command is built in a way that innately runs slow, and if you can accomplish the same with other commands it typically is faster.
The only thing preventing me from using another command so far is that the "maxspan" argument on |transaction. I will write out some other use cases and post later.
... View more
09-11-2014
12:04 PM
I'm curious if there is a way to get the same effect of transaction w/maxspan, without having to use that process intensive transaction function. This searches by what I refer to as a "rolling time bucket", which allows there to be multiple results in my overall search window, for what I am grouping by.
For example, lets say I am searching for 5 minute windows where the number of authentication failures for any user is greater than 25. With transaction, the search would look like the following:
"Authentication Failed" |transaction user maxspan=5m |where eventcount>=25
I know there is a way to "bin" the data with stats, but this doesn't give the same effect. The logs at the end of each bin would not be associated with the logs at the beginning of the next, which could miss alerts. Another problem I have with stats, is that whenever running stats sorted by user, it groups over the whole duration of my search window. This is not what I want, since I need the ability to catch multiple groups of single user failures over my main search window.
Does anyone have a suggestion on how to approach this? This would help out a lot, since I have very large data sets that I am running similar queries on, which turn out to be quite inefficient due to the use of the transaction command.
Thanks in advance for any help!
Edit: Here are some use cases where I use transaction with maxspan
1) Find a combination events where a user swipes a badge in one location, and within 8 hours attempts to authenticate from a location over 500 miles away (pulled from ip geolocation).
Total Search Window: 24 hours
transaction maxspan window: 8 hours
2) Find a group of 20 or more failed authentication attempts followed by a Successful authentication, all within 30 minutes of each other.
Total Search Window: 24 hours
transaction maxspan window: 30 minutes
3) Find a group of 3 or more users attempt to connect on the same host within a 1 hour window.
Total Search Window: 24 hours
transaction maxspan window: 1 hour
I hope that gives more insight into what I am trying to do with the |transaction maxspan command. Thanks again for all the help.
... View more
09-11-2014
11:35 AM
I have logs that come in the following format:
Sep 1 2014 12:00:00 UTC [13defc34] Client connected on IP 193.18.20.15
Sep 1 2014 12:00:30 UTC [ac21bf43] Client connected on IP 162.74.10.24
Sep 1 2014 12:01:15 UTC [13defc34] username 'johnsmith'
Sep 1 2014 12:01:30 UTC [13defc34] Authentication Failed: invalid username/password
Sep 1 2014 12:01:40 UTC [ac21bf43] username 'billsmith'
Sep 1 2014 12:01:55 UTC [ac21bf43] Authentication Succeeded: Assigned internal IP: 10.0.0.100
Sep 1 2014 12:02:35 UTC [13defc34] Session terminated unsuccessfully: bytes_in=1231 bytes_out=2134
...
...
Sep 2 2014 20:30:55 UTC [ac21bf43] Session terminated successfully: bytes_in=12213211 bytes_out=21323334
As you can see, I don't have the necessary details for the session in every line. In the case of user johnsmith, session '13defc34', the first entry shows his external_ip, second shows his username, third is a failed authentication, and lastly an entry identifying the end of the session with bytes statistics. This causes problems for me seeing that I have to transaction or eventstats every time i want to look for a successful authentication, ended session, authentication fails, etc.
I have preferred eventstats thus far, but i am starting to notice a significant flaw when using that. For instance, lets say I have a 24 hour search looking for bytes data, by using eventstats to associate the username and external IP address from earlier events. Well, if run this soon after billsmith's session is terminated, I won't be able to catch this result. Why? because his authentication happened the previous day, outside the 24 hour search window, and will not be able to associate his username or external IP.
My question is this: Is there a way in Splunk to associate main session details (user, externalIP, internalIP, etc.) with any subsequent event that comes in after that with that same sessionID? I'm fairly certain at this point that it cannot be done at search time for the same reason eventstats will not work for me.(Again, since some of the logs with these details could be outside of my search window).
I've looked into summary indexing, data models, and tscollect/tstats(Still not exactly clear how this works), and I cannot figure out a way to handle this data.
Bonus!: It would also be very helpful to add a field to every log of a session, identifying the status of the session at that instance. For example, once we have session details identifying the user and externalIP, the status would be set to "Pre-Authentication". Once splunk sees a log containing an internalIP, this status would change to something such as "Active" or "In-Progress". And this would end with a status of "Terminated Successfully" or "Terminated Unsuccessfully" at the last line identifying the bytes details.
Thanks in advance for any help!
... View more
08-28-2014
08:26 AM
Awesome! That works perfectly for what i needed. Who knew it would be that simple, haha
... View more
08-27-2014
06:03 PM
1 Karma
One of my VPN log sources is indexed all in different events, correlated by a session_id. This is making things very difficult for me when I am looking for specific events in the session. Consider every time I need a field other than what is in that specific log entry, I have to |transaction on the session to grab a field such as an IP or Username from that another event in that same session.
For example,
Lets say I want to alert on a user who has 20 failed VPN login attempts within a 10 minute period. Seems pretty easy? Well, this is pretty easy as long as I only want the timestamps, username, and the number of fails. I simply do the following:
sourcetype="VPN" "Authentication Fail" |transaction user maxspan=10m |where eventcount>=20 |eval numFails=eventcount |eval FirstFailure=_time |eval LastFailure=_time+duration |table user, FirstFailure, LastFailure, numFails
So thats not too bad. But now lets say I want the IPs associated with these failures, this gets a lot more complicated. The only way I have found to do this is the following:
sourcetype="VPN" "Authentication Fail" |transaction user maxspan=10m |where eventcount>=20 |eval numFails=eventcount |eval FirstFailure=_time |eval LastFailure=_time+duration |streamstats count as id |stats values(user) as user, values(FirstFailure) as FirstFailure, values(LastFailure) as LastFailure, values(numFails) as numFails, values(session_id) as session_id by id|append [search sourcetype="VPN" |stats values(user) as user values(external_ip) as external_ip by session_id |search user=* external_ip=*] |stats values(session_id) as session_id values(FirstFailure) as FirstFailure values(LastFailure) as LastFailure list(external_ip) as external_ip by user
As you can see, I have to subsearch for the VPN logs and stats on the session_id to pull back a external ip to the search, then stats again on the results to associate the IPs with the sessions I got in the main search. I also realize I could be missing a much easier way of doing this, since I am fairly new to these advanced searches in Splunk.
I am hoping to find suggestions on how to better deal with the sessions so i don't have to have a sub-search anytime i need a field from another log in the session. When getting into more complicated searches and correlations, this slows the search down a significant amount as well.
I have provided a sample of the logs below for everyone to take a look at. Thanks in advance!
Aug 26 18:21:35 (session_id): Received User-Agent header:
Aug 26 18:21:35 (session_id): New session from client IP 111.111.111.111
Aug 26 18:22:05 (session_id): Username 'user'
Aug 26 18:22:05 (session_id): SECURID module: authentication with 'user' failed: Authentication failed.
Aug 26 18:22:36 (session_id): Username 'user'
Aug 26 18:22:36 (session_id): Retry Username 'user'
Aug 26 18:22:38 (session_id): Access policy result: Full
Aug 26 18:22:41 (session_id): Assigned PPP IPv4: 101.101.101.101
... View more
