Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About gesman
gesman
Communicator
Member since:
09-24-2014
06-05-2020
Community Statistics
| Posts | 68 |
| Solutions | 5 |
| Karma Given | 9 |
| Karma Received | 18 |
| Member Since | 09-24-2014 |
Activity Feed
- Karma Re: iplocation draws blank for joshd. 06-05-2020 12:47 AM
- Karma Re: Is it possible to configure the memory allocation for a Splunk forwarder to prevent abrupt termination of the splunkd process? for MuS. 06-05-2020 12:47 AM
- Karma Re: Why does increasing the value of maxopentxn reduce the number of returned transaction events? for martin_mueller. 06-05-2020 12:47 AM
- Karma Re: Splunk App for Stream: Is there a way to diagnose if a network interface is working as expected or pushing its limits and dropping packets? for mdickey_splunk. 06-05-2020 12:47 AM
- Karma Re: Throttling out repeated events for woodcock. 06-05-2020 12:47 AM
- Karma Re: How to force Splunk to reindex renamed log files? for acharlieh. 06-05-2020 12:47 AM
- Karma Re: Why does my subsearch maxtime setting in limits.conf have no effect? for martin_mueller. 06-05-2020 12:47 AM
- Karma Re: What is the most efficient approach to create a new field from the last portion of the source field's value? for lguinn2. 06-05-2020 12:47 AM
- Karma Re: What is the most efficient approach to create a new field from the last portion of the source field's value? for martin_mueller. 06-05-2020 12:47 AM
- Got Karma for Why does Splunk UI search hang or timeout with ResourceNotFound HTTP 404 error?. 06-05-2020 12:47 AM
- Got Karma for Re: What is the most efficient approach to create a new field from the last portion of the source field's value?. 06-05-2020 12:47 AM
- Got Karma for Re: What is the most efficient approach to create a new field from the last portion of the source field's value?. 06-05-2020 12:47 AM
- Got Karma for How to force Splunk to reindex renamed log files?. 06-05-2020 12:47 AM
- Got Karma for How to generate a summary index containing only unique values when running hourly searches?. 06-05-2020 12:47 AM
- Got Karma for Re: Why does Splunk UI search hang or timeout with ResourceNotFound HTTP 404 error?. 06-05-2020 12:47 AM
- Got Karma for Splunk App for Stream: Is there a way to diagnose if a network interface is working as expected or pushing its limits and dropping packets?. 06-05-2020 12:47 AM
- Got Karma for Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?. 06-05-2020 12:47 AM
- Got Karma for How do I prevent the collect command from flattening multivalue fields when writing into a summary index?. 06-05-2020 12:47 AM
- Got Karma for How do I prevent the collect command from flattening multivalue fields when writing into a summary index?. 06-05-2020 12:47 AM
- Got Karma for How do I prevent the collect command from flattening multivalue fields when writing into a summary index?. 06-05-2020 12:47 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
07-12-2015
05:25 PM
Thank you.
I think my specific issue is that forwarder actually watching directory tree with 160,000+ files in it.
Although 99% of these files are very rare being modified - not sure of that helps or not.
... View more
07-12-2015
01:27 PM
We have a situation where a Splunk forwarder is abruptly dying on one of the servers once a day or so.
Upon further investigation this was discovered:
root@intelsat [/var/log]# cat messages | grep splunkd
Jul 12 06:25:01 intelsat kernel: [30182] 0 30182 706988 480205 1 0 0 splunkd
Jul 12 06:25:01 intelsat kernel: [30183] 0 30183 13200 92 1 -17 -1000 splunkd
Jul 12 06:25:01 intelsat kernel: Out of memory: Kill process 30182 (splunkd) score 260 or sacrifice child
Jul 12 06:25:01 intelsat kernel: Killed process 30182 (splunkd) total-vm:2827952kB, anon-rss:1920820kB, file-rss:0kB
Jul 12 06:25:01 intelsat kernel: splunkd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Jul 12 06:25:01 intelsat kernel: splunkd cpuset=/ mems_allowed=0
Jul 12 06:25:01 intelsat kernel: Pid: 30201, comm: splunkd Not tainted 3.2.13-grsec-xxxx-grs-ipv6-64 #1
Jul 12 06:25:01 intelsat kernel: [30201] 0 30182 706988 480567 3 0 0 splunkd
Jul 12 06:25:01 intelsat kernel: [30183] 0 30183 13200 92 1 -17 -1000 splunkd
Jul 12 06:25:02 intelsat kernel: [30210] 0 30182 706988 480543 3 0 0 splunkd
Jul 12 06:25:02 intelsat kernel: [30183] 0 30183 13200 92 1 -17 -1000 splunkd
As I understand, it's happening because Splunk forwarder requests more memory than available on the system?
Is it possible to configure forwarder for some form of "safe" memory allocation strategy to prevent this from happening?
Ideally i'd want to configure forwarder to auto-restart as well...
... View more
07-08-2015
07:19 AM
These commands shows that Splunk honors the limits i set in limits.conf. Which means that ...time limit (60 seconds) reached. message is a bug?
Although I did experiment by comparing results of two queries - one using subsearch and another one using hardcoded search using values that subsearch suppose to return:
index=x page=hello [search index=x user=joe| dedup ip | fields ip] | stats c - this returned c=150
with:
index=x user=joe | fields ip | dedup ip | mvcombine ip | eval ip="(ip=" + mvjoin(ip, " OR ip=") + ")" | table ip
- this returned fragment of search query: (ip=1.2.3.4 OR ip=5.6.7.8 OR ip=...)
- So i copy/pasted this fragment and rerun main query like this:
index=x page=hello (ip=1.2.3.4 OR ip=5.6.7.8 OR ip=...) | stats c - this returned c=200
Which means query with subsearch still missed something, even with high limits value set?
... View more
07-07-2015
04:05 PM
I have /my-app/local/limits.conf with the following content:
[subsearch]
maxtime = 600
[join]
subsearch_maxtime = 600
subsearch_timeout = 800
Yet when search finished - job inspector still claims that:
[subsearch]: Search auto-finalized after time limit (60 seconds) reached.
Does this means the setting is ignored, or does this mean that this message is actually incorrect?
... View more
06-26-2015
11:58 AM
inotify does not do recursive monitoring and also - i want to avoid adding too many moving parts outside of Splunk.
... View more
06-26-2015
11:44 AM
Yes, it does exactly what I need.
And yes, I already using _indextime to do necessary tasks. Hint: these are not log files with events that I am indexing, hence _indextime is the only time reference that I have and use.
Care needs to be exercised to configure crcSalt before enabling (or populating) this data source - otherwise Splunk would unnecessarily re-index everything.
inotify probably would be a cleaner solution to detect pure act of renaming but I also need to have access to the latest file contents. The drawback of it - it does not monitor recursively inside subfolders. So in my case I made Splunk to do better inotify job.
Renaming back to the same filename is not a problem (for my case) because I'll still have access to the latest content of fileA (even after second or third rename).
... View more
06-26-2015
10:53 AM
1 Karma
We want to monitor situations where a log file gets renamed to a different name within the same directory or moved to another directory (under the same or different filename).
Re-indexing the contents of the renamed log file is the preferred approach - we don't care about duplicate events, but the fact that the log file got renamed is an important event by itself that we need to monitor.
Splunk by default does not index renamed logs with the same content - how to override this behavior?
... View more
06-11-2015
11:22 AM
Close to impossible on Windows, but here's a solution. There is no built-in facility in Splunk to test accessability of the given data input and see if Splunk has access to it this moment or not.
To be able to test if Windows' local system account (on behalf of which Splunk is running) has access to UNC location we need to launch CMD (command prompt) running on behalf of local system account.
Please note this is not the same as Administrator.
To launch CMD on behalf of local system account do this:
Download pstools from here: http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx
Run .\PSTools\PSEXEC -i -s -d CMD (this will create desired command prompt)
Within new CMD window issue: dir \\server123\share4 to test access and visibility
Alternatively you may access share via IP address (if you get warning about duplicate resource or so)
Alternatively - if that share requires username password do this:
net use x: \\server123\share4 PassW0Rd /user:Username
and then test accessability via dir X:\
... View more
06-10-2015
09:09 AM
I agree, but in current setup it is not possible to use UF. The only facility I can manage is share permissions.
... View more
06-10-2015
08:45 AM
We have Splunk on Windows instance that is used to monitor UNC input like \\server123\share4
This worked well until the permissions on share4 were adjusted to tighten security. This caused Splunk to silently lose visibility of files on that share.
We want to re-configure permission to make sure Splunk can see the data, but is there any tool or approach that quickly shows that given Splunk instance can/cannot read data on specified network share?
Alternatively - is it possible to change existing splunk windows installation to run on behalf of different domain/user (instead of local system account)?
... View more
05-21-2015
08:16 AM
This doesn't returns anything:
| stats c | eval ip="107.181.233.178" | iplocation ip allfields=1 | table ip, Country, Region, City
likely due to iplocation's usage of limited geoip DB?
Is there a way (perhaps for an extra fee?) to plug into Splunk more robust and rich IP location and assignment resolution capabilities?
... View more
- Tags:
- iplocation
05-19-2015
01:56 PM
You mean: log.txt -> postprocessed to -> log_processed.txt -> indexing only log_processed.txt files ?
... View more
05-12-2015
04:40 PM
Is there a way to run custom subsearch per each event?
Pseudo: index=logs "error" | foreachevent [search index=extra_data | ... ] | ...
When few web hits within incoming traffic are red flagged - each such hit needs to be correlated with data at other indexes to enrich results with metadata.
Short of writing custom search command or forcing search query to generate another query dynamically - I don't know any more straightforward ways to make it happen.
... View more
- Tags:
- subsearch
05-12-2015
03:51 PM
I'd guess so too, but doc mentions only ./system/:
http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/fieldsconf
Is it documented anywhere regarding /apps/ as a possible location as well?
... View more
05-10-2015
12:37 PM
Yes, I think this is good approach.
It's reasonably flexible and can be applied to more complex throttling logic as well.
Thank you.
... View more
05-09-2015
12:21 PM
I have a hypothetical search that runs every 5 minutes and scans last hour worth of data for certain errors:
index=log "error" earliest=-1h | eventstats c as user_errors by username | where user_errors>5 | dedup username | table ip, username, user_errors .
I want to alert on users who cause more than 5 errors per any hour period.
This search generates approx. 10 alerts each time, where 6-8 users are the same because "sliding" window is an hour wide and is being scanned every 5 minutes.
Ideally i'd want to throttle alerts for the same username .
So I want to receive alerts only on all "unique" users and never receive alerts for the same user more often than once an hour.
The only way I see how to do that is to set "Alerting mode"-"Once per result" and set "Per result throttling fields"=username
What happens after i did that - is that I receive email alert for only 1 user and it misses all the other users. And then i do not receive any more alerts whatsoever till the next hour - and again for one user only.
Any way to fix that?
... View more
- Tags:
- throttle
- throttling
05-03-2015
04:18 PM
How can I return calculated (variable) number of top events?
This doesn't work:
... | eval x=5 | head x
... View more
- Tags:
- head
05-01-2015
01:16 PM
We have data set which aggregated sessions with it's eventcount for each event.
We are looking at setting up an alert for sessions where eventcount exceeded "normalcy".
For Bell-curved data we'd setup an alert for 2x or 3x STDEV. But in our case eventcount is not really Bell-curved - as it starts right away very high at low eventcount and then gradually gets lower in this manner
x
x
xx
xxxx
xxxxxxxxxxxxxxx
Does Splunk has built-in ways to handle deviations for other types of non-Bell curved data sets?
... View more
05-01-2015
12:37 PM
To note: there is actually benefit of having multivalues flattened and separated by some character.
"Flattened" values (say 'usernames') is searchable via index=logs usernames=*johnsmith* | ... query vs. multivalues are not.
So in above case if I'd need to find only events where one of the username is (or contains) 'johnsmith' - that would work nicely and reduce number of events before pipe.
If usernames would be stored in multivalued format - we'd need to use slower logic to either flatten usernames first or use functions like mvfilter to search everything.
... View more
04-30-2015
07:00 AM
I wanted to auto convert data within these logs: field_a="value1|value2|value3", field_b="value_x|value_y" to multivalues at search time.
The perfectly working solution seems to be to add this to ./etc/system/local/fields.conf :
[field_a]
TOKENIZER = ([^\|]+)
[field_b]
TOKENIZER = ([^\|]+)
The only problem is that above approach seems to have global effect across all indexes and all sourcetypes.
Is there a way to limit the scope of stanzas within fields.conf?
I just want it to apply to specific index or to specific sourcetype.
... View more
04-28-2015
07:06 PM
1 Karma
One of the working solution is to add this to ./etc/system/local/fields.conf :
[ips]
TOKENIZER = ([^\|]+)
[uas]
TOKENIZER = ([^\|]+)
[usernames]
TOKENIZER = ([^\|]+)
This would do the same job as this: ... | makemv delim="|" ips | makemv delim="|" uas | makemv delim="|" usernames ...
(doesn't seems to require using of MV_ADD in transforms)
... View more
