About DEAD_BEEF

DEAD_BEEF · ‎06-03-2020

So when ingest_pipe is not there, it gives it a value of 0, and when it is present, it does nothing?

DEAD_BEEF · ‎06-03-2020

I am trying to make an area chart which shows the average size of the parsing queue over time. I would like to add a horizontal bar as a threshold. I noticed that some logs have different values for the max_size_kb, so I thought I could use max to get the value and set my threshold to that, but for some reason, my search is returning zero results. I don't know why it's not working. If I hardcode a number for zzz, it'll work, but doesn't seem to work the way it is written now. The value changes between my hosts, so I don't want to hard code it. Current SPL index=_internal host=$hostToken$ group=queue name=parsingqueue | stats max(max_size_kb) AS zzz | timechart avg(current_size_kb) by ingest_pipe | eval threshold = zzz

DEAD_BEEF · ‎06-03-2020

I am trying to create a dashboard that graphs the parsing queue size for a HF by ingest_pipe . I noticed that most of these logs have that field but some don't (i'm not sure why). sample logs 06-03-2020 12:21:30.964 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0 06-03-2020 12:21:27.144 -0400 INFO Metrics - group=queue, ingest_pipe=3, name=parsingqueue, max_size_kb=6144, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0 06-03-2020 12:21:27.142 -0400 INFO Metrics - group=queue, ingest_pipe=2, name=parsingqueue, max_size_kb=6144, current_size_kb=0, current_size=0, largest_size=11778, smallest_size=0 current SPL index=_internal host=$hostToken$ group=queue name=parsingqueue | timechart avg(current_size_kb) by ingest_pipe I can't modify the search with ingest_pipe=* because I have tokenized the host field in the search and some of my HF's only have 1 ingest pipe. In that scenario, there is no ingest_pipe field at all so hardcoding that into the search will result in 0 results when the HF only has 1 pipeline. The solution I came up with is to count the # of events where ingest_pipe exists (yesPipe), count the # of events where it does not exist (noPipe), and assign my count by foo value to the field that is greater. If yesPipe is greater, count by ingest_pipe , else count by host . I don't have the query for these counts and checks. Alternatively, I thought I could use a lookup table that has a "count by field" column, where per host I simply specify either ingest_pipe or host to count by. I feel like there is an easy solution and I'm overthinking it. Any ideas?

DEAD_BEEF · ‎05-27-2020

Exactly what I was looking for!

DEAD_BEEF · ‎05-27-2020

I have a table that shows me the username, the web resource they accessed, total number of times they accessed each file (FileCount) and the summation of all web resources they accessed. The problem I am seeing is that when a user accessed say 8+ resources, the results in my table grows very long for that user. In some cases, some users hit over 50 resources. My question is 2 parts: a) Is there a way to TRUNCATE or limit this part of the table? I've seen results show up as TRUNCATED before in a table but don't recall how that was done. I want no more than 5 rows per user with largest counts first. But I still want Total FileCount to be the full number and accurate count #. b) sort the FileCount list? How the table currently looks user Total FileCount Resource FileCount jsmith 5 file1 5 jdoe 30 file1 1 file10 3 file2 2 file3 2 file4 1 file5 7 file6 3 file7 1 file8 9 file9 1 How I WANT the table to look user Total FileCount Resource FileCount jsmith 5 file1 5 jdoe 30 file8 9 file5 7 file10 3 file6 3 file2 2 Current SPL | makeresults count=35 | streamstats count | eval user = case(count=1 OR count=2 OR count=3 OR count=4 OR count=5, "jsmith", count=6 OR count=7 OR count=8 OR count=9 OR count=10 OR count=11 OR count=12 OR count=13 OR count=14 OR count=15 OR count=16 OR count=17 OR count=18 OR count=19 OR count=20 OR count=21 OR count=22 OR count=23 OR count=24 OR count=25 OR count=26 OR count=27 OR count=28 OR count=29 OR count=30 OR count=31 OR count=32 OR count=33 OR count=34 OR count=35, "jdoe") | eval resource = case(count=1 OR count=2 OR count=3 OR count=4 OR count=5 OR count=6, "file1", count=7 OR count=8, "file2", count=9 OR count=10, "file3", count=11, "file4", count=12 OR count=13 OR count=14 OR count=15 OR count=16 OR count=17 OR count=18, "file5", count=19 OR count=20 OR count=21, "file6", count=22, "file7", count=23 OR count=24 OR count=25 OR count=26 OR count=27 OR count=28 OR count=29 OR count=30 OR count=31, "file8", count=32, "file9", count=33 OR count=34 OR count=35, "file10") | stats count AS Files by user resource | eventstats sum(Files) AS TotalFiles by user resource | stats sum(Files) AS "Total FileCount", list(resource) AS Resource, list(TotalFiles) AS "FileCount" by user

DEAD_BEEF · ‎02-12-2020

Okay, but let's say I have 300 directories that I want (but there are over 5,000 I don't want)... must I still write them all out? I omitted sourcetype and everything else for brevity and assuming they are all the same sourcetype.

DEAD_BEEF · ‎02-11-2020

I've combed through inputs.conf and the various questions on answers but can't seem to get a definitive example in how to employ a whitelist or modify my monitor stanza to match on specific folders and their sub-directories per my use case. Example: match on /mnt/data/apple/desired_folder/*/* match on /mnt/data/apple/dir_1/*/* match on /mnt/data/apple/folder_two/*/* DONT match /mnt/data/apple/junk/*/*] DONT match on too many others to list Each directory in the whitelist, has one more sub-directory, then the log files themselves, of which I want everything in the folder. Do I have to write 3 monitor stanzas for this? failed attempts - no logs get pulled in [monitor:///mnt/data/apple/(dir_1|folder_two|index_this)/*/*] and [monitor:///mnt/data/apple/*/*/*] whitelist = (dir_1|folder_two|index_this) For now I've resorted to 3 monitor stanza's but I thought there is a cleaner way to do this in Splunk that I've completely forgotten/missed.

DEAD_BEEF · ‎02-08-2020

Just finished a rolling restart and no errors anymore after increasing the timeout to 30mins. Thank you both for the assistance!

DEAD_BEEF · ‎02-05-2020

I will try adjusting this. Each idx takes on average 15-20 mins, my current timeout setting is 15mins, so maybe I just expand it to 30m to be safe?

DEAD_BEEF · ‎02-05-2020

Most indexers were taking 15-20 mins. I will try adjusting the restart_timeout value but this is the first time I've seen these errors and I have restarted this cluster many times with each taking 15-20 mins just like always. That's what prompted me to ask about this issue. So this setting needs to be changed on the CM's server.conf, not the indexers themselves?

DEAD_BEEF · ‎02-02-2020

Using Splunk 7.3.3, after I initiated a rolling restart from the cluster master (multi-site indexer cluster), the first indexer began to restart. Then it showed batch adding, then the Indexer Clustering: Master Node page, showed that the indexer failed to restart [Mon Feb 2 12:47:52 2020] Failed to restart peer=<GUID> peer_name=<hostname>. Moving to failed peer group and continuing. [Mon Feb 2 12:47:52 2020] Failing peer=<GUID> peer_name=<hostname> timed out while trying to restart. I did a ping from the CM to this indexer and it returned fine. Connectivity was not an issue before the rolling restart and network connectivity appears to be working fine. Is there a timeout window or setting I can adjust to better accommodate network latency and give the CM more time to reach the peer? What does this mean for my rolling restart, will remaining peers be restarted but I should restart this one manually? How can I list this "failed peer group" to see all systems that may fail to restart?

DEAD_BEEF · ‎01-13-2020

We recently upgraded from 7.2.1 to 7.3.3 and from the _internal logs I can see that these new warnings are showing up across my indexer cluster. What is it saying and how do I go about fixing this? I've noticed that now I have an indexer just randomly lock up once a week. Any insight would be appreciated! 01-14-2020 00:58:27.994 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=_introspection Path coldPath='/opt/splunk/var/lib/splunk/_introspection/colddb' (realpath '/mnt/local/hot/_introspection/colddb') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by coldPath will *not* be volume-mananged. Please check indexes.conf for configuration errors. 01-14-2020 00:58:27.995 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=_telemetry Path coldPath='/opt/splunk/var/lib/splunk/_telemetry/colddb' (realpath '/mnt/local/hot/_telemetry/colddb') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by coldPath will *not* be volume-mananged. Please check indexes.conf for configuration errors. 01-14-2020 00:58:28.008 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=firedalerts Path coldPath='/opt/splunk/var/lib/splunk/firedalerts/colddb' (realpath '/mnt/local/hot/firedalerts/colddb') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by coldPath will *not* be volume-mananged. Please check indexes.conf for configuration errors. 01-14-2020 00:58:28.042 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=wineventlog Path homePath='/opt/splunk/var/lib/splunk/wineventlog/db' (realpath '/mnt/local/hot/wineventlog/db') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by homePath will *not* be volume-mananged. Please check indexes.conf for configuration errors. indexes.conf # global settings [default] lastChanceIndex = lastchance malformedEventIndex = malformedevent [volume:primary] path = /mnt/local/hot maxVolumeDataSizeMB = 14000000 [volume:cold] path = /mnt/local/cold maxVolumeDataSizeMB = 58200000 [volume:_splunk_summaries] path = /mnt/local/hot maxVolumeDataSizeMB = 1000000 homePath = volume:primary/$_index_name/db coldPath = volume:cold/$_index_name/colddb thawedPath = /mnt/local/cold/$_index_name/thaweddb homePath.maxDataSizeMB = 2000000 maxWarmDBCount = 250 maxDataSize = auto enableDataIntegrityControl = true frozenTimePeriodInSecs = 188697600 [main] homePath = volume:primary/defaultdb/db coldPath = volume:cold/defaultdb/colddb coldToFrozenDir = /mnt/local/cold/frozen/defaultdb thawedPath = /mnt/local/cold/defaultdb/thaweddb maxDataSize = auto_high_volume frozenTimePeriodInSecs = 31536000 ... ... ...

DEAD_BEEF · ‎10-10-2019

I was asked to come up with some rough numbers on how long it would take to rebuild an indexer if one completely died. So, if I were to remove an existing indexer from my multi-site cluster (2 sites) and replace it with a new one in it's place. I know there are a lot of variables but I am asking for help on how to get some rough numbers. The last time this happened, it took about 12 hours for the cluster to meet RF/SF after replacing a single indexer. How can I calculate an estimate for this? To simplify the question, assume the following: 20 indexers (10 at each site) 10TB of data (hot+cold) on each indexer RF=3, SF=2 Splunk recommended hardware (800 IOPs) Minimal WAN latency between the two sites (100-150ms) Default 5 fix-up tasks per indexer 50,000 buckets per indexer 10 Gigabit circuit In essence, the cluster would need to reproduce 10TB of data, or 5TB would be done by indexers in 1 data center and 5TB by the other (assuming 50% split in work load). Would this just be 10TB = 80,000 Gb / 5 Gbps = 16,000 seconds (4.5 hours)? That's very conservative compared to my real life experience where it took 12 hours. What am I missing in my calculation?

DEAD_BEEF · ‎08-28-2019

My app is on my HF. Does it also need to be on the SH to access the search-time field extractions?

DEAD_BEEF · ‎08-27-2019

I have a custom set of logs where I wrote out the regex to parse it. I then created a field extraction via the search head GUI and everything worked perfectly. I decided to delete the local SH field extractions and instead add the field extractions to the app I wrote (for portability) to ingest these logs and now none of the extractions are working when I ingested new data and searched. My app is installed on a HF where the logs are being monitored. I copy+pasted the exact regex I was using from the GUI extractions and using search and nothing is parsed, no fields are displayed, punct is being calculated, and KV_MODE is being ignored... what am I doing wrong? inputs.conf [monitor:///mnt/data/monitor/foo/foo/bar.log] index = main sourcetype = custom:dns queue = parsingQueue disabled = 0 props.conf [custom:dns] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = ^.*?\-\s TIME_FORMAT = %s TZ = GMT ANNOTATE_PUNCT = false KV_MODE = none EXTRACT-custom_dns_fields = \d+\s\-\s\d+\s(?<timestamp>[^,]*),(?<src>[^,]*),(?<src_port>[^,]*),(?<query>[^,]*),IN,(?<query_type>[^,]*),(?<EDNS0>[^,]*),(?<EDNS0_size>[^,]*),(?<DNSSEC>[^,]*),(?<TCP>[^,]*), TRANSFORMS-custom_dns_response = custom_dns_response transforms.conf [custom_dns_response] REGEX = (\d+):([\.a-zA-Z0-9-]+)\s FORMAT = response_code_id::$1 response_code::$2 CLEAN_KEYS = 0 MV_ADD = 1

DEAD_BEEF · ‎08-27-2019

I am using a CDN and have obtained my DNS logs. Some of the DNS logs have multiple values for the field response ID and response . I am trying to make a field extraction for this but not sure how to write this to match "1 or more times". So in some cases, 1 log can have multiple response_id and response values. Can this be done via the GUI field extractions or does it HAVE TO be done via props/transforms as here and here? In this case, I want to extract all the response ID's and responses ( 300:response ) from the log below. The number of responses can vary (sometimes 1, sometimes 3, etc.). Six responses in the log snippet below. sample log 394347 - 1566861761 26/08/2019 23:22:41,1.2.3.4,12345,mywebsite1.com,IN,NS,E,4096,D,,300:a8-cdm.com 300:a7-cdm.com 300:a16-cdm.com 300:a22-cdm.com 300:a1-cdm.com 300:a9-cdm.com desired parsing response_id response 300 a8-cdm.com 300 a7-cdm.com 300 a16-cdm.com 300 a22-cdm.com 300 a1-cdm.com 300 a9-cdm.com my regex (which isn't working) (.*?,){10}((?<response_code_id>\d+):(?<response_code>[\.a-zA-Z0-9-]+)\s?){1,}

DEAD_BEEF · ‎07-09-2019

Okay, I got it. outputting it to /dev/null caused it to send stdout to the trash. I removed that, but left the wget -O- ... and it started indexing immediately.

DEAD_BEEF · ‎07-09-2019

I checked all time with index=main sourcetype=test and nothing... Also, no timestamps in this data. I assumed Splunk will write the index time for all the events in the file.

DEAD_BEEF · ‎07-09-2019

Hi, I am sending to stdout now using what @rob_jordan said (as I didn't know how). I don't see data in main index but seeing this log. 07-10-2019 02:07:56.729 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/my_app/bin/script.sh" 2019-07-10 02:07:56 (8.61 MB/s) - written to stdout [4011976]

DEAD_BEEF · ‎07-09-2019

@rob_jordan I changed the script to just be the 1-liner that you put (as I didn't know how to send data to stdout). Now it shows this and no data in index=main 07-10-2019 02:07:56.729 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/my_app/bin/script.sh" 2019-07-10 02:07:56 (8.61 MB/s) - written to stdout [4011976]

DEAD_BEEF · ‎07-09-2019

Is there a way I can adjust the script to have splunk read the stream and index it, rather than save the file locally then index it?

DEAD_BEEF · ‎07-09-2019

So I put a full path /var/log/file.txt and now the file is saved on the HF, but nothing was indexed...

DEAD_BEEF · ‎07-09-2019

If I just want splunk to read the stream and then send it to indexers rather than download a log file and then send it, how would I adjust the script?

DEAD_BEEF · ‎07-09-2019

-rwxr-xr-x. 1 splunk splunk 217 Jul 9 23:35 script.sh

DEAD_BEEF · ‎07-09-2019

Hello. I have a HF and I want it to download a .csv file from another internal server. Right now, I can download it as the splunk user using wget on CLI so I know connectivity and permissions are no issue. I looked at scripted inputs but I don't think that's the right way about it, as I can't get it to work. Not sure how to go about this? I just want to download a csv file and then send it to my indexer tier. /opt/splunk/etc/apps/my_app/bin/script.sh /usr/bin/wget -O file.csv 'https://myserver.com/feeds/list?v=csv&f=indicator&tr=1' exit 0 /opt/splunk/etc/apps/my_app/local/inputs.conf [script://./bin/script.sh] index = main sourcetype = test interval = 600.0 disabled = 0 splunkd.log 07-10-2019 00:08:18.082 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/my_app/bin/script.sh" file.csv: Permission denied I checked and the app is owned by splunk:splunk. The script is 755. I ran the ad-hoc command below as the splunk user and it downloaded the file just fine /opt/splunk/bin/splunk cmd ../etc/apps/my_app/bin/script.sh I tried adding the input through the HF's gui (Settings > Data Inputs > Scripts > Add new) but my app and script are not showing up in the dropdown...

Posts	251
Solutions	11
Karma Given	158
Karma Received	54
Member Since	‎08-14-2014

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

How to create an area chart that displays an avera...

Stats count by when field exists, otherwise use an...

How to truncate the number of results in my table?

Splunk inputs and whitelists --- how to?

Multi-site Indexer rolling restart - indexer fails...

Volume configuration will not manage space used by...

How to estimate indexer data replication time from...

Field extractions for my app not showing up in sea...

How to field extract one or more times to the same...

Scripted input permission denied

Re: Stats count by when field exists, otherwise us...

How to create an area chart that displays an avera...

Stats count by when field exists, otherwise use an...

Re: How to truncate the number of results in my ta...

How to truncate the number of results in my table?

Re: Splunk inputs and whitelists --- how to?

Splunk inputs and whitelists --- how to?

Re: Multi-site Indexer rolling restart - indexer f...

Re: Multi-site Indexer rolling restart - indexer f...

Re: Multi-site Indexer rolling restart - indexer f...

Multi-site Indexer rolling restart - indexer fails...

Volume configuration will not manage space used by...

How to estimate indexer data replication time from...

Re: Field extractions for my app not showing up in...

Field extractions for my app not showing up in sea...

How to field extract one or more times to the same...

Re: Scripted input permission denied

Re: Scripted input permission denied

Re: Scripted input permission denied

Re: Scripted input permission denied

Re: Scripted input permission denied

Re: Scripted input permission denied

Re: Scripted input permission denied

Re: Scripted input permission denied

Scripted input permission denied