About jhupka

jhupka · ‎05-30-2017

When you look at the queue order of Parsing Queue - Aggregator Queue - Typing Queue - Index Queue, the one farthest to the right that is full most likely has the root cause, and then things backup to the left (and once the Parsing Queue is full, that's when it affects the Forwarders). Since the Index Queue is full the culprit often comes down to read/write issues on the Indexer - writing to disk is what happens after the Index Queue. These are the questions I would ask of your environment next: Are you running out of disk space on your Indexers? Are your events coming in from Forwarders too fast or too much data, and the IOPS you have available not enough for the two Indexers you have? If you take a look in the DMC for License Usage, what are your trends and how does that compare to when the Indexers have a problem? As a side note, I would definitely look into adding a third Indexer no matter what. You have all of these other supporting Splunk instances around your Index Cluster, but if one goes down you immediately lose 50% of your capacity. Adding a third will give you that much more headroom. Also, Splunk scales wonderfully horizontal at the Indexer layer...it is cheap/easy to improve all aspects of your environment by adding more there (indexing speed, search speed, disk space, etc).

jhupka · ‎05-30-2017

Do you have the Splunk Monitoring Console (splunk_monitoring_console app) configured on one of your instances (not your SH Cluster)? If you do not, start by configuring that: http://docs.splunk.com/Documentation/Splunk/latest/DMC/ Once it is, go to the Indexing menu and start looking through those pages, especially the individual instance page, to see how things are performing. Are particular queues filling up? Running into memory issues? CPU issues? Also, if you search for any errors that your Forwarders are reporting, what things proceed the blocked message you outline above? Essentially what you find from these questions will help narrow down the root cause. For example, if your Merging Pipeline on the Indexer(s) are always at 100%, then that will bubble back to the Forwarders since the IDX will say, "Hey, I'm backed up...stop sending data..." Merging Pipeline issues in turn have very specific root-causes, such as relying on the default event breaking and merging settings for sourcetypes defined in your props.conf instead of setting stuff like LINE_BREAKER and SHOULD_LINEMERGE manually.

jhupka · ‎10-03-2016

There's a lot of different ways to approach this - are you planning on creating an alert from a scheduled search running every 15m? Do you want to just create a dashboard with uncommon or recent errors? What do you want to tell the consumer of the search result about the error - what host it came from? The error itself? Any other info? For now we can choose a simple case. Let's assume your data's sourcetype is sagineshmk_logs, and the error is in a field named error_name. You could do a search like the following: sourcetype=sagineshmk_logs earliest=-30d@d | stats count as error_count, max(_time) as _time by error_name | where error_count=1 For a production environment, though, I might not want to run the above search every 15m and constantly look over 30 days of data. In that case I may split the search into two separate pieces - one search to compile a list of unique errors in the last 30 days (e.g. run that daily), then a second search that only looks at the past 15 minute's worth of errors and compares it to my daily unique error list.

jhupka · ‎10-03-2016

There's several ways to see information about these scheduled searches. In addition to looking through your scheduled search configurations like you mention (and they're also in all of the savedsearches.conf files under the hood), you can manually find these by clicking on the Activity->Jobs menu in the upper right of the Splunk UI. From there you can use the various drop-downs to show All searches, then limit to just the ones that are Running. It is also possible to search for this information. Here is a simple search that will find any information about Splunk queuing your searches due to hitting these limits. Run this over the timeframe you're concerned about (+/- 15 minutes, for example) to see what could be going on (you'll typically need to be an admin to search the _internal index): index=_internal sourcetype=splunkd component=DispatchManager queued If you want to search for them, the following search will show currently running searches: | rest /services/search/jobs | search dispatchState=RUNNING BUT, you can end up in a Catch-22 if the Search Head is queuing your search and you're trying to run a search to see what is/was running.

jhupka · ‎07-20-2016

That is a great idea...although I'm not 100% sure if our sys admins will be available for this and the powers that be are pushing all of this type of work off to be manually handled by each individual application team.

jhupka · ‎07-20-2016

Has anyone seen what happens to a Universal Forwarder when the filesystem it is running from goes away? I just found out about some weekend maintenance to our network storage that will cause connectivity issues with the SAN mount points we have our Splunk UFs installed on. I'm not sure what Splunk will do when the mount disappears, and may not have a lot of time to test this scenario. A few basic thoughts I have on what would occur: Splunk can’t log its own internal log files Splunk can’t update its fishbucket data Splunk can't read/run scripted inputs (not too worried about this, though - it is ok if we are missing that data since it is mostly *nix) Will Splunk continue forwarding data during this scenario? How I could approach the handling of this: Manually Splunk Forwarders down before hand, and manually start up after filesystem comes back (most work for jhupka, but safest scenario) Let things be, then trick Splunk Forwarders into restarting via Deployment Server after filesystem comes back to start fresh (minimal work/coordination) Do absolutely nothing (least work, jhupka gets to sleep in on Saturday morning)

jhupka · ‎03-18-2016

Solution: There is a bug in Red Hat that causes things to go awry: https://access.redhat.com/solutions/1386323 In our case, the solution was to patch from kernel 2.6.32-504.8.1 to 2.6.32-504.16.2

jhupka · ‎03-18-2016

You could do it in the search with an eval to merge the two hosts' data into one: sourcetype=rsa_auth AUTHN_LOGIN_EVENT | eval host=if(host="serverA.foo.com" OR host="serverB.foo.com", "single_server.foo.com", host) | timechart span=1d count by host

jhupka · ‎03-18-2016

This is less of a question and more of a record on Splunk Answers of an issue we ran into. Symptoms: You are on Red Hat 6.6, 7.0, or 7.1 The Indexer stops receiving data from Forwarders, but looks to be up and running fine otherwise. In a sense it seems stuck for our TCP input port on splunkd. For example, the Indexer still participates as a peer for any searches, and the Indexer also looks to continue indexing any data generated locally - internal Splunk logs or scripted inputs running on that Indexer. Essentially the TCP input port is bad, but the splunkd admin port is fine. When running the following search, any Indexers having this problem would show up has having 1 or 0 Distinct Hosts since they weren't able to receive anything from Forwarders: index=_internal sourcetype=splunkd earliest=-15m | stats count as event_count, dc(host) as distinct_hosts by splunk_server Also, we would notice a lot of TCP connections sitting in a CLOSE_WAIT or SYN_RECV state when running netstat -an on the Indexer. Workaround: Use pstack to dump a list of processes associated with splunkd. This will magically fix things without having to restart the Indexer: pstack `head -1 $SPLUNK_HOME/var/run/splunk/splunkd.pid` Solution: There is a bug in Red Hat that causes things to go awry: https://access.redhat.com/solutions/1386323 In our case, the solution was to patch from kernel 2.6.32-504.8.1 to 2.6.32-504.16.2 Environment Details: - Splunk: 6.2.5 (build 272645) - OS: Red Hat Linux 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux - HW: Cisco UCS C240

jhupka · ‎12-04-2015

I've had similar issues when going from 6.2.5 to 6.3.1 - missing user, broken images, menus don't work. It sucked and I would get so upset not even kitten pictures on the internet would cheer me up. This is due to a bug in Python 2.7.9, a new version of Python Splunk uses under the hood in 6.3. See this Splunk Answer for a solution. It worked like a charm for me: https://answers.splunk.com/answers/312247/after-upgrading-a-search-head-cluster-to-splunk-63-1.html http://docs.splunk.com/Documentation/Splunk/6.3.1/ReleaseNotes/Knownissues#Splunk_Web_and_Home_interface_issues

jhupka · ‎02-20-2015

This a such a great question...not necessarily the topic (which is good use case), but mainly in the sense it shows that BorrajaX has stated what is not known, what is known, what has been tried, and evidence of trying to find an answer on their own. Great job being thorough versus just half-assing a quick question that are so hard to answer on this site w/o requesting more info. Also, a well explained answer by dwaddle. Karma awarded to both BorrajaX and dwaddle for modeling what a good question-answer interaction should be on Splunk Answers.

jhupka · ‎02-12-2015

You could do something like this: index=ind source=src1.log Name=* info=HIGH | table Name | evenstats count by Name | search count>1

jhupka · ‎12-10-2014

Can you provide a sample event? If it is listed in the Field Extractions settings that means Splunk knows it is defined...but if you're not seeing it within the Existing Field list on the left of your search results there might be some other issue in your RegEx that isn't working right so nothing ever gets extracted. As a side note, it looks like maybe the less-than and greater-than portions of your field extractions got left out by the website - you might want to try editing it and markup your EXTRACT lines as blocks of code.

jhupka · ‎12-10-2014

One possibility is to get the dynamic value via a subsearch: index=_internal | tail [ | stats count | eval search=10 | fields search ] That will effectively expand to a search that looks like this: index=_internal | tail ((10)) So in your case just come up with a subsearch that dynamically obtains your value.

jhupka · ‎08-07-2014

Making this change in distsearch.conf is the "Big Giant Hammer" solution to this...I suggest trying my2ndhead's steps below since that tries to address just the hiccup that caused this problem. This way you get rid of the bad-stuff that happened to be created instead of hiding the problem behind a setting change. If you have big bundles, turning off the delta functionality will cause unnecessary overhead if the root cause was a minor hiccup in the bundle delta replication process.

jhupka · ‎08-07-2014

Thanks, my2ndhead! We had this issue, and this answer led us in the right direction. We found specific issues between one of our SH and one of our Indexers: Determined which SH was causing the issue (e.g. mySearchHead2) and which Indexer from the error message (e.g. myIndexer3) Shutdown SH "mySearchHead2" Go to the $SPLUNK_HOME/var/run/searchpeers directory on myIndexer3 On Indexer, do: rm -rf mySearchHead2* Start SH mySearchHead2 back up Problem solved (a whole new bundle will get sent from mySearchHead2 to myIndexer3 and any bad delta's are gone)

jhupka · ‎07-23-2014

Great answer - works like a charm.

jhupka · ‎05-30-2014

If you only want the percAddTrials and percAddSub to remain just do the following with your search (add fields fommand): XXXXXX NOT(resultType=XXXX) activity=foo OR activity=bar | timechart span=1h count by activity | eval total = foo + bar | eval percAddTrials = round(foo100/total,1) | eval percAddSub = round(bar100/total,1) | fields percAddTrials, percAddSub

jhupka · ‎05-30-2014

So I don't really have a quick and dirty answer to shutting off an index that doesn't come with its own caveats. E.g. if I had to do this I might try something like: Just manually off the inputs going to that index and deploy it out real quick using DS. If you don't care about anything in that index for now, you could just remove its config from your index.conf. But you'll get warnings when stuff goes to that index, but those events get dropped and I'm pretty sure don't affect license. Just don't delete the index's directories so when you re-enable the index old data is still there

jhupka · ‎05-30-2014

Look into what is called a Scripted Input, defined in the inputs.conf of your app. Running a search to execute a script is a bit "heavy." The scripted input can be set to execute however often you want, and splunkd will execute it for you on that schedule. Also, the Std Out of your script would get indexed as well, so make sure you configure the index, source, sourcetype for it to show where you want that data to go. The script itself could do anything - write your file, run a unix command (that's how the unix app stuff works in Splunk), fetch data from an external source for indexing, etc. Also, what you did in commands.conf above would be for creating a custom search command in Splunk. E.g. you want to make a new command called my_script: ... | eval foo="bar" | my_script | head 10

jhupka · ‎05-29-2014

Wouldn't your REGEX=. just match single character events for sending to nullQueue? Maybe try REGEX=.* to make it more greedy and match everything.

jhupka · ‎05-28-2014

Would something like this pattern be sufficient...you end up with rows of each tuple. Example using the search I had above: index=* earliest=-5m | eval marker=splunk_server." - ".host | chart count, first(_time) as timestamp by marker Example with the last part of your search: ... | eval marker=city."=".type | chart limit=0 last(duration) AS Total last(p_duration) AS Percent by marker

jhupka · ‎05-28-2014

You would do something like this: index=* earliest=-5m | chart format="$VAL$:$AGG$" count, first(_time) by host, splunk_server Note that this is something new in Splunk 6...if you're on Splunk 5 you'll get an error about the format option.

jhupka · ‎05-28-2014

If your xlatesrc is always empty in your subsequent events, can you do something like: ... | eval tuple = if(isnull(xlatesrc), src.s_port.dst.service, dst.service.xlatesrc.xlatesport) | stats <whatever values you want> by tuple Essentially conditionally build your tuple of the proper IP+port+IP+port based on the xlatesrc being there or not for the fields that have your unique identifier.

jhupka · ‎05-27-2014

In terms of why Search A is way faster than Search B is Splunk is doing two different searches under the hood. In the first Search A, Splunk essentially looks for "somepath" - The Splunk Indexer is only going to return events that have "somepath" in them. But in Search B, Splunk will likely breakup your terms because of the dash and look for "some" AND "path" So in a way your search could be looking for many more events than necessary. A good thing to do is look at the Job Inspector and see what Splunk is doing under the hood. Also, you might try this Search C and see if its performance is on par with Search A: TERM(some-path) | chart count by date_mday The TERM search operator tells Splunk to use the contents as a single term instead of splitting up your search terms at the dash.

Posts	36
Solutions	13
Karma Given	8
Karma Received	44
Member Since	‎12-08-2011

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

What happens when Universal Forwarder loses its fi...

Indexer stops receiving data from forwarders

Any reason not to install PDF server on every Linu...

Re: Forwarding to indexer group [myGroupName] bloc...

Re: Forwarding to indexer group [myGroupName] bloc...

Re: query on splunk for new error(exeption on serv...

Re: How do i get a list of all these concurrent se...

Re: What happens when Universal Forwarder loses it...

What happens when Universal Forwarder loses its fi...

Re: Indexer stops receiving data from forwarders

Re: How would I merge the events from two log file...

Indexer stops receiving data from forwarders

Re: After upgrading to Splunk 6.3, why do users ha...

Re: How to send an alert email the first time (sin...

Re: Is there an opposite to dedup?

Re: Problem extracting field from a field

Re: How can I pass a variable to the tail command?

Re: [indexer] Streamed search execute failed becau...

Re: [indexer] Streamed search execute failed becau...

Re: Drill down to absolute URL using click.value

Re: Using calculated values to create timechart --...

Re: How to drop all entries to a specific index?

Re: How to write a file into the app folder throug...

Re: How to drop all entries to a specific index?

Re: How to use the chart command optional argument...

Re: How to use the chart command optional argument...

Re: How to reassemble bidirectional flows with tra...

Re: Splunk returns some queries much faster than o...