About hexx

hexx · ‎02-25-2013

The 'fwdType' field is read from metrics.log events for group=tcpin_connections, recording activity and information on a per-forwarder basis. Here's an example: 02-20-2013 21:20:02.316 -0500 INFO Metrics - group=tcpin_connections, 20.31.175.123:2783:9997, connectionType=cooked, sourcePort=2783, sourceHost=thing.example.net, sourceIp=20.31.175.123, destPort=9997, _tcp_Bps=96.75, _tcp_KBps=0.09, _tcp_avg_thruput=0.29, kb=2.94, _tcp_Kprocessed=12246.86, _tcp_eps=0.13, build=140868, version=5.0, os=Windows, arch=Intel, hostname=thing, guid=AB557B33-AF37-4D9E-BE4C-D6F3F859E1C9, fwdType=uf, ssl=false, lastIndexer=20.31.175.89:9997, ack=false It does seem somewhat surprising that your Deployment Server is registered as 'lwf' given that if it is indeed sending it's _internal index to your indexers, it is doing this as a regular forwarder. You may want to look for your DS' IP and/or hostname in metrics.log and see what value of 'lwf' is registered there for the corresponding group=tcpin_connections events.

hexx · ‎02-19-2013

Thank you. I will make some inquiries.

hexx · ‎02-19-2013

@tmeader said: Splunk support stated that there is a bug opened on this setting though, as in "to be changed in the future." Were you given a reference number for this bug? If yes, would you please share it?

hexx · ‎02-19-2013

This is important feedback that we (Splunk Support) would like to (and should!) bring to our developers. If you don't have a case already open against this particular issue, please create a new one.

hexx · ‎02-19-2013

The lowering of the historical search concurrency limit in 5.x is not inadvertent. This was done by design: The idea is that in 5.x, job queueing in combination with a lower search concurrency will provide a better user experience than high search concurrency and no job queueing. For more details, see this Splunk Answer. That being said, if you have established that your system can and should allow a higher number of concurrent searches, feel free to increase the value of these parameters in limits.conf. I would typically recommend to do this progressively and not jump back to max_searches_per_cpu = 4 immediately, but rather start with a value of 2 or 3.

hexx · ‎02-14-2013

A Windows 2008R2 Domain Controller in another geographical, and the Security Events are perpetually multiple days behind. There's no limit on outgoing forwarder throughput; and watching a local file on the DC with the forwarder works fine, with no notable latency sending data to the indexers. Clearing the Windows Security log allowed the events to catch-up for a short while, but they quickly fell behind again. The other Event Log channels were turned off, and that didin't help either.

hexx · ‎02-12-2013

Unfortunately, the form search modules in "vanilla" Splunk do not provide an option to assemble the terms provided by the user into a "TERM1 OR TERM2 OR TERM3" expression. You would have to write your own module to perform this task.

hexx · ‎02-12-2013

The answer is "no". Splunkweb cannot be configured to require the browser to present a client certificate.

hexx · ‎02-07-2013

Is the percentage 25% of one CPU, or 25% of the total available processing power? That percentage represent the approximate usage of one CPU core. Does this (relatively) high use of CPU time in the regexreplacement queue suggest that my regexes are inefficient? Possibly, that does seem to be on the high side for the regexreplacement processor. That being said, what really matters is: Is the regexreplacement processor being a bottleneck? This is true if the typing queue fills up and remains saturated while the indexing queue is starved or near empty. What is the proportion of the CPU usage of the indexer processor vs. the regexreplacement processor? In your situation, it seems that indexer processor is using less CPU power than regexreplacement, which is indeed a bit imbalanced. Are there suggestions to keeping this processing queue a little less busy? Yes, but they are not always trivial to implement: Reduce the number of index-time transformations using regular expressions Avoid regular expressions in index-time transformations that operate against _raw Optimize your regular expressions in index-time transformations to run faster. Anchoring can sometimes do wonders to prevent unnecessary back-tracking.

hexx · ‎02-05-2013

According to serverclass.conf.spec: # Property inheritance # Stanzas in serverclass.conf go from general to more specific, in the following order: # [serverClass] -> [serverClass:<name>] -> [serverClass:<scname>:app:<appname>] # # Some properties defined at a general level (say [serverClass]) can be # overridden by the more specific stanzas as it applies to them. All inheritable # properties are marked as such. (...) filterType = whitelist | blacklist (...) * Can be overridden at the serverClass level, and the serverClass:app level. It seems that a filtering statement at the app level (most specific) will override a colliding statement at the class-level (least specific).

hexx · ‎01-30-2013

Did the answer provided here ultimately allow to resolve this issue?

hexx · ‎01-30-2013

The compressed attribute only matters if you are forwarding without SSL. It determines whether Splunk will or not perform "native" compression on a per-data chunk (UF, LWF) or per-event (HWF) basis for outgoing data. This must be enabled on both ends for things to work. If you are forwarding with SSL, unless you explicitly set useClientSSLCompression to false, you will automatically benefit from SSL compression over the data stream. This is significantly more efficient than Splunk-native compression and should be favored in the case of bandwidth restrictions between forwarder and indexer.

hexx · ‎01-02-2013

Such a correlation is indeed possible: Search for search strings containing the delete command in events from source $SPLUNK_HOME/var/log/splunk/audit.log in the _audit index: 01-02-2013 05:57:29.121 -0800 INFO AuditLogger - Audit:[timestamp=01-02-2013 05:57:29.121, user=odisciullo, action=search, info=granted , search_id='1357135049.34106', search='search index=_internal | head 1 | delete', autojoin='1', buckets=0, ttl=600, max_count=10000, maxtime=8640000, enable_lookups='1', extra_fields='', apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME', savedsearch_name=""][n/a] Search for the SID (search ID) string of those searches against events from source $SPLUNK_HOME/var/log/splunk/web_access.log in the _internal index: 10.160.255.78 - odisciullo [02/Jan/2013:05:57:30.590 -0800] "GET /en-US/splunkd/search/jobs/1357135049.34106/timeline?offset=0&count=1000 HTTP/1.1" 200 - "https://undiag.splunk.com:8000/en-US/app/support_diag/flashtimeline?q=search%20index%3D_internal%20%7C%20head%201%20%7C%20delete&earliest=0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.101 Safari/537.11" - 50e43cca977fd3acab1850 13ms Here's a search that should be able to do this correlation for you and build a table of results showing time, search string, user and client IP for each search that contains the delete command: index=_internal source=*/web_access.log "/services/search/jobs/" | rex "/services/search/jobs/(?<search_id>[\d\.]+)" | join search_id [search index=_audit delete | rex "search='(?<search>.*?)', autojoin" | regex search="\|\s*delete" | replace '*' with * in search_id | fields _time, user, search, search_id] | stats min(_time) AS _time first(clientip) AS "client IP" first(user) AS User values(search) AS search by search_id | rename search_id AS "search ID"

hexx · ‎12-12-2012

No, this is because heavy-weight forwarders do not record per_*_thruput metrics. You should be able to see events recording the queue sizes, though.

hexx · ‎12-12-2012

Are you sure that you are forwarding the _internal events from your HWF to your indexers? Also, are you sure that you added your forwarder to the splunk_servers_cache.csv lookup file under the right hostname?

hexx · ‎12-10-2012

@Drainy is correct. This problem is caused by a regression in Splunk 5.0 and 5.0.1 which impacts certain methods to list indexes, such as: The eventcount command REST API calls to the /services/data/indexes endpoint In 5.0 and 5.0.1, these methods fail to use cached assets and end up iterating over the files of your indexes themselves. This results in a much slower execution, which eventually affects whatever consumer calls the method. In the Manager user interface, this affects some of the pages that need to present a list of indexes such as: The page showing the details of a saved search (Manager > Searches & Reports > $SAVED_SEARCH_NAME) The setup page for most data inputs (Manager > Data Inputs > Files & Directories > Add new) The page where you can list and edit indexes (Manager > Indexes) The page where you can edit roles (Manager > Access controls > Roles) If the index-listing method takes more than 30s to complete, Splunkweb will time out on it and display a "500 - internal server error" message. This issue will be fixed in maintenance release 5.0.2, which we are aiming to deliver early in 2013. We are also working on intermediary patch release 5.0.1.1 to specifically address this issue. For reference, the bugs behind this problem are SPL-57518 and SPL-58650.

hexx · ‎12-10-2012

This problem is caused by a regression in Splunk 5.0 and 5.0.1 which impacts certain methods to list indexes, such as: The eventcount command REST API calls to the /services/data/indexes endpoint In 5.0 and 5.0.1, these methods fail to use cached assets and end up iterating over the files of your indexes themselves. This results in a much slower execution, which eventually affects whatever consumer calls the method. In the Manager user interface, this affects some of the pages that need to present a list of indexes such as: The page showing the details of a saved search (Manager > Searches & Reports > $SAVED_SEARCH_NAME) The setup page for most data inputs (Manager > Data Inputs > Files & Directories > Add new) The page where you can edit roles (Manager > Access controls > Roles) If the index-listing method takes more than 30s to complete, Splunkweb will time out on it and display a "500 - internal server error" message. This issue will be fixed in maintenance release 5.0.2, which we are aiming to deliver early in 2013. We are also working on intermediary patch release 5.0.1.1 to specifically address this issue. For reference, the bugs behind this problem are SPL-57518 and SPL-58650. IMPORTANT NOTE: We are aware of a residual issue in version 5.0.2 that is responsible for timeouts on a smaller set of Manager pages: Manager » Indexes Manager » Data inputs » * » Add new This issue will primarily affect systems with many (typically, thousands) of buckets in the hot/warm index directories and running on Windows (which does not passively use available system memory for filesystem cache). The bug reference for this issue is SPL-61718 and it is slated to be fixed in 5.0.3. We are also working on the release of a 5.0.2.x patch to address this issue, which you'll be able to obtain by contacting support.

hexx · ‎12-05-2012

Today we have published a maintenance release (2.3.1) of the S.o.S app to address certain compatibility issues with Splunk 5.x. I encourage you to test this new version which will hopefully resolve any problems you currently are experiencing.

hexx · ‎12-05-2012

The SoS app absolutely will run on a Windows search-head and/or indexer. In order to enable resource usage monitoring for Splunk processes on Windows, you'll need to enable the ps_sos.ps1 powershell-based scripted input. Please consult the app README file or the in-view help for the "Splunk CPU/Memory Usage" view for details.

hexx · ‎12-05-2012

The file you want to look at is located in %SPLUNK_HOME%\var\log\splunk\splunkd.log . It's a simple text file, so you can open it with any text editor. If you have purchased a license, I would suggest that you open a support case and attach a splunk diag to it.

hexx · ‎12-05-2012

If you tail -f splunkd.log while you attempt to login, do you see any errors or warnings?

hexx · ‎12-05-2012

Although we cannot commit to a date, we are hoping to release 5.0.2 early next year.

hexx · ‎12-05-2012

As of Splunk 5.0.1, the Cluster Master cannot push apps, the only "app container" that is pushed from the master's $SPLUNK_HOME/etc/apps/master-apps to the slaves' $SPLUNK_HOME/etc/apps/slave-apps is the cluster "app". As Deployment Server and 3rd party configuration deployment software such as Puppet or Chef are currently unsupported, we will be bringing in the following changes in Splunk 5.0.2 to address this situation: The Cluster Master will be able to push apps from its $SPLUNK_HOME/etc/apps/master-apps repository to the slaves' $SPLUNK_HOME/etc/apps/slave-apps . As the "cluster" app in that repository is special, it will be renamed to "_cluster".

hexx · ‎11-27-2012

It was actually kind of hard to find, because it's in the etc/apps/gettingstarted/local/inputs.conf file (could that be affecting it?) No, that's not the issue. That location just happens to be where the UI committed your configuration to when you enabled this input. More details on how Splunk layers configuration files can be found here.

hexx · ‎11-26-2012

Another possibility is that the S3 modular input uses a checkpoint logic that somehow got messed up. I would recommend looking in splunkd.log for messages indicating that the scripted input is running and what the outcome of the command execution is.

Posts	721
Solutions	220
Karma Given	1917
Karma Received	2180
Member Since	‎07-02-2010

Online Status	Offline
Date Last Visited	‎05-03-2022 09:33 PM

On Linux, how can I leverage operating system faci...

How do optimizations for field-based searches work...

Why are my real-time alerting searches no longer s...

What is the cause of these socket errors reported ...

Forwarder shows extreme lag or latency when sendin...

When is it appropriate to set followTail to 'true'...

How can I trigger the re-indexing of a single file...

How do I check if my custom TIME_FORMAT is success...

Can the Splunk on Splunk app query a heavy forward...

Time stamp of historical syslog data gets incorrec...

Re: fwdType for Deployment Server

Re: Not a question, but tip for all those running ...

Re: Not a question, but tip for all those running ...

Re: Not a question, but tip for all those running ...

Re: Not a question, but tip for all those running ...

Forwarder shows extreme lag or latency when sendin...

Re: Form with a multi-line text box that will OR e...

Re: Require Client Certificate from Browser to Spl...

Re: Reduce time spent in regexreplacement queue?

Re: Simplifying serverclass.conf?

Re: SoS not being able to fill in the host to quer...

Re: useClientSSLCompression vs compressed on forwa...

Re: Splunk on splunk SOS activity logging

Re: Heavy Forwarder Thruput

Re: Heavy Forwarder Thruput

Re: Splunk 5.0 Error - An error occurred while ren...

Re: How to fix 500 internal server error on Splunk...

Re: SOS 2.3.0 on Splunk 5

Re: SoS cannot run on windows Splunk server?

Re: When logging in, an error appears: "The splunk...

Re: When logging in, an error appears: "The splunk...

Re: App deployment in Splunk 5.0 clustering from a...

Re: App deployment in Splunk 5.0 clustering from a...

Re: Splunk for Amazon S3 Add-on not able to fetch ...

Re: Splunk for Amazon S3 Add-on not able to fetch ...

Join the Conversation