Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About jnicholsenernoc
jnicholsenernoc
Path Finder
Member since:
03-19-2012
06-05-2020
Community Statistics
| Posts | 36 |
| Solutions | 2 |
| Karma Given | 0 |
| Karma Received | 11 |
| Member Since | 03-19-2012 |
Activity Feed
- Got Karma for Re: What is _indextime, really?. 11-14-2023 02:20 PM
- Got Karma for Re: How to hide an app from the home page, but still have access to the views inside the app?. 06-05-2020 12:47 AM
- Got Karma for Re: Why is my TCP Input data being cut off? Is there a limit I can configure?. 06-05-2020 12:47 AM
- Got Karma for Re: How to subtract field values every 5 minutes and display these results on a graph?. 06-05-2020 12:47 AM
- Got Karma for Re: Troubleshooting why only part of a log file was imported. 06-05-2020 12:47 AM
- Got Karma for Re: Troubleshooting why only part of a log file was imported. 06-05-2020 12:47 AM
- Got Karma for Re: How to create and share users/roles between search heads on Splunk 6.2 Search Head Clustering deployment?. 06-05-2020 12:47 AM
- Got Karma for Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?. 06-05-2020 12:47 AM
- Got Karma for Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?. 06-05-2020 12:47 AM
- Got Karma for How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL?. 06-05-2020 12:47 AM
- Got Karma for How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL?. 06-05-2020 12:47 AM
- Posted Re: What is _indextime, really? on Knowledge Management. 03-01-2016 03:13 PM
- Posted Re: What is _indextime, really? on Knowledge Management. 03-01-2016 03:09 PM
- Posted Re: How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL? on Deployment Architecture. 01-14-2016 08:12 AM
- Posted How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL? on Deployment Architecture. 11-17-2015 10:31 AM
- Tagged How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL? on Deployment Architecture. 11-17-2015 10:31 AM
- Tagged How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL? on Deployment Architecture. 11-17-2015 10:31 AM
- Tagged How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL? on Deployment Architecture. 11-17-2015 10:31 AM
- Tagged How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL? on Deployment Architecture. 11-17-2015 10:31 AM
- Tagged How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL? on Deployment Architecture. 11-17-2015 10:31 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 2 | |||
| 2 | |||
| 0 | |||
| 0 |
03-01-2016
03:13 PM
The search I posted in my answer below seems to show that the _indextime field doesn't mean that the particular event is available for search. For a couple of our base searches, the _indextime is never more than 2-3 seconds after the event time, but the most recent event available to a particular search can be 30+ seconds old.
... View more
03-01-2016
03:09 PM
1 Karma
I came up with a search that shows _indextime almost certainly does not mean "available for search".
The search uses the addinfo command to get the time when the search was executed, then compares that time to the most recent available event's _indextime as well as the most recent event's event time (_time).
Here is the search to tack on to the end of some base search you are interested in:
YOUR_SEARCH_HERE
| eval lag_sec=_indextime-_time |
stats max(lag_sec) as max_lag max(_indextime) as max_index_time max(_time) as max_event_time
| addinfo
| eval index_lag_for_search = info_search_time - max_index_time
| eval event_lag_for_search = info_search_time - max_event_time
| table info_search_time info_min_time info_max_time, max_event_time, max_index_time, max_lag, index_lag_for_search, event_lag_for_search
For the timeframe, set it to be small and recent, earliest = -1m and latest = now, so you are sure to get the most recent event data available. The base search should be against a high frequency data source (in my case the events are written at least once per second to the index involved).
For my base search, I found that the maximum indexing lag never exceeds 2-3 seconds for any of the events in the index, but the "event_lag_for_search" measurement, which shows the most recent event available for the search can sometimes fall behind by 30+ seconds.
@gsumner , it seems like this can help answer at least one of your questions:
Is it when the first searchable copy is written? NO, _indextime does not seem to mean that the event is searchable/available for search
... View more
01-14-2016
08:12 AM
Unfortunately, there isn't a "rewrite_all" option for an Amazon ELB like there is for an F5 LB.
... View more
11-17-2015
10:31 AM
2 Karma
We are running 6.2.3 and are using search head clustering.
We would like to use an AWS ELB to terminate SSL, and then send the data to port 8000 on the search head nodes. The problem is that Splunk Web will redirect to HTTP and URLs in the UI will be plain HTTP and not HTTPs.
Is it possible to configure Splunk so that it is aware it is behind an SSL proxy?
... View more
09-19-2015
02:10 PM
Not that unusual. A few ideas.
Check out search macros - this could help if the searches are large, complex and similar. You can define a couple of macros that help with your metadata looksups and use them:
http://docs.splunk.com/Documentation/Splunk/6.2.5/Search/Usesearchmacros
Next, you can define one large crazy search at the top of the dashboard and control the reporting in sub-searches:
http://docs.splunk.com/Documentation/Splunk/6.2.5/AdvancedDev/PostProcess
I don't see why exporting the data to another system would help in the case. I'm not sure exactly what you are doing, but having used Oracle CEP and Streambase CEP, I don't see that as making it any easier to manage the data/dashboards/reports.
... View more
09-10-2015
06:17 AM
No, not crazy. We did however try to install it directly on a search head and that didn't seem to work either.
... View more
09-10-2015
06:16 AM
For the search in splunk, it is filtering by the sourcetype=kafka and search all time for it in the index (currently sending it to main).
We haven't tried to wireshark it to see if it is passing any data. We can see that it is connected at the network socket level but can try to do this.
Is there a way to set the JAVA process to log more DEBUG information?
... View more
09-09-2015
06:56 AM
Any ideas on how to troubleshoot this further? I tried the setup from the start with new installation, still seems to be connected but not actually reading data.
... View more
08-31-2015
06:27 AM
[kafka://logging_spark_bdap_dev]
group_id = splunk_dev
index = main
sourcetype = kafka
topic_name = logging_spark_bdap_dev
zookeeper_connect_chroot = kafka-development
zookeeper_connect_host = zookeeper-1.x.x.x
zookeeper_connect_port = 2181
zookeeper_session_timeout_ms = 5000
... View more
08-28-2015
10:35 AM
2 Karma
We have followed the troubleshooting steps, but are still not able to get this input working:
1) JAVA_HOME is set and java is in the path (for openJDK)
2) Splunk 6.2.3
3) Java OpenJDK, 1.7.0_85
4) Kafka version 0.8.1.1.
5) on linux
6) the only errors in splunkd.log are the SLF4J errors shown below
7) Running the command line invocation for the scheme doesn't show any errors, just prints out what the arguments are and their descriptions.
Additional troubleshooting performed:
On both the splunk forwarder host and the host running Kafka, we used netstat -anp to verify that that the Java process on splunk was connected to kafka, and we could see an ESTABLISHED socket.
Otherwise, all we see in splunkd.log is shown below. Is there a way to enable DEBUG or further troubleshooting information for the Java process/kafka consumer?
Log messages:
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Defaulting to no-operation (NOP) logger implementation
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
08-28-2015 17:26:11.298 +0000 INFO ExecProcessor - interval: run once
08-28-2015 17:26:11.298 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py
... View more
08-27-2015
02:02 PM
So should we be using the OpenJDK or the Oracle one?
Currently using the openJDK.
Been following the troubleshooting steps:
1) JAVA_HOME is set and java is in the path (for openJDK)
2) Splunk 6.2.3
3) Java OpenJDK, 1.7.0_85
4) Kafka version 0.8.1.1.
5) on linux
6) the only errors in splunkd.log are the SLF4J errors
7) Running the command line invocation for the scheme doesn't show any errors, just prints out what the arguments are and their descriptions.
I can see the Java process running on the system, not sure how to tell if its doing anything.
... View more
08-27-2015
01:40 PM
What is the tested JDK vendor for this input, Oracle JDK or does OpenJDK work?
When I first tried using the input, I was using the OpenJDK, but got some library errors:
ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Defaulting to no-operation (NOP) logger implementation
ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
When I tried to install the Oracle JDK, I started to get this error:
08-27-2015 20:31:26.244 +0000 ERROR ModularInputs - Unable to initialize modular input "kafka" defined inside the app "kafka_ta": Introspecting scheme=kafka: script running failed (exited with code 1).
Which should I be trying to make work?
... View more
08-27-2015
10:46 AM
Is the Kafka Messaging Modular Input add-on compatible with search head clustering?
... View more
06-23-2015
12:58 PM
Are you using a self-signed cert? Keystore up to date?
... View more
06-23-2015
11:44 AM
What host are you trying to access? Localhost or somewhere else? Line 24
Do you get an error message when you try to use this class?
... View more
06-23-2015
11:42 AM
1 Karma
Use the delta command:
index=main pdweb | delta pdweb.sescache as hits | timechart last(hits)
http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/Delta
... View more
06-23-2015
10:40 AM
That looks good. If you are putting slow, cheap storage in make sure you test it out at full load and trigger a bucket roll to cold and make sure the whole thing doesn't lock up.
http://wiki.splunk.com/Things_I_wish_I_knew_then
"Think twice before putting warm and cold storage on separate file systems. While it may be tempting to put warm data on faster storage, as soon as you put cold data on separate storage moving buckets from warm to cold will become much more expensive I/O-wise because all data will now have to be read from the warm storage and then written to the cold storage, instead of the simple and quick inode move operation that can be used when both warm and cold are on the same storage. "
... View more
06-23-2015
10:23 AM
2 Karma
If you aren't using useACK=true in your outputs.conf, that may be why. They were in flight before the forwarder detected the receiver was down.
... View more
06-23-2015
10:18 AM
I would use a subsearch here and the lookup table. You could also just list the users out as well.
Assuming you base search is for example, "index=www login", and you loaded your list of users into lookup "listofyourusers"
index=www login | search NOT [ | inputlookup listofyourusers]
... View more
06-23-2015
10:18 AM
I think that error is from when you are hitting splunk web and not splunkd. Make sure you are going against the proper hostname and especially port. Splunk web, on port 80 or 443 or 8000 isn't the same as talking to splunkd on port 8089.
http://dev.splunk.com/view/python-sdk/SP-CAAAEFC
... View more
06-23-2015
10:17 AM
1 Karma
Add "Truncate" to your props.conf:
[tcp://558]
connection_host = dns
index = myIndex
sourcetype = json
TRUNCATE = 16000
http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Propsconf
... View more
06-23-2015
10:08 AM
Should be the Splunk management port, 8089
http://dev.splunk.com/view/python-sdk/SP-CAAAEFC
... View more
06-23-2015
09:59 AM
Make sure you are using the right port for your instance.
... View more
06-23-2015
09:57 AM
Usually it is some problem with sendmail.
Did you check the python.log source too, usually that does yield other clues, like this one showing diskspace being exhausted:
/opt/splunk/var/log/splunk/python.log.1:2014-06-19 17:41:10,702 -0400 ERROR sendemail:112 - Sending email. subject
/opt/splunk/var/log/splunk/python.log.1:2014-06-19 17:41:10,702 -0400 ERROR sendemail:348 - (452, '4.4.5 Insufficient disk space; try again later', u'splunk@splunk-search-head-2.x.x.x') while sending mail to: x@x.com
If that doesn't show anything, the next troubleshooting step is to run sendmail from the command line as the splunk user: /usr/sbin/sendmail -f testfrom@yourdomain.com testfto@yourdomain.com
This will show any possible errors. If you are in linux, you can also check /var/log/maillog
... View more
06-23-2015
07:12 AM
I think that error is from when you are hitting splunk web and not splunkd. Make sure you are going against the proper hostname and especially port. Splunk web, on port 80 or 443 or 8000 isn't the same as talking to splunkd on port 8089.
http://dev.splunk.com/view/python-sdk/SP-CAAAEFC
... View more
