Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About thirusama
thirusama
Path Finder
Member since:
08-23-2016
07-09-2024
Community Statistics
| Posts | 29 |
| Solutions | 3 |
| Karma Given | 5 |
| Karma Received | 0 |
| Member Since | 08-23-2016 |
Activity Feed
- Karma Re: Forget Password Keyfor splunk Indexer cluster for codebuilder. 07-09-2024 11:57 AM
- Karma Re: Condition-Match and set token in Drilldown not working for akheraj_splunk. 06-05-2020 12:48 AM
- Karma Re: What is tstats and why is so much faster than stats? for skawasaki_splun. 06-05-2020 12:47 AM
- Karma Re: Ever wonder which dashboards are being used and what users are using them? for MattZerfas. 06-05-2020 12:47 AM
- Karma Re: mvexpand gives "mvexpand output will be truncated due to excessive memory usage" for marcokrueger. 06-05-2020 12:46 AM
- Posted Re: JSON extracting multiple times? on Monitoring Splunk. 08-29-2019 10:46 AM
- Posted Re: Why are JSON fields extracted and displayed twice? on Getting Data In. 08-29-2019 09:56 AM
- Posted Re: Why are JSON fields extracted and displayed twice? on Getting Data In. 08-29-2019 09:53 AM
- Posted Re: Why are JSON fields extracted and displayed twice? on Getting Data In. 08-29-2019 09:47 AM
- Posted Re: Why are JSON fields extracted and displayed twice? on Getting Data In. 08-28-2019 10:04 AM
- Posted Re: Why are JSON fields extracted and displayed twice? on Getting Data In. 08-27-2019 11:51 AM
- Posted Re: Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 04:59 PM
- Posted Re: Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 04:34 PM
- Posted Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 03:07 PM
- Tagged Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 03:07 PM
- Tagged Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 03:07 PM
- Tagged Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 03:07 PM
- Tagged Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 03:07 PM
- Tagged Why are JSON fields extracted and displayed twice? on Getting Data In. 08-26-2019 03:07 PM
- Posted Re: JSON extracting multiple times? on Monitoring Splunk. 08-26-2019 02:58 PM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
08-29-2019
10:46 AM
@vinayakwagh Please if below post helps you. We had faced similar issue and is resolved now
https://answers.splunk.com/answers/768573/why-are-json-fields-extracted-and-displayed-twice.html
... View more
08-29-2019
09:56 AM
We ended up doing below which works the way we want i.e. no duplicate json values.
On UF, do NOT define any props.
On Indexers, nothing specific to JSON props, but we had defined props related time field
[my_sourcetype]
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=14000
TIME_PREFIX=timestamp":"?
On SH, do NOT define any props
With this set up, The JSON values are by default extracted in Indexing layer. Because on Indexers, this property is set up in system/default location.
[default]
AUTO_KV_JSON = true
... View more
08-29-2019
09:53 AM
We ended up doing below which works the way we want i.e. no duplicate json values.
On UF, do NOT define any props.
On Indexers, nothing specific to JSON props, but we had defined props related time field
[my_sourcetype]
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=14000
TIME_PREFIX=timestamp":"?
On SH, do NOT define any props
With this set up, The JSON values are by default extracted in Indexing layer. Because on Indexers, this property is set up in system/default location.
[default]
AUTO_KV_JSON = true
... View more
08-29-2019
09:47 AM
I checked that on SH.
... View more
08-28-2019
10:04 AM
Thanks for your response.
I checked above steps and props are coming/used from where I defined. They are same as what you mentioned in step-4. Still same issue.
$/opt/splunk/bin/splunk btool props list --debug | grep "my_sourcetype"
/data/splunk/etc/apps/my_app/local/props.conf [my_sourcetype]
... View more
08-27-2019
11:51 AM
Anyone has anymore clues as how to debug this?. I have also run the query on CM, there also I see the duplicate JSON values.
... View more
08-26-2019
04:59 PM
It shows this
[root@lgpbdus4101 bin]# ./splunk btool props list --debug my_sourcetype
/data/splunk/etc/apps/my_app/local/props.conf [my_sourcetype]
/data/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/data/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/data/splunk/etc/apps/my_app/local/props.conf AUTO_KV_JSON = false
/data/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/data/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/data/splunk/etc/system/default/props.conf CHARSET = UTF-8
/data/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/data/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/data/splunk/etc/system/default/props.conf HEADER_MODE =
/data/splunk/etc/apps/my_app/local/props.conf KV_MODE = none
/data/splunk/etc/system/default/props.conf LEARN_MODEL = true
/data/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/data/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/data/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/data/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/data/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/data/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/data/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/data/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/data/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/data/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/data/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/data/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/data/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/data/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/data/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/data/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/data/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/data/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/data/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = True
/data/splunk/etc/system/default/props.conf TRANSFORMS =
/data/splunk/etc/system/default/props.conf TRUNCATE = 10000
/data/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/data/splunk/etc/system/default/props.conf maxDist = 100
/data/splunk/etc/system/default/props.conf priority =
/data/splunk/etc/system/default/props.conf sourcetype =
... View more
08-26-2019
04:34 PM
Correct. I tried below as well on SH.
[my_sourcetype]
KV_MODE=none
AUTO_KV_JSON = false
... View more
08-26-2019
03:07 PM
JSON fields are extracted twice.
On Universal forwarder (7.0.3) the settings props.conf are like this
[my_sourcetype]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=timestamp
On Search Head(7.2.6), tried all combinations of below in props.conf
[my_sourcetype]
INDEXED_EXTRACTIONS=json
KV_MODE=none
AUTO_KV_JSON = false
... View more
08-26-2019
02:52 PM
We have Similar issue (json fields are extracted twice)
On Universal forwarder (7.0.3) the settings are like this
[my_sourcetype]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=timestamp
On Search Head(7.2.6), tried all combinations of below
[my_sourcetype]
INDEXED_EXTRACTIONS=json
KV_MODE=none
AUTO_KV_JSON = false
Does anyone have a working solution? Also when we apply props on SH member, do we have to restart Splunk on it? We just did _debug/refresh.
... View more
02-01-2019
09:47 AM
Have the wildcards for the file name. But I think Splunk adds the WATCH on that path, which means it might look for sub-directories by default? Anyways, we will add the setting recursive = false and monitor for a few days.
... View more
01-29-2019
09:34 AM
Yeah, sure. Just 22 files in the monitored directory with the name nodemanager. There is a sub-directory and it contains around 57 sub-directories, but the name does not contain nodemanager. So probably around 57*5*3=855 unwanted files.
Should we try adding recursive = false , just to avoid scanning sub-directory?
... View more
01-29-2019
09:00 AM
@witski But we are already on forwarder version 7.0.3. The issue still seem to exist. Are we missing anything?
... View more
01-29-2019
08:52 AM
Yes, I agree. For this exact reason, we had put to monitor all files with additional setting of ignoring older files (>2h). We had seen issue of missing last few events active file, before getting rolled in the past.
... View more
01-28-2019
03:25 PM
No. We monitor around 500+ nodes/hosts. Each node will have 20 (archived .log.*)+1(latest .log)+1(latest .out) i.e. total of 22 files in each node/host. We still feel that there should be a straightforward setting/solution for this. It will be very difficult to have work around(soft links) on 500+ nodes that too convincing other team.
The problem occurs on around 5-10 nodes each day.
... View more
01-28-2019
02:25 PM
So in this case, will there be a delay of 5 minutes? or may be not? Please clarify. We will have to check/work with different team to put this cron on 500+ nodes
... View more
01-28-2019
02:18 PM
Okay. Wanted to see if it helps to reduce load on Forwarder as there are 20 files archived.
Also I forgot to mention that there is other file with extension .out yarn-mapr-nodemanager-host_name.out , which seem to be ingesting fine under the same sourcetype, when the other file(s) has issue.
... View more
01-28-2019
01:28 PM
We noticed that, right after a log rotation, the data is not being indexed until the next log rotation. That is, lets say, one file was rotated at 8 AM (until which the data was already indexed). The next file is written from 8 AM to 7 PM. But this file is not indexed until around 7 PM.
We are on a Universal forwarder 7.0.3
Below is the monitoring stanza
[monitor:///opt/mapr/hadoop/hadoop/logs/*nodemanager*]
sourcetype = my_st
index = my_index
disabled = 0
ignoreOlderThan = 2h
We added ignoreOlderThan = 2h recently to see if it helps. But the issue still persists.
The latest file will be with yarn-mapr-nodemanager-host_name.log and the latest archived file be with yarn-mapr-nodemanager-host_name.log.1 .
What is interesting is intermittently on certain servers, the current file gets indexed only at the time of its roll/archival i.e. (lets say after 10-11 hours) but with actual file name but not archive file name. And the issue of live/current file not getting indexed on time does not happen all the time. The next live file might get indexed on time. There should be an ideal settings to avoid this.
Any insights on this will be helpful.
Whatever Splunk says about handling log rotation files, seems to have some bug. Are we missing anything here? Please suggest.
... View more
01-28-2019
01:10 PM
We are on forwarder 7.0.3. We seem to have similar issue. What should be the ideal fix from Splunk monitoring side?
Below is the monitoring stanza
[monitor:///opt/mapr/hadoop/hadoop/logs/nodemanager]
sourcetype = my_st
index = my_index
disabled = 0
ignoreOlderThan = 2h
The latest file will be with yarn-mapr-nodemanager-host_name.log and the latest archived file be with yarn-mapr-nodemanager-host_name.log.1 .
What is interesting is intermittently on certain servers, the current file gets indexed only at the time of its roll/archival i.e. lets say after 6 hours. And the issue of live/current file not getting indexed on time does not happen all the time. The next live file might get indexed on time. To me, it sounds like what @sjohnson_splunk has mentioned. But there should be ideal settings to avoid this. Any insights on this will be helpful.
... View more
10-25-2018
10:28 AM
We found out the root cause for this. Looks like Splunk_TA_jmx app requires port 8089 to be enabled. In one of our server.conf , below configuration is enabled. After commenting this out, JMX data has started coming in. We had to also install latest version of app (3.2.0).
[httpServer]
disableDefaultPort = true
We found this bu running below command on the working and non-working nodes.
sudo netstat -tulpn | grep 8089
... View more
10-23-2018
02:50 PM
Does anyone know anything about this? We are still getting these errors on some of the Universal Forwarders
... View more
10-24-2017
10:48 AM
Any help on this is highly appreciated.
... View more
10-16-2017
04:27 PM
JMX monitoring stopped working on 4 of our VMs, where as the other servers (around 100) are still working. There was an upgrade of OS on all of these machines along with Java upgrade. Nothing seem different from working ones to non-working ones.
We are using config file & this is how our inputs.conf looks like
[jmx://jmx]
config_file = config.xml
_TCP_ROUTING = myindexset
polling_frequency = 60
sourcetype = jmx
index = my_index_jmx
disabled = 0
interval = 60
Below are the errors/messages, we are getting. Did anyone face similar issue?
Taking one non-working host as as example, it stopped receiving data since "9/30/17 1:25:11.542 AM".
This is from "/opt/splunkforwarder/var/log/splunk/jmx.log"
2017-09-30 01:25:49,924 - com.splunk.modinput.ModularInput -159035 [Thread-1] ERROR - Probing socket connection to SplunkD failed.Eith
er SplunkD has exited ,or if not, check that your DNS configuration is resolving your system's hostname (127.0.0.1) correctly : Connection refused
2017-09-30 01:25:57,735 - com.splunk.modinput.ModularInput -166846 [main] ERROR - Error executing modular input : Connection refused : java.lang.RuntimeException: Connection refused
at com.splunk.HttpService.send(Unknown Source)
at com.splunk.Service.send(Unknown Source)
at com.splunk.HttpService.get(Unknown Source)
at com.splunk.ResourceCollection.list(Unknown Source)
at com.splunk.ResourceCollection.refresh(Unknown Source)
at com.splunk.ResourceCollection.refresh(Unknown Source)
at com.splunk.Resource.validate(Unknown Source)
at com.splunk.ResourceCollection.validate(Unknown Source)
at com.splunk.ResourceCollection.values(Unknown Source)
at com.splunk.jmx.InfoManager.getAccounts(Unknown Source)
at com.splunk.jmx.JMXModularInputV3.doRun(Unknown Source)
at com.splunk.modinput.ModularInput.init(Unknown Source)
at com.splunk.jmx.JMXModularInputV3.main(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:160)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
... 13 more
Below is from splunkd.log
09-30-2017 01:25:44.479 -0700 INFO WatchedFile - Will use tracking rule=modtime for file='/etc/alternatives/java_sdk_oracle/lib/missioncontrol/plugins/com.jrockit.mc.rjmx_5.5.1.172852/plugin.properties'.
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py" Error executing modular input : Connection refused : java.lang.RuntimeException: Connection refused
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.HttpService.send(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.Service.send(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.HttpService.get(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.ResourceCollection.list(Unknown Source)
java -version on both WORKING & NON-WORKING VM
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
We tried to compare everything, tried re-push the app from deployment server. No luck.
A few things we validated on non-working VM
1) port is open to the localhost
2) port is configured in JVM/JMX
3) JMX metrics are coming fine in the JConsole
Please let us know if anyone faced similar issue and were able to fix.
... View more
12-19-2016
01:38 PM
Sure. Let me try it out. Will get back.
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
07-09-2024
07:33 PM
|
Karma given to
| User | Karma Count |
|---|---|
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 |
