Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
- About Hemnaath
Hemnaath
Motivator
Member since:
03-15-2015
11-20-2025
Community Statistics
| Posts | 903 |
| Solutions | 12 |
| Karma Given | 1 |
| Karma Received | 55 |
| Member Since | 03-15-2015 |
Activity Feed
- Posted Re: How to create heatmap visualization in dashboard studio? on Dashboards & Visualizations. 11-20-2025 09:06 PM
- Posted How to create heatmap visualization in dashboard studio? on Dashboards & Visualizations. 11-20-2025 03:56 AM
- Got Karma for How to troubleshoot and fix the website input 504 Gateway Timeout Error while configuring an input ?. 01-12-2024 02:44 PM
- Posted Re: Where I can get the Splunk Add-on for AWS 6.0.0 version documentation details on All Apps and Add-ons. 08-29-2023 03:34 AM
- Posted Where I can get the Splunk Add-on for AWS 6.0.0 version documentation details on All Apps and Add-ons. 08-28-2023 04:58 AM
- Tagged Where I can get the Splunk Add-on for AWS 6.0.0 version documentation details on All Apps and Add-ons. 08-28-2023 04:58 AM
- Got Karma for Re: Splunk has found 1 orphaned searches owned by 1 unique disabled users.Click to view the orphaned scheduled searches. Reassign them to a valid user to re-enable or alternatively disable the searches.. 06-13-2023 04:41 AM
- Posted What splunk precedence rules should be followed to upgrade splunk clustered environment ? on Splunk Enterprise. 01-26-2023 01:29 AM
- Got Karma for Unable to break the multi-line events into single event from kinesis log?. 11-07-2022 02:14 AM
- Posted Re: How to write a Splunk query to find the Splunk UF version for specific set of hosts in Splunk Enterprise on Splunk Enterprise. 10-31-2022 11:13 PM
- Posted How to write a Splunk query to find the Splunk UF version for specific set of hosts in Splunk Enterprise on Splunk Enterprise. 10-31-2022 06:44 AM
- Got Karma for Re: ERROR: kvstore port [8191] - port is already bound. Splunk needs to use this port.. 10-13-2022 02:16 PM
- Posted Where can I get the release notes for Splunk enterprise v8.2.6.1 release notes ? on Splunk Enterprise. 07-26-2022 11:17 PM
- Posted How to fix this error: could not use the strptime to parse timestamp from “2022-26-05T11:29:57”? on Getting Data In. 05-26-2022 09:01 AM
- Posted Re: How to write a monitoring stanza to monitor the windows Event Viewer logs on Getting Data In. 05-12-2022 08:59 AM
- Posted Re: How to write a monitoring stanza to monitor the windows Event Viewer logs on Getting Data In. 05-12-2022 06:14 AM
- Posted How to write a monitoring stanza to monitor the windows Event Viewer logs? on Getting Data In. 04-29-2022 03:13 AM
- Posted Re: How to write time format for 2022-04-07 20:40:03.360 -06:00 [XXX] XXXXX? on Splunk Enterprise. 04-16-2022 07:18 PM
- Posted How to write time format for 2022-04-07 20:40:03.360 -06:00 [XXX] XXXXX? on Splunk Enterprise. 04-14-2022 09:17 AM
- Posted Why is Splunk ingesting additional information from the source, when there is no actual data present in the source ? on Splunk Enterprise. 04-13-2022 05:11 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
07-13-2018
08:47 AM
Hi All,
We have been notified by the security team as they are seeing low security events counts and the logs are not parsing at realtime, based on the alert set to trigger in the splunk.
Query details
| inputlookup LOW-EVENT-COUNT-SECURITY.csv
| join type=outer sourcetype [metadata type=sourcetypes]
| eval recent=strftime(recentTime,"%m/%d/%y %H:%M:%S")
| eval last=strftime(lastTime,"%m/%d/%y %H:%M:%S")
| eval nowTime=now()
| eval now=strftime(nowTime,"%m/%d/%y %H:%M:%S")
| eval diff=(nowTime-recentTime)
| table data sourcetype recentTime recent lastTime last now nowTime diff
data sourcetype recentTime recent lastTime last now nowTime diff
F5 BIG-IP f5:bigip:syslog 1531492099 07/13/18 10:28:19 1531506498 07/13/18 14:28:18 07/13/18 10:28:22 1531492102 3
When checked for error logs in splunkd.log found below error details from the search heads
07-13-2018 11:01:30.655 -0400 ERROR DistributedBundleReplicationManager - HTTP response code 400 (HTTP/1.1 400 Bad Request). Error applying delta=/opt/splunk/var/run/searchpeers/C090FDA2-105E-4875-A110-3F13FF986151-1531493889-1531494056.delta, searchHead=C090FDA2-105E-4875-A110-3F13FF986151, prevTime=1531493889, prevChksum=4389239684162954908, curTime=1531494056, curChksum=16674274590358345014: Error copying /opt/splunk/v
Error logs in splunkd.log from Indexer instances
07-13-2018 10:32:35.344 -0400 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/C090FDA2-105E-4875-A110-3F13FF986151-1531492185.bundle
Error logs in splunkd.log from Heavy forwarder instances
a) 07-13-2018 07:01:03.262 -0400 ERROR DistributedBundleReplicationManager - Reading reply to upload: rv=-2, Receive from=https://splunk03:8089 timed out; exceeded 60sec, as per=distsearch.conf/[replicationSettings]/sendRcvTimeout
b) 07-13-2018 07:01:39.014 -0400 ERROR DistributedBundleReplicationManager - got non-200 response from peer. uri=https://splunk02:8089, reply="HTTP/1.1 204 No Content" response_code=204
c) Unable to upload bundle to peer named splunk02 with uri=https://splunk02:8089.
Questions:
1) We had recently upgraded the search head/indexer instance to 7.0.4 from 6.6.1 and but we have not upgraded the HF instances where all the data are parsing before getting indexed. whether this might be causing the issue ?
2) How/where to start the investigation and trouble this issue to find out the root cause for the low event count.
Kindly guide me to fix this issue.
... View more
07-11-2018
06:45 AM
hey pradeep thanks, how/where to find the default max lookup size in the limits.conf file. And also how to get the lookup file which has maximum size configured.
... View more
07-11-2018
06:03 AM
Hi All,
We could see the below message are popping out in the search heads, recently we have upgraded the indexer/search head instances to 7.0.4 version from 6.6.1. we have not yet upgrade the Heavy forwarder will this might be causing the issue.
Error Message:
Distributed Bundle Replication Manager: The current bundle directory contains a large lookup file that might cause bundle replication fail. The path to the directory is /opt/splunk/var/run/C090FDA2-105E-4875-A110-3F13FF986151-1531313061-1531313166.delta.
Kindly guide me to troubleshoot this issue.
... View more
07-11-2018
05:55 AM
hey woodcock, sorry i did not understand your context ? please let me know what do you mean by UpVote ?
... View more
07-10-2018
04:47 AM
Hi All, can any one throw some lights on above subjects.
thanks in advance.
... View more
07-10-2018
04:45 AM
7 Karma
Hi All,
The issue got fixed by following the below steps: Reassign unshared, orphaned knowledge objects
1) After logging into the splunk portal, we need to first get the orphaned searches name and owner details and that information can be obtained from the orphaned scheduled search reports. And you can also execute the below query manually to fetch the same result.
Query Details:
| rest timeout=600 splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0
| search orphan=1 disabled=0 is_scheduled=1
| eval status = if(disabled = 0, "enabled", "disabled")
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing
2) On getting the owner name / search name, we need to re-create a invalid owner by accessing the settings-->access control-->user (with minimum role and capabilities as the user is temporary)
3) Once the user is re-created, go to the settings - Knowledge objects --> All configuration -->Reassign Knowledge object -->Orphaned tab and select Orphaned search and assign it to the re-created user.
4) After assigning the Orphaned search, go to the settings -->Knowledge objects -->Searches, Reports and Alerts --> select the search which was re-assigned to the invalid user, in my case the report was more specify to the user and it was not important, so we had disabled the Reports.
Note: Suppose if the Report/Alerts/searches are important, then you need to re-assign to the valid user, before disabling/removing the user from splunk.
5) Once the Reports are disabled, we can delete the invalid user by accessing the settings-->access control-->user - invalid user-delete.
6) Validate whether, by executing the above search query and also whether the pop message is cleared from the portal.
Referred below splunk document.link text
... View more
06-22-2018
07:14 AM
hey I could see the same message popped up again in the search head portal and have identified the search name SEC-GRC-ALERT-OIM-Email-DirSync-Errors and the user name who had owned the search.
But the challenge is now the user got deleted from the splunk, so unable to find the particular orphaned search in the setting-->searches-reports-alerts. I had gone through the splunk documentation
link text
But unable to find after re-creating the user from settings-- access control --users.
And at the same time on the search head cluster node, could see the search details under this path /opt/splunk/etc/users/test01/local/search/savedsearches.conf. Can I # it in this location will that work. Kindly guide me how to fix this issue.
... View more
06-21-2018
10:26 AM
Hey it did not work, could you please guide me on this
USERNAME LINE HOSTNAME TIME
host = tt3crp00 source = who sourcetype = who
6/21/18
1:11:25.000 PM
USERNAME LINE HOSTNAME TIME
solarwinds pts/21 vmswpep02.XXXX.com Jun 21 13:11
host = tt3dev00 source = who sourcetype = who
6/21/18
1:11:25.000 PM
USERNAME LINE HOSTNAME TIME
solarwinds pts/0 vmswpep05.xxxx.com 2018-06-21 13:11
solarwinds pts/1 vmswpep05.xxxx.com 2018-06-21 13:11
the above props/transforms.conf are placed in the Heavy forwarder instances.
Kindly guide me to fix this issue.
... View more
06-21-2018
08:38 AM
thanks Somesoni2, let me try that.
... View more
06-21-2018
07:50 AM
Hi Somesoni2, by applying the above stanza in props.conf and transforms.conf, the entire event information are removed from splunk and it throws the below message.
No results found. Please change source type, adjust source type settings, or check your source file
Could you please guide me on this to events which does not contain any information in it.
... View more
06-21-2018
05:36 AM
Hi Dal Jeanis, thanks for your support on this, but I do not see the alert message popping out currently in the Splunk GUI. So did not test the search.
... View more
06-21-2018
03:52 AM
Hi Splunk experts,
Need to understand the basic function of these stanza's FORMAT, DEST_KEY, SOURCE_KEY and MV_ADD and where/which circumstances we need to use these in transforms.conf.
I had gone through the splunk transforms.conf documents. But still finding it difficult to understand and apply them in real time, so it will be really helpful, if any one can explain these stanza in layman language(with examples). Sorry for asking like this but really wanted to understand these things.
... View more
06-21-2018
02:40 AM
Hi SomeSoni2,
Yes I had given a try with the above set of configuration in my test environment by uploading the data in raw text format via Add data -->upload --> select sourcetype but it did not fetch the required output.
Instead getting the below message.
No results found. Please change source type, adjust source type settings, or check your source file
Props.conf details:
[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
TRANSFORMS-set= setnull,setparsing
Transforms.conf details:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
#basically keeping anything that has header and at least one data row
[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue
Kindly guide me on this.
... View more
06-20-2018
12:49 PM
Hey we are splitting multiple events in to single individual events by using the line_breaker stanza in props.conf. And also we wanted to remove the events which does not contain any information in it.
1) To split multiple events in to single individual events.
[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
2) For removing the events which does not contain any information.
Props.conf
[who]
TRANSFORMS-null= setnull
Transforms.conf details:
[setnull]
REGEX = ^USERNAME\s+LINE\s+HOSTNAME\s+TIME$
DEST_KEY = queue
FORMAT = nullQueue
Kindly guide me on this.
... View more
06-20-2018
12:33 PM
Hi cpetterborg, thanks for your effort on this, I had added the sample events containing the data, so can I use the above regex to remove events which does not contain any information in it.
... View more
06-20-2018
12:31 PM
Hi Somesoni2, thanks for your effort on this, we are monitoring wtmpx file from the unix machines using the Splunk Add-on for Unix. Below are the sample events contains the information.
Index=unix sourcetype=who host=*
6/20/18
3:06:05.000 PM
USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
host = ttbmt02 source = who sourcetype = who
6/20/18
3:03:35.000 PM
USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
... View more
06-20-2018
09:55 AM
Hi Splunk experts,
Just want to know how can I remove events which does not contain any information in it?
Example
sample events which have information on them.
I know that, we have to configure Route and Filter data in Props/transforms to achieve this but not sure about the Regex.
Props.conf details:
[who]
TRANSFORMS-null= setnull
Transforms.conf details:
[setnull]
REGEX =
DEST_KEY = queue
FORMAT = nullQueue
Kindly guide me on this
... View more
06-20-2018
07:53 AM
1 Karma
Hi All,
Got how to split the multiple event in to single individual event using the Line_Breaker stanza in props.conf
[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
[lastlog]
## Override system/default lastlog sourcetype invalidation
invalid_cause =
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
... View more
06-20-2018
05:37 AM
Any help on this issue, will be much appreciated !!!
... View more
06-20-2018
05:33 AM
Hi All, We are monitoring the wtmpx data from the Unix machines via splunk using the Splunk add-on for Unix, based on this add-on we could see the data from the wtmpx file in splunk, but currently we could see the data ingesting from the wtmpx file into splunk with multiple line like shown below.
6/20/18
6:26:03.000 AM
USERNAME LINE HOSTNAME TIME
HXXX019 pts/1 w442xty1.XXXX.com Jun 19 18:40
Hxxx007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
6/20/18
6:23:33.000 AM
USERNAME LINE HOSTNAME TIME
HXXX019 pts/1 w442xty1.XXXX.com Jun 19 18:40
HXXX007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
But instead of multiple line we want to have it as single event like this
USERNAME LINE HOSTNAME TIME
HXXX019 pts/1 w442xty1.XXXX.com Jun 19 18:40
USERNAME LINE HOSTNAME TIME
HXXX007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
And also if there is no content in the log then it should be removed from splunk.
example:
USERNAME LINE HOSTNAME TIME
host=r3crp00 source=who sourcetype=who
Props.conf details for the sourcetype=who
[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=^()$
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
sourcetype=lastlog
[lastlog]
## Override system/default lastlog sourcetype invalidation
invalid_cause =
SHOULD_LINEMERGE=false
LINE_BREAKER=^()$
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
Kindly guide me how to configure this in the props.conf file.
thanks in advance.
... View more
- Tags:
- multiple-lines
- split
06-19-2018
10:27 AM
Can any one guide me on this.
... View more
06-19-2018
04:03 AM
1 Karma
Hi All,
We are getting this pop-up message in the splunk console, based on the below link provide in the answer.com, I had tried to detect the orphaned searches in our environment but unable to get the details.
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Knowledge/Resolveorphanedsearches
Splunk version details : 6.6.1
Methods to detect the orphaned searches.
1) Look at the Orphaned Scheduled Searches, Reports, and Alerts dashboard and report
Query details:
| rest timeout=600 splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0
| search orphan=1 disabled=0 is_scheduled=1
| eval status = if(disabled = 0, "enabled", "disabled")
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing
Getting no result found.
2) Run the Monitoring Console health check
Settings--> Monitoring console --> Health Check --> start
Getting the below message details
Message
This health check item was successful. Everything is good here. All the splunk instance result are good.
3) Use the Reassign Knowledge Objects page in Settings
Settings --> All Configurations -->Reassign Knowledge objects -->orphaned to filter out the orphaned searches.
Getting no result found.
Kindly guide me how to find out which schedule search is left as orphaned in the search head cluster.
... View more
06-18-2018
06:46 AM
Hi dstaulcu, thanks and sincerely appreciate for your time and effort. great work !!!
hey do I need to upload all the file which are present in the github to the Splunk_TA_windows app folder to achieve the results. Kindly guide me on this.
... View more
06-13-2018
11:13 PM
Hi All, could any one guide me on how to troubleshoot this issue.
... View more
06-13-2018
11:12 PM
2 Karma
The above issue got fixed, on investigating the problem we found that outputs.conf file was configured in with below stanza, due to this all the data when we uploaded via splunk test portal, the data was being ingested in to production.
Steps :
1) Checked by executing the index="_internal" and found that there was no data being ingested. This showed that some thing is really going wrong as we could not see the splunk internal data.
2) Executed splunk btool command to find out outputs.conf list to check the configuration
./splunk btool outputs list --debug | less
3) We found that in one of the app, the out puts.conf was configured to route the data to the production indexers.
[tcpout]
defaultGroup = all_indexers
maxQueueSize = 1GB
[tcpout:all_indexers]
server = splunk.test.com:9997
4) we had disabled the app by editing the app.conf file in local
# Autogenerated file
[install]
state = disabled
5) Then restarted the splunk service and check whether we are getting the splunk internal data and found to be ingesting.
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
11-20-2025
08:58 PM
|
