Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About tsunamii
tsunamii
Path Finder
Member since:
08-25-2011
06-05-2020
Community Statistics
| Posts | 35 |
| Solutions | 0 |
| Karma Given | 13 |
| Karma Received | 20 |
| Member Since | 08-25-2011 |
Activity Feed
- Karma Re: How to improve Hunk performance when accessing Hive tables with many small orc files? for rdagan_splunk. 06-05-2020 12:48 AM
- Karma Re: What are alternatives to using the join command for my search? for sideview. 06-05-2020 12:48 AM
- Got Karma for Splunk Universal Forwarder and TCP Data: What exactly is Splunk looking for to determine EOF?. 06-05-2020 12:48 AM
- Got Karma for Splunk Universal Forwarder and TCP Data: What exactly is Splunk looking for to determine EOF?. 06-05-2020 12:48 AM
- Got Karma for Splunk Universal Forwarder and TCP Data: What exactly is Splunk looking for to determine EOF?. 06-05-2020 12:48 AM
- Got Karma for What are alternatives to using the join command for my search?. 06-05-2020 12:48 AM
- Got Karma for What are alternatives to using the join command for my search?. 06-05-2020 12:48 AM
- Got Karma for What are alternatives to using the join command for my search?. 06-05-2020 12:48 AM
- Got Karma for srchFilter Role Inheritance: If a user is a member of two roles, how to have the srchFilter apply to indexes of one role, but not the other?. 06-05-2020 12:48 AM
- Karma What Hunk search language command generates a map reduce job in hadoop cluster? for splunkIT. 06-05-2020 12:47 AM
- Karma Re: Can slow search head pool disk i/o lead to "Splunkd daemon is not responding" errors? for splunkIT. 06-05-2020 12:47 AM
- Karma Re: What is the curl command used on the deployer to apply shcluster-bundle? for jreuter_splunk. 06-05-2020 12:47 AM
- Karma Re: How to display search DEBUG messages in Job Inspector UI? for splunkIT. 06-05-2020 12:47 AM
- Karma Re: ERP.hive-tt - Exception in thread "main" java.lang.OutOfMemoryError: Java heap space for rdagan_splunk. 06-05-2020 12:47 AM
- Got Karma for How to display search DEBUG messages in Job Inspector UI?. 06-05-2020 12:47 AM
- Got Karma for Splunk archive app: need advice on script to cleanup Hadoop data. 06-05-2020 12:47 AM
- Got Karma for How to display search DEBUG messages in Job Inspector UI?. 06-05-2020 12:47 AM
- Got Karma for How to display search DEBUG messages in Job Inspector UI?. 06-05-2020 12:47 AM
- Got Karma for Why are we getting "Invalid value =" for parameter='INDEXED_EXTRACTIONS'" in the search.log when running a search using Hunk?. 06-05-2020 12:47 AM
- Got Karma for What is the curl command used on the deployer to apply shcluster-bundle?. 06-05-2020 12:47 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 1 | |||
| 0 | |||
| 3 | |||
| 3 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 1 |
05-16-2016
01:38 PM
We are seeing excessively slow performance when accessing Hive tables with many small orc files.
We are looking for ways to improve performance. From what we can see, Hunk is causing Hadoop to create many thousands of mappers. This is because each individual orc file is causing Hadoop to create a unique search job for each file. It can take hours to even get one panel on a dashboard to populate.
Does Hunk use the CombineFileInputFormat API? This seems it would allow for relief of the number of Mappers that are generated in order to complete a search.
... View more
05-09-2016
10:07 AM
We have a user who has created a large csv lookup file (600 Mb). It seems that this file is being replicated to every Yarn server to it's /tmp directory (/tmp/splunk/nodename/splunk/var/run/searchpeers) apparently as every search is executed. There are multiple copies of this bundle. This is filling the /tmp directory and causing a major problem, it also slows every search as this file has to be copied before the search begins to execute.
We have specified the following in the hunk servers distsearch.conf to no effect:
[replicationBlacklist]
Everything = Servers.csv
How do we block this file being copied with every search?
Can the file be moved into an HDFS directory instead?
Can it be cached so that it doesn't need to be replicated with every search?
... View more
For example, I have a user (test_user) that is a member of these two roles:
[role_role_a]
importRoles = user
srchIndexesAllowed = _audit;_internal
srchIndexesDefault = _audit;_internal
srchMaxTime = 0
[role_role_b]
cumulativeRTSrchJobsQuota = 0
cumulativeSrchJobsQuota = 0
importRoles = user
srchDiskQuota = 130
srchFilter = (ConfigVersionId=86 (Address=*))
srchIndexesDefault = iselogs
srchMaxTime = 0
This works well, and when this user runs the search, the srchFilter from role_b appears to work:
litsearch ( index=* ) ( ( ( ConfigVersionId=86 ( Address=* ) ) ) )
However, what I would like is only to have the srchFilter be applied to the indexes of role_b, but not role_a (ie. the filtering should not be applying to both _audit and _internal indexes).
I have tried setting srchFilter = * in role_a:
[role_role_a]
importRoles = user
srchFilter = *
srchIndexesAllowed = _audit;_internal
srchIndexesDefault = _audit;_internal
srchMaxTime = 0
But that seems to override the srchFilter for role_b.
... View more
04-27-2016
10:27 AM
At the moment, we have worked around the issue by configured our app to pause every 15 sec so that UF will send done key. We have also disabled forceTimebasedAutoLB.
Btw, fyi: according to splunk support, if a sourcetype is set, the tcp client should not need to pause for some time. Tcp client can send data to forwarder continuously. When the "sourcetype" is not set correctly, there could be quite some event loss. If you have sourcetype set, but is still experiencing event drops, please contact splunk support and refer to SPL-117189.
... View more
04-13-2016
09:34 AM
According to the folks at Hurricane Labs, the above method/steps should work:
https://vimeo.com/139620964
... View more
04-13-2016
09:10 AM
We want to upgrade a splunk universal forwarder to heavy forwarder and also it should have the same fishbucket the universal forwarder contains.
After upgrade the heavy forwarder should start reading data from point where universal forwarder stopped.
Would the following method work?
Fresh install of the splunk heavy forwarder
Copy fishbucket database and persistentstorage ($SPLUNK_HOME/var/lib/splunk/*) from UF into same location of newly installed heavy forwarder.
Copy inputs.conf and outputs.conf from UF into HWF
Deploy necessary props.conf and transforms.conf to HWF
Biggest concern is if we can copy the fishbucket database from UF to the HWF.
... View more
04-05-2016
03:26 PM
Hi Splunkers,
Is there a way to disable a search peer via the CLI or an API call?
Specifically, I would like to set this param via CLI or REST API, and without having to restart splunk:
# distsearch.conf
disabled_servers = <comma separated list of servers>
* A list of configured but disabled search peers.
Thanks.
... View more
04-04-2016
07:48 PM
3 Karma
According to the doc here:
http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/Setuploadbalancingd
Important: Universal forwarders are not able to switch indexers when monitoring TCP network streams of data (including Syslog) unless an EOF is reached or an indexer goes down, at which point the forwarder will switch to the next indexer in the list. Because the universal forwarder does not parse the data and identify event boundaries before forwarding the data to the indexer (unlike a heavy forwarder), it has no way of knowing when it's safe to switch to the next indexer unless it receives an EOF.
We would like to know what exactly is Splunk UF looking for to determine EOF?
Additional info:
Currently, our app sending to the UF's TCP port does not use an EOF marker. This causes the UF to send data to the same indexer since it cannot switch to another indexer. As a result, we set forceTimebasedAutoLB=true to force the UF to switch indexers. However, our tests show that the UF fails to send events when this is set. For example, the following configuration:
autoLB = true
autoLBFrequency = 5
forceTimebasedAutoLB = true
results in approximately 80% event loss when sending events received via TCP port at a rate of 1 event per second. From our testing:
autoLB=F forceTimebasedAutoLB=F -> okay
autoLB=F forceTimebasedAutoLB=T -> dropped events
autoLB=T forceTimebasedAutoLB=T -> dropped events
autoLB=T forceTimebasedAutoLB=F -> okay
... View more
04-01-2016
04:22 PM
Thank you very much @sideview, for the detailed information and advice.
... View more
04-01-2016
03:41 PM
3 Karma
Hello Splunkers, I would like to seek advice on how to achieve the same goal without having to use the join command. This is the current search, with the use of the join command:
index=myidx sourcetype=myapp_log status=startSecurityUpgrade earliest=-6h@h latest=now | eval Start_Time=strftime(_time, "%m/%d/%y %H:%M:%S")
| join host_ip
[search index=idms_sat sourcetype=myapp_log status=sendCode earliest=-6h@h latest=now | eval Enrolled_Time=strftime(_time, "%m/%d/%y %H:%M:%S") ]
| where Enrolled_Time > Start_Time | timechart span=10m count(host_ip)
The problem I have with the above search (join command) is that if the data set is large, the count would become inconsistent.
... View more
03-22-2016
10:28 AM
In the Archive dashboard, I see two panels for archiving via coldToFrozen by index, I've googled it and looked through the documentation, but don't see how to configured this for HDFS. Am I correct to assume that it's not for the HDFS or S3 archiving, but for the archiving scripts method?
... View more
03-22-2016
10:25 AM
When HUNK does its bucket pushes to HDFS, it also pushes a couple small supporting files, metadata, etc... With Hadoop's issues handling small files, I was wondering if that is something that's been looked at or not?
For using the HUNK archiving on clustered indexers, I understand that if the buckets have 2 or greater searchable copies of the data, searches against that archived data will return duplicated results, is that correct?
... View more
02-26-2016
08:39 AM
We realized recently that our Hunk 6.3.0 doesn’t seem to work with Hive (version 0.13) with Char(N) definitions.
... View more
11-02-2015
09:45 AM
1 Karma
When running a search using Hunk, we are seeing a lot of these errors listed below in search log:
10-29-2015 22:22:56.942 ERROR IndexedExtractionsConfig - Invalid value='' for parameter='INDEXED_EXTRACTIONS'.
Is this expected? If not, does it impact the search performance?
... View more
10-28-2015
06:20 PM
Thanks @jreuter. Seems to be working on my 6.2.5 SHC:
root@sup-ubuntu1-cu:~/splunk62$ curl -ku admin https://localhost:8089/services/apps/deploy -d 'target=https://sup-centos1-cu:8089&action=all&advertising=true' -X POST
Enter host password for user 'admin':
["https://xx.xx.x.xxx:8089","https://xx.xx.x.xxx:8089","https://xx.xx.x.xxx:8089"]root@sup-ubuntu1-cu:~/splunk62$
... View more
10-28-2015
05:09 PM
5 Karma
Looking for a REST equivalent of this apply shcluster-bundle -target command:
./splunk apply shcluster-bundle -target https://xx.xx.x.xxx:8089 -auth admin:changeme --no-prompt --answer-yes
I want to get the session so I do not have to have hard coded credentials, and I want it in the form of a curl command.
... View more
10-13-2015
10:24 AM
Thanks @rdagan for the suggestion.
I have found the problem:
We recently upgraded the grid package, which also updated the file hadoop-env.sh and removed the line I had commented out the export HADOOP_CLIENT_OPTS="-Xmx128m ${HADOOP_CLIENT_OPTS}" and thus I was not getting the HADOOP HEAP SIZE I thought.
I fixed that and now my job runs for a long time.
... View more
10-12-2015
10:26 AM
For example, I want to accomplish this with only one alert saved search: to send email notification to admin_a if the search returns between 10 and 100 events within 5 mins (or when the search is ran), but to run some additional scripts or send email alert to admin_a and manager, if more than 100 events returned within 5 minutes.
... View more
- Tags:
- splunk-enterprise
10-09-2015
01:54 PM
What setting should I try increasing to avoid getting this Hunk + Hive error?
10-08-2015 00:31:55.626 INFO ERP.hive-tt - Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readDiskRanges(RecordReaderImpl.java:3085)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:3194)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2796)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:3213)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:3255)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:322)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:534)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:234)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:166)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1133)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.HiveRecordReader.vixInitialize(HiveRecordReader.java:206)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.BaseSplunkRecordReader.initialize(BaseSplunkRecordReader.java:95)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.JobSubmitterInputFormat.createRecordReader(JobSubmitterInputFormat.java:66)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkBaseMapper.stream(SplunkBaseMapper.java:323)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:644)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR$SearchHandler$1.accept(SplunkMR.java:656)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR$SearchHandler$1.accept(SplunkMR.java:653)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.HiveSplitGenerator.sendSplitToAcceptor(HiveSplitGenerator.java:80)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.FileSplitGenerator.generateSplits(FileSplitGenerator.java:68)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VirtualIndex$FileSplitter.accept(VirtualIndex.java:1418)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VirtualIndex$FileSplitter.accept(VirtualIndex.java:1396)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VirtualIndex$VIXPathSpecifier.addStatus(VirtualIndex.java:576)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VirtualIndex$VIXPathSpecifier.listStatus(VirtualIndex.java:609)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VirtualIndex$Splitter.generateSplits(VirtualIndex.java:1566)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1485)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1437)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:55)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:674)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:936)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:771)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1518)
10-08-2015 00:31:55.626 INFO ERP.hive-tt - at com.splunk.mr.SplunkMR.run(SplunkMR.java:1300)
... View more
09-29-2015
03:06 PM
1 Karma
We are now using Splunk archiving. I understand that there is no mechanism to delete the Hadoop Splunk data that has been archived. I would like to write a general script for deletion based on date (e.g. might want to delete data more than 60 days.)
Here is a sample archived directory with the timestamps and identify directories to be deleted that are older than n days. There are timestamps on the directory names. Would I recurse down to the directory that has the journal.gz e.g. 1440460800_1437696000/1440460800_1437696000/db_1440973083_1439867820_1/journal.gz and then check the timestamps of 1440973083 and 1439867820 and if these were OLDER than n days ago delete the directory and files: db_1440973083_1439867820_1/journal.gz or what? Please advise.
drwx------ - splunk splunk 0 2015-09-12 21:38 /projects/splunk/archive/036F3BB1-486F-4225-B461-521174F8B918/1440460800_1437696000
drwx------ - splunk splunk 0 2015-09-12 21:38 /projects/splunk/archive/036F3BB1-486F-4225-B461-521174F8B918/1440460800_1437696000/1440460800_1437696000
drwx------ - splunk splunk 0 2015-09-12 22:38 /projects/splunk/archive/036F3BB1-486F-4225-B461-521174F8B918/1440460800_1437696000/1440460800_1437696000/db_1440973083_1439867820_1
-rw------- 3 splunk splunk 117 2015-09-12 21:38 /projects/splunk/archive/036F3BB1-486F-4225-B461-521174F8B918/1440460800_1437696000/1440460800_1437696000/db_1440973083_1439867820_1/archive.content-md5_9ff9fb525c137adf5aac9184b62a22f2.receipt
-rw------- 3 splunk splunk 0 2015-09-12 22:38 /projects/splunk/archive/036F3BB1-486F-4225-B461-521174F8B918/1440460800_1437696000/1440460800_1437696000/db_1440973083_1439867820_1/archive.valid
-rw------- 3 splunk splunk 14002 2015-09-12 21:38 /projects/splunk/archive/036F3BB1-486F-4225-B461-521174F8B918/1440460800_1437696000/1440460800_1437696000/db_1440973083_1439867820_1/bucket-metadata.seq
-rw------- 3 splunk splunk 2507794 2015-09-12 21:38 /projects/splunk/archive/036F3BB1-486F-4225-B461-521174F8B918/1440460800_1437696000/1440460800_1437696000/db_1440973083_1439867820_1/journal.gz
... View more
09-08-2015
11:31 AM
3 Karma
Looking to how to enable the message block starting with "The following messages were returned by the search subsystem:" and the DEBUG messages that follow. (I've seen INFO and WARN also displayed.)
... View more
06-24-2015
12:32 PM
RE: A map-only MR job will be submitted to Hadoop...
So Hunk will never kick-off a reducer job on the hadoop side?
... View more
11-10-2014
04:29 PM
I recently upgraded my splunk 6.0.5 on Windows 2008 successfully. However, now I cannot start splunkweb service. The splunkd service has started up fine. I do not see any error in splunkd.log or web_service.log.
I have tried "splunk start" command and splunkweb appeared to have started, but about 2 seconds later, stopped.
Symptom similar to the one described here:
http://answers.splunk.com/answers/95467/splunkweb-service-wouldnt-start.html
... View more
08-26-2014
05:23 PM
Interestingly, if I hit the REST service in question, it took almost 1 minute before I got a response from splunkd:
curl -k https://localhost:8089/servicesNS/admin/search/saved/searches -d"username=admin&password=changeme"
... View more
08-26-2014
04:59 PM
1 Karma
I have two search heads with search head pooling setup. The search head pool is on a NFS filer, and that has been working well for quite sometime. However, over the weekend, the NFS filer has failed. As a temporary solution, I have decided to elect one of the search head Linux host servers to also acts as NFS filer for the search head pool, knowing that it's probably not as fast (disk i/o wise) as the actual filer.
However, since after that change is made, my users are not able to login to splunk via splunkweb UI on both search heads. They kept getting "Splunkd daemon is not responding" message in the UI. After digging through the logs, I did find the following messages in the web_service.log:
2014-08-22 23:54:14,879 ERROR [53f83a78bb1c5284d0] __init__:468 - Socket error communicating with splunkd (error=The read operation timed out), path = /servicesNS/galens/search/saved/searches
2014-08-22 23:54:14,879 ERROR [53f83a78bb1c5284d0] decorators:379 - Splunkd daemon is not responding: ('Error connecting to /servicesNS/admin/search/saved/searches: The read operation timed out',)
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 365, in handle_exceptions
return fn(self, *a, **kw)
File " ", line 1, in
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 420, in apply_cache_headers
response = fn(self, *a, **kw)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/view.py", line 1007, in render
can_alert, searches = self.get_saved_searches(app)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/view.py", line 870, in get_saved_searches
searches = en.getEntities('saved/searches', namespace=app, search='is_visible=1 AND disabled=0', count=500, _with_new='1')
File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities
atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 469, in simpleRequest
raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))
SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /servicesNS/admin/search/saved/searches: The read operation timed out',)
Could slow search head pool be the root cause? How can I temporary workaround this issue?
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
06-05-2020
02:03 AM
|
Karma given to
