Awesomee..!! I understood now. Thankyou so much @kpawar Do we use Hadoop Data roll for sending data to EMR Hadoop as well? Also as @ByteFlinger asked - Is there no way to access S3 bucket rather than using access and secret key? ... View more

Hi @Anonymous , We have a clustered Indexer , so If I use coldtofrozen.py script , it would copy redundant data as well, right? Thus by using Hadoop Data Roll I can overcome this redundancy of data copied. I am not using any Hadoop here , I want to move it to S3.... But I still didnt understand what these below HADOOP parameters refer to? The doc says the below: Add the following configuration to your providers: Hadoop Home: /absolute/path/to/apache/hadoop-2.6.0 vix.fs.s3a.access.key: vix.fs.s3a.secret.key: vix.env.HADOOP_TOOLS: $HADOOP_HOME/share/hadoop/tools/lib vix.splunk.jars: $HADOOP_TOOLS/hadoop-aws-2.6.0.jar,$HADOOP_TOOLS/aws-java-sdk-1.7.4.jar,$HADOOP_TOOLS/jackson-databind-2.2.3.jar,$HADOOP_TOOLS/jackson-core-2.2.3.jar,$HADOOP_TOOLS/jackson-annotations-2.2.3.jar ... View more

Hi @ awurster , We have a similar goal of moving data to S3. I was going through your link and had few queries. https://bitbucket.org/asecurityteam/atlassian-add-on-cold-to-frozen-s3/overview Would be really helpfull if you can clarify please as I'm new to AWS. 1) What does the below "launching splunk with an instance profile" mean ? Where do I configure this? Usage Ensure your Splunk indexers are launched with an instance profile which permits uploading to the S3 bucket. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::example-bucket" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject" ], "Resource": [ "arn:aws:s3:::example-bucket/*" ] } ] } 2) Can your script be implemented in a clustered Splunk architecture as well?? ... View more

Hi @nawneel , That worked..!! Thankyou 🙂 ... View more

Hi @MuS , Is there a way to remove the list of splunk's default apps like "Home , learned , Search & Reporting etc ... I want the list of the apps deployed by me via the deployer.. ... View more

Im also getting this: DBX Server is not available, please make sure it is started and listening on 9998 port or consult documentation for details. ... View more

@sansay , Could you please let me know wht this actually means if you are aware of it now? apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME' ... View more

We are using 6.3.7 and I was trying to save it as a report. ... View more

@jkat54 , I'm saving via UI. Did you get to figure the root cause for this? ... View more

@rjthibod I dont get any output for that command. sh-4.1$ ./splunk btool check sh-4.1$ ... View more

I'm getting same error although no duplicate name exists. ... View more

This blog was really informative..Thanks a lot..!! @rdagan 🙂 ... View more

Ohhh...Let me try to resolve it first then..Thankyou for the info... Also it would be great if you could help me on thsi issue as well 😞 https://answers.splunk.com/answers/482063/splunk-add-on-for-kafka-jmxlog-how-to-resolve-jmx.html ... View more

hI @rdagan , Sorry to bug you again. So my understanding is that - " Hunk/Splunk Analytics for Hadoop " requires additional license for analysing Hadoop data ( Hunk license depends on no. of Hadoop nodes) . 1.My goal is to analyse Hadoop data and neither import or export data.. My understanding is that Hadoop connect just explores the files/directories using search commands and cannot be used for analysis , reporting etc unless the data is imported from Hadoop to Splunk Indexers. Where as Hunk can perform this without indexing the data via Virtual Indexes? Am I right? 2.My MAIN doubt is When they say Hadoop Connect can "Explore and browse HDFS directories and files" --> what all features can I achieve via this? Just view the contents of HDFS files, is it ? Is searching restricted here and in what way? I can use stats,chart,timechart..commands for hadoop data using a Hadoop connect ? ... View more

Hi @rdagan , That was really informative. Thankyou so much for your help..!! 🙂 ... View more

Thanks a lot for the input..!! Understood now.. ... View more

Hi @rdagan , Thankyou so much for guiding me. Actually None of the commands worked on the SH command line. But when I tried on the Hadoop cluster, I was getting below WARNINGS. So I deleted these corrupt blocks and re-added data to datanodes. But are you aware of why blocks get corrupt in Hadoop. I know this is out of splunk scope, but asking out of curiosity. 16/12/20 02:21:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/12/20 02:21:17 INFO hdfs.DFSClient: No node available for BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 file=/data/input/splunk/linux2/LICENSE.txt 16/12/20 02:21:17 INFO hdfs.DFSClient: Could not obtain BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 from any node: java.io.IOException: No live nodes contain block BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 after checking nodes = [], ignoredNodes = null No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry... 16/12/20 02:21:17 WARN hdfs.DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 127.55817333292474 msec. 16/12/20 02:21:17 INFO hdfs.DFSClient: No node available for BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 file=/data/input/splunk/linux2/LICENSE.txt 16/12/20 02:21:17 INFO hdfs.DFSClient: Could not obtain BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 from any node: java.io.IOException: No live nodes contain block BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 after checking nodes = [], ignoredNodes = null No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry... 16/12/20 02:21:17 WARN hdfs.DFSClient: DFS chooseDataNode: got # 2 IOException, will wait for 7945.315192663103 msec. 16/12/20 02:21:25 INFO hdfs.DFSClient: No node available for BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 file=/data/input/splunk/linux2/LICENSE.txt 16/12/20 02:21:25 INFO hdfs.DFSClient: Could not obtain BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 from any node: java.io.IOException: No live nodes contain block BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 after checking nodes = [], ignoredNodes = null No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry... 16/12/20 02:21:25 WARN hdfs.DFSClient: DFS chooseDataNode: got # 3 IOException, will wait for 6954.636509794202 msec. 16/12/20 02:21:32 WARN hdfs.DFSClient: Could not obtain block: BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 file=/data/input/splunk/linux2/LICENSE.txt No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException 16/12/20 02:21:32 WARN hdfs.DFSClient: Could not obtain block: BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 file=/data/input/splunk/linux2/LICENSE.txt No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException 16/12/20 02:21:32 WARN hdfs.DFSClient: DFS Read org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1826813176-10.109.137.83-1480588903531:blk_1073741842_1018 file=/data/input/splunk/linux2/LICENSE.txt ... View more

Hi @rdagan , I'm facing the same error. Moreover Im using a single node cluster in DEV and my block does exit . 2016-12-19 02:30:33,798 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-1826813176-10.109.137.83-1480588903531 Total blocks: 21, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0 But still the below error while running query: BlockMissingException - Could not obtain block: BP-1826813176-10.109.137.83-1480588903531:blk_1073741844_1020 file=/data/input/splunk/linux2/README.txt I restarted the nodes also. But I still face the error. @sbrice - Were you able to fix the error? ... View more

Hi @lguinn , Sorry I'm back to a similar doubt. In out organistaion we have 4 clustered SHs set up in Non-Prod but they have set one single common URL for this Non-prod enivironment. So my doubt here is - Which SH will this URL point to when we run a query? Will the load balancer decide and execute the query on a particular SH? Another doubt is if I need to check the logs for some troubleshooting purpose. In which Search head should I query for errors since I'm not aware which search head is executing the saved searches etc. ? ... View more

Is there a way to find out from the logs or something else, if the app was actually deleted? ... View more