Hi @rdagan, I did follow the same steps , installed Hadoop binaries and copied teh configs into splunk hosts. But Im unable to run simple hadoop commands on the splunk hosts. I get the below error. bash-4.1$ hdfs dfs -ls / 17/12/07 03:04:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.ExceptionInInitializerError at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2138) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2103) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2223) at org.apache.hadoop.security.Groups. (Groups.java:99) at org.apache.hadoop.security.Groups. (Groups.java:95) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:420) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:284) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:806) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:776) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:649) at org.apache.hadoop.fs.FileSystem$Cache$Key. (FileSystem.java:2874) at org.apache.hadoop.fs.FileSystem$Cache$Key. (FileSystem.java:2866) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2729) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:385) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:184) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:102) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:372) Caused by: java.lang.RuntimeException: Bailing out since native library couldn't be loaded at org.apache.hadoop.security.JniBasedUnixGroupsMapping. (JniBasedUnixGroupsMapping.java:46) ... 30 more What could be the reason? ... View more

Hi @rdagan , Thankyou for your response. But I'm confused with with point 2 and 3. If I want to send Splunk data to Hadoop via Hadoop Data Rol l and use Hadoop techniques like Hive, Spark etc as search engine , that means I DO NEED Bucket Reader ? Am I right? However If Im using Hadoop Connect for export , I DO NOT NEED Bucket Reader ? ... View more

Hi @SloshBurch , I did go through the documentation and had few queries - I have posted these queries at the below link as well: https://answers.splunk.com/answers/577310/difference-between-data-format-for-the-data-sent-t.html?minQuestionBodyLength=80 Firstly what is the difference between the data format of data that is sent via Hadoop Connect Export and Hadoop Data Roll? I believe Hadoop Connect exports search results and Hadoop Data Roll send the raw data journal.gz Can I use Hadoop Techniques like Hive, Pig..etc for analytics on the archived data sent to Hadoop via Hadoop Data Roll? AND Can I use Hadoop Techniques like Hive, Pig..etc for analytics on the archived data sent to Hadoop via Splunk Hadoop Connect Export? I came across Splunk Archive Bucket Reader - an additional app is required to analyze the archived data via Hadoop's applications like Pig , Hive , Spark. Is this a mandatory app required If I want to analyse the Hadoop data? Is Splunk Analytics for Hadoop license license required for sending archived data to Hadoop? ... View more

Thakyou for the notes. I will go through the documentation and get back incase of further queries. ... View more

Hi BUrch, The goal is to move the data from Splunk to Hadoop/S3 for longer data retention. Currently we store data in Splunk only for two months. We want to send data to Hadoop and the analytics team wants to further analyse this HIVE , etc I understand 1) We can simply forward data to another system as it arrives at Splunk or 2) we can export data after it has been cooked by Splunk. However not sure how to achieve first part. Hadoop connect allows data export to Hadoop but it doesnt seem to be supported in a clustered architecture. Thus looking out for options. I believe Hadoop Data Roll is used when I want to use Splunk again for archived data analytics, which is not needed for us. This archived index will follow the same two month retention period right? -- Please correct me if Im wrong. ... View more

Hi Burch , Since the goal was automation so I thought of editing the conf files and pushing it to SHC from the deployer. No specific reason as such. But yes , I guess REST or CLI would be a better approach , but I think they aren't supported for DBX V3 as far as I've researched. However Awaiting a response for the enhancement case that I've submitted. ... View more

Hi @garethatiag , This URL doesnt update password/username for DB Connect V3. I assume it works only for DBX V2: https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/ ... View more

Hi @garethatiag , I noticed that the password is not getting encrypted even after Splunk restart was the root cause. How does the password get encrypted if restart is not the solution? ... View more

Hi @Anonymous , Did you create DB coonection and Identity via CLI initially and then did a reload via Splunk RESTAPI? What was your approach? Im only creating a DB connection for DB Data Query and I do not want to create DB Outputs or inputs so reload inputs URL is not needed for me I think: However I did try both the URLs that you suggested , reload task-server and reload inputs but it didnt work 😞 https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs/_reload I see the connection in UI but it gives me an error as seen in screenshot: The splunk_app_db_connect_server.log shows below error: 2017-08-03 08:33:46.048 -0400 [dw-49 - GET /api/connections/SPOTFIRE60_DEV/status] ERROR io.dropwizard.jersey.errors.LoggingExceptionMapper - Error handling a request: 94204c89a78800be java.lang.IllegalArgumentException: Illegal base64 character 5f at java.util.Base64$Decoder.decode0(Base64.java:714) at java.util.Base64$Decoder.decode(Base64.java:526) at java.util.Base64$Decoder.decode(Base64.java:549) at com.splunk.dbx.crypto.AESUtils.decrypt(AESUtils.java:75) ... View more

Hi @Anonymous , I also wanted to automate the creation of DB Connection in DBX3 and I tried the below but it didnt seem to work: 1) Updated via CLI identities.conf and db_connections.conf with the new DB details 2) Did a reload on the host curl -k -u admin:password https://myhost:8089/servicesNS/nobody/splunk_app_db_connect/configs/conf-db_connections/_reload Is there anything I'm missing out? Could you please guide.. ... View more

Hi @ SloshBurch , 1)I tried creating a connection via CLI by updating identities.conf and db_connections.conf and it didn't seem to work. I would like know if DB Connection creation via CLI is possible or not at all..?? 2) Also I believe Splunk doesnt provide automation via REST API for db connections... I didnt find them in splunk docs ... View more

Hi @dwaddle , I figured out that it was my python script that was causing the problem. It was modifying the file in a fashion that we couldnt figure out all although the contents looked fine. Thankyou for your support. However I had another query, how do we push the config changes for the authentication.file via the deployer to the SHs? Doesn't shcluster apply bundle push the changes from $SPLUNK_HOME/splunk/etc/shcluster to the SHs? However the authentication.file is located at $SPLUNK_HOME/splunk/etc/system/local right? ... View more

Hi @dwaddle All I did was a script to replace the ldap account and its password. The file has got replaced with accountname and password but the password has not got encrypted after the splunk restart: auth.conf Before [lassie_np_ldap] sslenabled = 1 anonymous_referrals = 1 binddn = cn=ezp_splunk_ldap,OU=ServiceAccounts,DC=im,DC=tcsco,DC=com binddnpassword = $1$Hx+uKRbWP/f3 charset = utf8 emailattribute = mail groupbasedn = OU=FIMT_Managed,OU=Groups,DC=im,DC=tcsco,DC=com groupbasefilter = (|(cn=FIMT-LASSIE-) (cn=FIMTMON_)) groupmappingattribute = dn groupmemberattribute = member groupnameattribute = cn host = ldapserver.tcsco.com nestedgroups = 0 network_timeout = 20 port = 636 realnameattribute = displayname sizelimit = 1000 timelimit = 15 userbasedn = DC=im,DC=tcsco,DC=com userbasefilter = (objectclass=*) usernameattribute = samaccountname auth.conf After: [lassie_np_ldap] sslenabled = 1 anonymous_referrals = 1 binddn = cn=ezp_splunk_ldap_dev,OU=ServiceAccounts,DC=im,DC=tcsco,DC=com binddnpassword = N3F1vSvt charset = utf8 emailattribute = mail groupbasedn = OU=FIMT_Managed,OU=Groups,DC=im,DC=tcsco,DC=com groupbasefilter = (|(cn=FIMT-LASSIE-) (cn=FIMTMON_)) groupmappingattribute = dn groupmemberattribute = member groupnameattribute = cn host = ldapserver.tcsco.com nestedgroups = 0 network_timeout = 20 port = 636 realnameattribute = displayname sizelimit = 1000 timelimit = 15 userbasedn = DC=im,DC=tcsco,DC=com userbasefilter = (objectclass=*) usernameattribute = samaccountname Im using $SPLUNK_HOME/splunk/etc/system/local/authentication.conf" on all the SHs to make the changes. ... View more

Hi @SloshBurch , I reset the entire settings and edited the server roles and applied the changes. As of now it looks good. But not sure if a splunk restart on the license host will change it. I will monitor for few more days and validate it. Thankyou for your constant support ... View more

Hi @SloshBurch , Im am setting up the DMC on the license Master since the old wasnt set up properly. Im facing this multiple server roles on the License Master DMC. ... View more