I am trying to solve a problem where Splunk DB Connect [DBX] can leverage Azure AD authentication through JDBC driver for Azure MS SQL Server access. Currently, upon trying to achieve similar, DBX Server ends up into a java exception -   2021-05-15 12:44:31.032 +0800 [dw-56 - POST /api/connections/status] ERROR io.dropwizard.jersey.errors.LoggingExceptionMapper - Error handling a request: be206a071cdf0267 java.lang.NoClassDefFoundError: com/microsoft/aad/msal4j/IClientCredential ... ... Caused by: java.lang.ClassNotFoundException: com.microsoft.aad.msal4j.IClientCredential at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 92 common frames omitted   Microsoft has already enhanced its JDBC Driver to work with Azure AD Auth instead of native MS SQL user based -  https://docs.microsoft.com/ja-jp/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver15  Point #1 & #2 are about the use-cases one of which I would want make use of, #3 is something that is currently documented and works just fine. 1 - ActiveDirectoryPassword Supported in driver version v6.0 and later, you authentication=ActiveDirectoryPassword can use your Azure AD username and password to connect to Azure SQL Database and Synapse Analytics. Illustration of the JDBC String Usage –   jdbc:sqlserver://<instance_url>:<instance_port>;database=<db>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=<instance_domain>;loginTimeout=30;authentication=ActiveDirectoryPassword   2 - ActiveDirectoryServicePrincipal Supported in driver version v9.2 and later, you authentication=ActiveDirectoryServicePrincipal can use the client ID and secret of the service principal ID to connect to Azure SQL Database and Synapse Analytics. Illustration of the JDBC String Usage –   jdbc:sqlserver://<instance_url>:<instance_port>;database=<db>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=<instance_domain>;loginTimeout=30;authentication=ActiveDirectoryServicePrincipal;aadSecurePrincipalId=<secure_principal_id>;aadSecurePrincipalSecret=<secure_principal_secret>     3 - SqlPassword - works just fine authentication=SqlPassword Use to connect to SQL Server using the userName or user and password properties.   - I have approached Splunk Support - They say its beyond break fix - Then, Professional Services - They say its more of OnDemand - Then, OnDemand - Says it needs to be an enhancement, so should be an Idea - Now, Idea - is where I am at this moment So, I am approaching to the community if someone has solved this problem then please share the solution / workaround, else please upvote the idea, so that it gets the due attention. Idea - https://ideas.splunk.com/ideas/EID-I-987    Splunk Premium Customers and Partners would like to see Splunk DB Connect [DBX] Server enhanced to leverage Azure AD authentication through JDBC driver for Azure MS SQL Server access. Currently, upon trying to achieve similar, DBX Server ends up into a java exception, detailed into the Support case attached with this idea request. Microsoft has already enhanced its JDBC Driver to work with Azure AD Auth instead of native MS SQL user based. Idea is simple - DBX [Req Initiation] ->> JDBC [Request Handover] ->> Azure AD [Request Auth] ->> Azure MS SQL [Request Fulfillment] ... View more

Hello Splunk Gurus, I would like to understand if Splunk has solved this problem about auto-scaling Splunk Indexer-Cluster depending upon the incoming data-volume in AWS via tools like K8s or Terraform or any-other? What the problem statement states is – Spin more indexing nodes as the data volume increases, automatically Provision an AWS instance with Splunk image Mount the data volume Add the indexer into existing cluster as a peer to store and replicate the buckets Remove indexing  nodes as data volume decreases, automatically Inform the Cluster Master about scaling down Remove the indexer(s) from the cluster Unmount the data volume and free-up the disk space back to AWS De-commission the AWS instances Making sure the data is fully available and searchable during this process Purpose of this exercise is - To save the AWS cost since its pay-as-you-use model and if, on the day of less incoming data, few of the indexing nodes can be shut-down since they are mostly underutilized on such days due to less search activities and less indexing data. My biggest concern about auto-scaling is - the fact that buckets are replicated randomly on all the indexers of the cluster, and if on a certain day when there is less data incoming, let's say over the weekends, if n indexer nodes can be shut-down to save cost, data is not completely available. And with SF=2, RF=2, if Cluster is recovered to its full-state with n nodes being shut-down, On Monday there will be so many excessive buckets with those node again becoming part of the cluster to handle the working week-day traffic. Answers I seek - I would like to know the insights about this problem-solving in terms of approach and strategy if someone and/or Splunk has solved it with their Splunk Cloud offering. I would also like to understand and have assessment inputs from the community and Splunk Gurus / Architects if its really a worthy problem to solve or if it makes sense at all, it may be an absurd idea and I am fine learning it. 🙂  Thanks! ... View more

How should I add '| dedup' as one of the constraints of the dat model ? We have a data model having a sourcetype as a base constraint and other fields using which we generate statistical reports by tstats searches. This sourcetype has got duplicated data. I would want to filter out the duplicated events so for accurate statistics reporting in the data model, so that the generated summaries are accurate. (some thing like '| dedup _raw' right before the 'stats' command in the usual searches) Please suggest some ideas. Thanks! ... View more

Child processes by lea_loggrabber do not terminate making later-triggered-parent-process into hung state. This happens only for one of the entity (lets call it entity1) while for other (entity2) it is fine. Due to this at the time of next run (after 60 secs), splunk finds the loggrabber for entity1 already running and does not trigger another instance and runs the loggrabber for entity2. Hence, we are able to pull logs from entity2 however not from entity1. Any suggestion about troubleshooting this would be much appreciated. Splunk 6.0.4 (build 207768) Splunk_TA_opseclea_linux22 - Version: 1.11.1 Posting below are the running processes at this moment (including child and hung parent processes) - [XXXXX@XXXXXXXXX ~]$ ps -ef | grep splunk splunk 3869 1 0 May21 ? 00:00:03 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 6503 1 0 May23 ? 00:03:04 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 root 9109 8644 0 04:55 ? 00:00:00 sshd: splunk [priv] splunk 9149 9109 0 04:55 ? 00:00:00 sshd: splunk@pts/0 splunk 9150 9149 0 04:55 pts/0 00:00:00 -bash splunk 10069 9150 0 04:59 pts/0 00:00:00 ps -ef splunk 10070 9150 0 04:59 pts/0 00:00:00 grep splunk splunk 12320 1 0 May21 ? 00:01:38 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 13010 1 0 May21 ? 00:01:24 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 13065 1 0 May21 ? 00:01:36 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 14604 30083 0 May26 ? 00:00:00 /bin/bash /opt/splunk/etc/apps/Splunk_TA_opseclea_linux22/bin/lea-loggrabber.sh --configentity LEAXX.XX.XX.XX splunk 14611 14604 1 May26 ? 00:03:55 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 24306 1 0 May25 ? 00:06:12 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 29825 1 0 May21 ? 00:01:04 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 30082 1 12 May26 ? 02:22:02 splunkd -p 8089 restart splunk 30083 30082 0 May26 ? 00:01:08 [splunkd pid=30082] splunkd -p 8089 restart [process-runner] splunk 30248 1 0 May26 ? 00:01:58 python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py restart splunk 30874 1 0 May21 ? 00:00:45 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 splunk 30904 1 0 May21 ? 00:00:51 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22 ... View more

Can I access datamodel objects using python SDK ? I am aware that I can access them via Java and Javascript SDK. Another question is, can I access the aggregated data sets generated out of accelerated data models or reports using python SDK and feed those data sets to the integerating components ? Its like, making available that aggregated data to the analytics platform via python SDK. Please advise and thanks in advance! ... View more

I am aware that the auto-lookup functionality is not supported with db-lookups, due to constrain of running the db-lookup locally only on the search head. Example - index=test | lookup local=1 mysql_table ip_address as clientip OUTPUT host | table clientip, host Which is not achievable with auto-lookup. Is there any work-around to this ? Can I install db-connect app on the indexers as well and have the streaming db-lookup running on both the instances, indexers + search head ? Using distributed environment (1 HFWD, 2 IDX, 1 SH), on Splunk 6.0.4 (build 207768), dbx 1.1.6. Any help is appreciated! Thanks! ... View more

Splunk Gurus! We have multiple small splunk setups in our environment with 1 HFWD, 2 IDXers, 1 DS and 2 SHs (One being job server and another for REST API calls) as per the geographical locations. And now we are moving them in to clusters (NOT multisite). So its like the 2 IDXers at California data center will be in a cluster with RF=2 and SF=1. Now to have 1 master server for a cluster of 2 IDXers is not justifiable to business and hence we came up with a solution to have our DS to act as the cluster master too. So, we are going to use deployment server to deploy the apps on to the clients and we will use the configuration bundle method to push apps to the peer nodes so this DS instance will have dual duties. More clearly, Config apps will be deployed from deployment -apps directory from DS to apps on to the DCs and configuration bundle will be pushed from master-apps from master node to slave-apps on to the peer node. Is this a recommended approach ? PS : We will eventually be moving in to multisite clustering too in near future. ... View more