Suggestions made by @PickleRick  are probably best to go with.  In terms of it still not working - you will most likely need to adjust the reg-ex pattern based on your logs. ... View more

I've never come across an Splunk environment that uses dynamic IP's for indexers (might be asking for trouble) , but there may be some use cases, I don't know, probably cloud environments. Normally one would use static IP's and DNS service and name's for UF to Indexers communications. You would then configure your outputs.conf with those DNS names. The UF's have inbuilt functionality to spray portions of the data across the Indexers. So DNS may be the way forward for you.  Example outputs.conf    [tcpout] defaultGroup = my_indexers [tcpout:my_indexers] server = indexer1.example.com:9997, indexer2.example.com:9997   If using Indexer cluster, you can use the Cluster Master Discovery Option - Read all about it here.  https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/indexerdiscovery    Regarding Smart Store - Read all about it here.  https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/AboutSmartStore  ... View more

With the new version, there are a number of changes, have a look through this doc, in short, you need to ensure a number of new indexes are in place _ds* see the doc and there's another setting in the output.conf of the deployment server and add the new whitelist indexes to the UF's. Try these if it still fails log a support call. https://docs.splunk.com/Documentation/Splunk/9.2.0/Updating/Upgradepre-9.2deploymentservers  ... View more

Hi  You might then be able to apply a regex pattern to say to NOT not match ERROR or FATAL, therefore keep them, and discard the rest.  Try this  ^(?!.*(ERROR|FATAL)).*$ ... View more

Well, it can be several things, network/config: You have shown the inputs but what about the outputs? Obviously, you will have a better understanding of your network / access / data flow details, but here's a number of area's for you to check  and investigate. Have you installed the Splunk Cloud UF App Package onto the HF (splunkclouduf.spl This contains the outputs.conf  / TLS config, you download this from your Splunk cloud stack). Have you allowed the HF for outbound connectivity to Splunk Cloud (Firewall changes) ? After you download and install the Splunk Cloud UF App Package onto the HF, can you see the HF's _internal logs in Splunk cloud? In Splunk cloud there is allow IP whitelisting feature, have you configured this for the HF to allow data to be sent to Splunk cloud? ... View more

In Splunk you need to configure alert actions, as you can see many come out of the box for your use case, you have a few options that you can explore.   1. Use this Add-on - it may help with some config/testing so needs to be installed - https://splunkbase.splunk.com/app/5520  2. Develop your own Action - https://dev.splunk.com/enterprise/docs/devtools/customalertactions/    ... View more

It shows out of memory in the log - this could be caused by large volumes of data coming in from 0365 events. You might consider changing the interval in the inputs for the collection. (I don’t know if this will fix it, but may help with the different inputs you may have, sounds like its bottlenecked somewhere ) Check the memory usage on the where this add-on is running (normally on a HF)  - perhaps you need to increase this if it’s very low. Have a look at the troubleshooting guide, there may items there to help further investigate. https://docs.splunk.com/Documentation/AddOns/released/MSO365/Troubleshooting ... View more

Sourcetype  is important because it categorises the raw data and should extract / parse the data into fields.  From the screen shot it looks like your data is not being parsed/extracted on the SH.  1. You most likely do not have the correct sourcetype or TA installed for you TA.     2. Obviously this is firewall data (I have never heard of a sourcetype firewall, but it could be a custom name, normally its called or set with a meaningful name like cisco:asa   etc.  Run this command and see if it returns any sourcetypes, if it still doesn't, identify the vendor of the firewall logs, find the TA in Splunk base, look at how you are ingesting this data, inputs and check and note the metadata settings, use the sourcetype from there. If not you will have to develop a custom one for this data source. | tstats count where index=firewall BY sourcetype, index | stats values(sourcetype) BY index .      ... View more

This could be a number of things as to why your not getting any results.  With Splunk you should be able to see the fields in the fields side bar provided you have access to the index (permissions) and the data has been onboarded correctly and fields are extracted. Run index="firewall" and see of you get data and then you should find the sourcetype associated with the data you want to search.   Example  index="firewall" sourcetype=<Add your sourcetype here> | table host, src_addr, dest_addr Note: The fields your interested based on your data may be different - so look at the left fields side bar.  If you  cant get anything, it may be that you don't have permissions to see that firewall index/data or the data has not been onboarded correctly     ... View more

The first thing about dashboards is that you should create draw out a design, what data, fields, and what kind of layout, table, chart, timechart, forms etc.   Then create a prototype dashboard based on that and refine it until you have the results.  Why not try and create the dashboards, have a look here there a several examples  https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchTutorial/Createnewdashboard  Even better if you run through this tutorial - by the end of the week you should be able to create some of you own dashboard.  https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/SearchTutorial/WelcometotheSearchTutorial    These are very simple examples of different ways to present your data and put them into a dashboard.  This shows just a table for the fields of interest to you:. index="wireless_retail" source="CPS.cpsLog" | fields ID, Date, Level, Logger, Message | table Date, ID, Level, Message   This shows how many events by Level field index="wireless_retail" source="CPS.cpsLog" | fields ID, Date, Level, Logger, Message | stats count by Level, ID, message   This shows using time period for Level by ID index="wireless_retail" source="CPS.cpsLog" | fields ID, _time, Date, Level, Logger, Message | timechart span=1h count by Level by ID You can also download this app and use the many great examples here  https://splunkbase.splunk.com/app/1603    ... View more

Using regex and spath commands can be used to extract fields, but it’s easier to INDEXED_EXTRACTIONS= JSON OR KV-mode=json and json data can change. If no events are getting auto extracted then it sounds like your sourcetype may not be applied.  There are some steps/investigations on your part to undertake. Check at the inputs level the data is getting set with your TA props.conf sourcetype you have set - verify this. (The data must be coming in from a JSON file or HEC type of inputs somewhere) Once you know the correct sourcetype, ensure that the KV-mode=json has been applied with other settings such as the below. Note: INDEXED_EXTRACTIONS= SON and KV-mode=json set for the same sourcetype together causes the Splunk software to extract the JSON fields twice: once at index time, and again at search time - advise do not do this, stick to KV-mode=json for now) Analyse the data, and workout out some of the settings – (known as magic 6)  for props.conf such as in the example below. Tip - Ideally you should always place new data into a test index and get the props working and the place into production once its all working as expected.   Example props   [my:json:data:sourcetype] KV_MODE = json #Tune the below to make Splunk more efficient MAX_TIMESTAMP_LOOKAHEAD = (look no further in the data for timestamp) SHOULD_LINEMERGE = false (leave default) TIME_PREFIX = (REGEX before the timestamp) TIME_FORMAT = (Check your time stamp and format it- example - %Y-%m-%d %H:%M:%S%:Z) TRUNCATE = 10000 (Leave as default, may need tuning) LINE_BREAKER = (REGEX to Work out where to break the line)     Apply the above to your TA based on your specific, deploy, test and adjust as required. Also, there may already be a props TA if this data is common data source from Splunkbase have you checked that? ... View more

It could a be several things post restart. Get some more info from the _internal logs - this may help further investigate and identify the issue index=_internal sourcetype=splunkd splunk_app_db_connect index="_internal" sourcetype=splunkd "db_connect" log_level=ERROR   Check the KVstore status | rest splunk_server=local count=1 /services/server/info | table kvStoreStatus   OR $SPLUNK_HOME/bin/splunk show kvstore-status --verbose   Check DB APP permissions chown -R splunk:splunk $SPLUNK_HOME/etc/apps/splunk_app_db_connect   Sometimes it won’t start due to default certs, as they may have expired,  If using the Splunk default certificates, move or rename the file .old etc the $SPLUNK_HOME/etc/auth/server.pem file and restart Splunk to regenerate the certificate.   Check the JAVA server java –version Make sure it's compatible https://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Prerequisites   See if there some additional help via troubleshooting page https://docs.splunk.com/Documentation/DBX/3.16.0/DeployDBX/TroubleshootingTool   If that all fails to help resolve the issue, log a  support case.               ... View more

@airforce Hi  The DB connect is what you need for integration with Snowflake Logging. So go with that.  https://docs.splunk.com/Documentation/DBX  https://splunkbase.splunk.com/app/2686  The Snowflake app is for Splunk SOAR (Security Orchestration And Response) application which is for Security Process Functionality, from your question it appears you don't need that .  ... View more

I've never used Kiwi syslog, but you can use the netcat (nc) utility to send test syslog messages to the SC4S server first and check, netcat needs to be installed.    UDP test echo "My Test UDP syslog message" | nc -w1 -u <YOUR SC4S Server> 514 OR locally from the SC4S server echo "My Test UDP syslog message" | nc -w1 -u localhost 514 And see if any messages are sent to the Splunk/HEC Also check SC4S to see if data is being sent, when you send data from the Kiwi system   sudo tcpdump -i any udp port 514   Other things to check:  Check the /opt/sc4s/env_file - these are the default ports, but I can't remember if you need to add these as they should be default, may be worth adding these and restarting and see if that could be the cause.   SC4S_LISTEN_DEFAULT_TCP_PORT=514 SC4S_LISTEN_DEFAULT_UDP_PORT=514   Check the logs  podman logs SC4S  You said the firewall is ok but might be worth disabling it temporarily. ... View more

yes, the Splunk admin can then add it to the correct app context and apply permissions.  ... View more

As you don't have admin access, you have some options: 1. Create the transforms.conf / collections config using a file editor if you know what your doing and give it your Splunk admin they can do the rest. 2. You can download a free instance of Splunk (Install it if you know what your doing)  and do the dev work there and then give the config to your Splunk admin. 3. You can also use the lookup editor app - https://splunkbase.splunk.com/app/1724  this is an easy way to create kvstores - you need to install this app and its popular, get you Splunk admin to install this. ... View more

@splunky_diamond  your welcome  Here's 's some more security tips to help you discovery some more. 1. Many Security people use this app to help them with there Security Use cases, I use it myself - so many good features, it can also make use case recommendations based on on your data sources. https://splunkbase.splunk.com/app/3435  2. ESCU - Provides regular Security Content updates to help security SOC / analysts to address ongoing time-sensitive threats, attack methods, and other security issues. https://splunkbase.splunk.com/app/3449  3. Here you will find so many use cases for reference - great place to baseline your security monitoring strategy. https://research.splunk.com/  ... View more

It  could be a permissions issue you need read the email address attribute ((&(objectClass=user)(objectCategory=person)(mail=*))) check the user permissions that is being used to pull the LDAP data, see your AD admin. Or run something like the below to check under that user account.   dsquery user -samid username | dsget user -email If not, find out how it’s being populated, normally its done via the ldap search command see references below. Check the ldap search that creates the lookup and you should have the data there, this may have been created already as a secluded search. Reference: Ldap Search using the command https://docs.splunk.com/Documentation/SA-LdapSearch/3.0.8/User/Theldapsearchcommand Ldap Add-on https://docs.splunk.com/Documentation/SA-LdapSearch/3.0.8/User/AbouttheSplunkSupportingAdd-onforActiveDirectory  ... View more

Normal searches run on the Raw Data and Datamodels are a populated dataset based of the Raw data target fields, hence the Data models are faster.   Data models normalize and standardise data across multiple indexes, allowing users to analyse data consistently regardless of the source. They include accelerations and summaries, such as data summaries, these accelerations speed up searches and make analysis faster and more efficient, especially for large datasets. Overall, using data models in Splunk enhances data analysis capabilities, improves performance, and simplifies the process of exploring and understanding data. ES is based on Datamodels, so you index you security data sources like Firewall/IDS,Network/Auth in the normal way, you then ensure you install the CIM (Common Information Model) Compliant TA's for those data sources, and after you tune the Datamodels for searching your target indexes and it will search and populate the Datamodels based on the types of data sources. Once all in and configured ES lights up, and you can deploy various Correlation Rules which mostly run on the datamodels. (Simple explanation)  Example: You want to ensure your Network Firewall Traffic Data Model is available for data for ES, you then ingest Cisco ASA data into your normal index, you then ensure you install the Cisco ASA TA from Splunkbase, you then tune CIM for this data so it searches it and populates Network_Traffic data model on a regular basis and from there you can run various Rules or create your own, using tstats etc.     See the below for the various data models and various normalised fields https://docs.splunk.com/Documentation/CIM/5.0.2/User/Howtousethesereferencetables ... View more

Example using makeresults command for the Json data | makeresults | eval json_data="{\"pyOptions\":{\"HasTelephonyPriv\":\"true\",\"isSnapshotOnly\":\"\",\"pyAutoLogin\":\"\",\"pyClientHandle\":\"HEWR40W8VLO39ZP5OVIBJKMZKEF8YETH5A\",\"pyDeviceState\":\"\",\"pyNumberOfLines\":\"3\",\"pyPegaCTIError\":\"\",\"pyTelephonyMode\":\"1\",\"pyThisPageAsJSON\":\"\",\"pyUserIdentifier\":\"user1234\",\"pyUserName\":\"\",\"pyUserPassword\":\"\",\"pyWorkMode\":\"Busy\",\"queue\":[\"\"]},\"pyPageExists\":\"false\",\"pyPort\":\"7017\",\"pyPresenceAgent\":\"H-GET\",\"pySelectedLinkName\":\"CHANNELSERVICES-ADMIN-CTILINK-LOCAL-JTAPI AVAYAPBX1\",\"pySSLProtocolVersion\":\"TLSv1.2\",\"pyStatusMessage\":\"Couldn't connect to server\",\"pyStatusValue\":\"Fail\",\"pySwitchType\":\"Avaya EAS CM\",\"pyVendor\":\"Avaya\",\"pyWorkgroupPhoneBook\":\"true\",\"pzInsKey\":\"CHANNELSERVICES-ADMIN-CTILINK-LOCAL-JTAPI AVAYAPBX1\",\"pzLoadTime\":\"May 3, 2024 9:00:35 AM CDT\",\"pzOriginalInstanceKey\":\"CHANNELSERVICES-ADMIN-CTILINK-LOCAL-JTAPI AVAYA-1\",\"pzPageNameBase\":\"D_CTILinkInfo\",\"LogoutReasonCodes\":[],\"NotReadyReasonCodes\":[],\"pyThisDN\":\"24181\",\"pyWorkMode\":\"Busy\"}" | eval pyUserIdentifier=spath(json_data,"pyOptions{}.pyUserIdentifier") | eval pyStatusMessage=spath(json_data,"pyStatusMessage") | stats count BY pyUserIdentifier,pyStatusMessage If using the spath command the data must be well-formatted as per standards https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Spath If you are using indexed_extractions=JSON or KV_MODE=JSON in the props.conf file, then you don't need to use the spath command as it auto extract the fields/values for you and you can then use the stats command based on your fields, and this is the preferred option as it auto extract the fields/values for you. If you don't know what this is Speak to your Splunk Admin to onboard the json data correctly.   ... View more