As your not an expert then it might be better for you to explore Splunks Add-on builder which will have options to create what you need and with credentials, have a look at the below as it may help.  https://docs.splunk.com/Documentation/AddonBuilder/4.2.0/UserGuide/CreateAlertActions https://docs.splunk.com/Documentation/AddonBuilder/4.2.0/UserGuide/ConfigureDataCollection ... View more

@PickleRick “Hear, hear!”  : -) The option's are better, @danroberts  go with PickRick's, same results but his is more efficient.    ... View more

The data comes from either the AD server or the Windows servers by the way of the Universal Forwarder, that's the source of the event logs.  You have data coming in from the AD server where a UF is installed and that's how the logs are collected , and the logs are configured by your AD admin, some times they need to enable further logging for advance events.  Try these first   and see if they exist as they may give you further info you need, if they don't , then it might be worth having a chat with your AD admin to find the exact event ID/log information you need.    Event ID 4771 - Kerberos pre-authentication failed. Event ID 644 - User account locked out. Event ID 4625 - An account failed to log on.    ... View more

Have a look at this example it may help, other than that work through the documentation splunk-app-examples/custom_alert_actions/slack_alerts/default/data/ui/alerts/slack.html at master · splunk/splunk-app-examples · GitHub   ... View more

So try this - think of this as version 1.0, its a bit of trial and error until you get it right. Tip - Its's always good practise to place the data into a test index first, to get it all working, once good move to a production index, just change the inputs.conf post testing/props dev work to prod index.    props.conf  [jlogs] TIME_PREFIX = Entry\s\d+\sstarting\sat TIME_FORMAT = %d/%m/%Y %H:%M:%S BREAK_ONLY_BEFORE =([\r\n]+)\.net MUST_BREAK_AFTER =local\stime([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 50 SHOULD_LINEMERGE=true TRUNCATE = 10000 NO_BINARY_CHECK = 1 KV_MODE = auto #Remove unwanted headers or data #This config is no longer needed - left for reference #TRANSFORMS-null = remove_unwanted_data_from_jlog   See what this looks like.  ... View more

This one is a bit tricky, but the below should get you started. Splunk is going into auto mode to determine the what it thinks on how to split the log into events, as this is log is not like the normal logs with Date and a line of information say for arguments sake (they come in shapes and sizes, and you normally want well formatted logs) anyway you have to create a custom props and transforms.conf file. Create the below props and transforms for the sourcetype, this should get you started at least and you will have to make tweaks. It looks like you have redacted some of the lines with XXX..., so you may need to tweak the regex in transforms with the words as they look like extra header type of information, that you don’t want. The main thing with this kind of log it as its multi line, so we need merge it.  props.conf [jlogs] TIME_PREFIX = Job\sCompleted\sat: TIME_FORMAT = %d/%m/%Y %H:%M:%S BREAK_ONLY_BEFORE =Job\sCompleted\sat: MUST_BREAK_AFTER =local\stime([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 25 SHOULD_LINEMERGE=true TRUNCATE = 5000 NO_BINARY_CHECK = 1 KV_MODE = auto #Remove unwanted headers or data TRANSFORMS-null = remove_unwanted_data_from_jlog transforms.conf [remove_unwanted_data_from_jlog] REGEX=^(?:X*|-+)\s DEST_KEY = queue FORMAT = nullQueue There's a whole load of settings to help you with understanding this this config https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Configureeventlinebreaking   ... View more

This example using makeresults can show you a case statement | makeresults count=10 | eval _raw="message success job, processed job, completed job, failed job," | multikv forceheader=1 | eval status = case( like(message, "%success%") OR like(message, "%processed%") OR like(message, "%completed%"), "success", like(message, "%failed%") OR like(message, "%failure%"), "failure", true(), "other" ) | table _time, message, status ... View more

I should have said change to windows path as the command I gave is for Linux  ... View more

You Windows Hosts should have an output.conf that sends to the HF only, if this is how you want your data flow architecture. (You don’t need the 100_tenant_splunkcloud installed on the Windows UF's Unless you want to send from them directly to Splunk cloud, this is also a viable solution.   I'm starting to think you may have the 100_tenant_splunkcloud and configured outputs to the HF on the Windows hosts, you need to have one or the other for this setup. Run the btool outputs command on the Windows UF let’s see what that shows? /opt/splunkforwarder/bin/splunk btool outputs list –debug   ... View more

Have a look ta the transpose command, see the examples, this may work for you.    https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Transpose   ... View more

Sounds like order of precedence issue- These two will help in figuring out what is take the priority setting: (Some config is taking place before the other) but go by what @gcusello  is saying.    Inputs config /opt/splunk/bin/splunk btool inputs list --debug  outputs config /opt/splunk/bin/splunk btool outputs list --debug   ... View more

If you done that, then best course or action might be log a support ticket, as there could be another underlying issue.     ... View more

Start by checking the logs These can also be set via the GUI $SPLUNK_HOME/bin/splunk set log-level HTTPServer -level DEBUG $SPLUNK_HOME/bin/splunk set log-level HttpInputDataHandler -level DEBUG $SPLUNK_HOME/bin/splunk set log-level TcpInputProc -level DEBUG Remember to set back to WARN once you have finished debugging Then search - this may give you some clues for you to further investigate index=_internal source=*splunkd.log* (component=HttpInputDataHandler OR component=TcpInputProc OR component=HTTPServer)   ... View more

Suggestions made by @PickleRick  are probably best to go with.  In terms of it still not working - you will most likely need to adjust the reg-ex pattern based on your logs. ... View more

I've never come across an Splunk environment that uses dynamic IP's for indexers (might be asking for trouble) , but there may be some use cases, I don't know, probably cloud environments. Normally one would use static IP's and DNS service and name's for UF to Indexers communications. You would then configure your outputs.conf with those DNS names. The UF's have inbuilt functionality to spray portions of the data across the Indexers. So DNS may be the way forward for you.  Example outputs.conf    [tcpout] defaultGroup = my_indexers [tcpout:my_indexers] server = indexer1.example.com:9997, indexer2.example.com:9997   If using Indexer cluster, you can use the Cluster Master Discovery Option - Read all about it here.  https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/indexerdiscovery    Regarding Smart Store - Read all about it here.  https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/AboutSmartStore  ... View more

With the new version, there are a number of changes, have a look through this doc, in short, you need to ensure a number of new indexes are in place _ds* see the doc and there's another setting in the output.conf of the deployment server and add the new whitelist indexes to the UF's. Try these if it still fails log a support call. https://docs.splunk.com/Documentation/Splunk/9.2.0/Updating/Upgradepre-9.2deploymentservers  ... View more

Hi  You might then be able to apply a regex pattern to say to NOT not match ERROR or FATAL, therefore keep them, and discard the rest.  Try this  ^(?!.*(ERROR|FATAL)).*$ ... View more

Well, it can be several things, network/config: You have shown the inputs but what about the outputs? Obviously, you will have a better understanding of your network / access / data flow details, but here's a number of area's for you to check  and investigate. Have you installed the Splunk Cloud UF App Package onto the HF (splunkclouduf.spl This contains the outputs.conf  / TLS config, you download this from your Splunk cloud stack). Have you allowed the HF for outbound connectivity to Splunk Cloud (Firewall changes) ? After you download and install the Splunk Cloud UF App Package onto the HF, can you see the HF's _internal logs in Splunk cloud? In Splunk cloud there is allow IP whitelisting feature, have you configured this for the HF to allow data to be sent to Splunk cloud? ... View more

In Splunk you need to configure alert actions, as you can see many come out of the box for your use case, you have a few options that you can explore.   1. Use this Add-on - it may help with some config/testing so needs to be installed - https://splunkbase.splunk.com/app/5520  2. Develop your own Action - https://dev.splunk.com/enterprise/docs/devtools/customalertactions/    ... View more

It shows out of memory in the log - this could be caused by large volumes of data coming in from 0365 events. You might consider changing the interval in the inputs for the collection. (I don’t know if this will fix it, but may help with the different inputs you may have, sounds like its bottlenecked somewhere ) Check the memory usage on the where this add-on is running (normally on a HF)  - perhaps you need to increase this if it’s very low. Have a look at the troubleshooting guide, there may items there to help further investigate. https://docs.splunk.com/Documentation/AddOns/released/MSO365/Troubleshooting ... View more

Sourcetype  is important because it categorises the raw data and should extract / parse the data into fields.  From the screen shot it looks like your data is not being parsed/extracted on the SH.  1. You most likely do not have the correct sourcetype or TA installed for you TA.     2. Obviously this is firewall data (I have never heard of a sourcetype firewall, but it could be a custom name, normally its called or set with a meaningful name like cisco:asa   etc.  Run this command and see if it returns any sourcetypes, if it still doesn't, identify the vendor of the firewall logs, find the TA in Splunk base, look at how you are ingesting this data, inputs and check and note the metadata settings, use the sourcetype from there. If not you will have to develop a custom one for this data source. | tstats count where index=firewall BY sourcetype, index | stats values(sourcetype) BY index .      ... View more

This could be a number of things as to why your not getting any results.  With Splunk you should be able to see the fields in the fields side bar provided you have access to the index (permissions) and the data has been onboarded correctly and fields are extracted. Run index="firewall" and see of you get data and then you should find the sourcetype associated with the data you want to search.   Example  index="firewall" sourcetype=<Add your sourcetype here> | table host, src_addr, dest_addr Note: The fields your interested based on your data may be different - so look at the left fields side bar.  If you  cant get anything, it may be that you don't have permissions to see that firewall index/data or the data has not been onboarded correctly     ... View more

The first thing about dashboards is that you should create draw out a design, what data, fields, and what kind of layout, table, chart, timechart, forms etc.   Then create a prototype dashboard based on that and refine it until you have the results.  Why not try and create the dashboards, have a look here there a several examples  https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchTutorial/Createnewdashboard  Even better if you run through this tutorial - by the end of the week you should be able to create some of you own dashboard.  https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/SearchTutorial/WelcometotheSearchTutorial    These are very simple examples of different ways to present your data and put them into a dashboard.  This shows just a table for the fields of interest to you:. index="wireless_retail" source="CPS.cpsLog" | fields ID, Date, Level, Logger, Message | table Date, ID, Level, Message   This shows how many events by Level field index="wireless_retail" source="CPS.cpsLog" | fields ID, Date, Level, Logger, Message | stats count by Level, ID, message   This shows using time period for Level by ID index="wireless_retail" source="CPS.cpsLog" | fields ID, _time, Date, Level, Logger, Message | timechart span=1h count by Level by ID You can also download this app and use the many great examples here  https://splunkbase.splunk.com/app/1603    ... View more

Using regex and spath commands can be used to extract fields, but it’s easier to INDEXED_EXTRACTIONS= JSON OR KV-mode=json and json data can change. If no events are getting auto extracted then it sounds like your sourcetype may not be applied.  There are some steps/investigations on your part to undertake. Check at the inputs level the data is getting set with your TA props.conf sourcetype you have set - verify this. (The data must be coming in from a JSON file or HEC type of inputs somewhere) Once you know the correct sourcetype, ensure that the KV-mode=json has been applied with other settings such as the below. Note: INDEXED_EXTRACTIONS= SON and KV-mode=json set for the same sourcetype together causes the Splunk software to extract the JSON fields twice: once at index time, and again at search time - advise do not do this, stick to KV-mode=json for now) Analyse the data, and workout out some of the settings – (known as magic 6)  for props.conf such as in the example below. Tip - Ideally you should always place new data into a test index and get the props working and the place into production once its all working as expected.   Example props   [my:json:data:sourcetype] KV_MODE = json #Tune the below to make Splunk more efficient MAX_TIMESTAMP_LOOKAHEAD = (look no further in the data for timestamp) SHOULD_LINEMERGE = false (leave default) TIME_PREFIX = (REGEX before the timestamp) TIME_FORMAT = (Check your time stamp and format it- example - %Y-%m-%d %H:%M:%S%:Z) TRUNCATE = 10000 (Leave as default, may need tuning) LINE_BREAKER = (REGEX to Work out where to break the line)     Apply the above to your TA based on your specific, deploy, test and adjust as required. Also, there may already be a props TA if this data is common data source from Splunkbase have you checked that? ... View more