The transform happens at index time. ... View more

Most of the important Splunk logs are in the _internal index.  Sourcetype splunkd has most of them, but there are others which you may find useful.  It all depends on what's important to you. ... View more

The problem is with the earliest and latest settings.  The CS is configured to search from the beginning of time (technically, 1 second into the Linux epoch) until the present time and to do so every hour.  That means it will pick up the same events every hour forever. Try these settings: Earliest = -61m Latest = -1m ... View more

That requires Splunk Professional Services.  Contact Splunk and they will provide steps, LOE, and a plan. ... View more

While SEDCMD will remove the quotation marks, it's one of the last props.conf settings processed so it has little to no effect on the other settings.  See https://www.aplura.com/assets/pdf/props_conf_order.pdf for the order in which props are processed. FTR, the FIELD_DELIMITER and FIELD_NAMES settings apply only when INDEXED_EXTRACTIONS is used. Do you have any control over how the event is generated?  If so, can the quotes be removed? I'd suggest using a transform, but it has the same precedence as SEDCMD. Changing TIME_PREFIX will get the timestamp extracted.   TIME_PREFIX = "   Another option is to parse the event using REGEX. props.conf: [sourcetype] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true CHARSET = UTF-8 disabled = false TRUNCATE = 20000 TIME_PREFIX = " TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N TRANSFORMS-parse = parseSourcetype transforms.conf: [parseSourcetype] REGEX = "(?<timestamp>[^\|]+)\s?\|\s?(?<type>[^\|]+)\s?\|\s?(?<num>[^\|]+)\s?\|\s?(?<area>[^\|]+)\s?\|\s?(?<code>[^\|]+)\s?\|\s?(?<uuid>[^\|]+)\s?\|\s?(?<text>[^\|]+)\s?\|\s?(?<message>[^"]+)" Consider using Cribl (cribl.io) to strip out the quotes before handing the events to Splunk. ... View more

Looks like we need to be more explicit with the time prefix.  Try this TIME_PREFIX = EventDateTime=" ... View more

There are examples of that query in the Splunk Security Examples app.  None of them fit your exact scenario, but they should be adaptable to your environment. ... View more

There are multiple ways to do that.  Here's one. | inputlookup mylookup.csv ``` Normalize the separator ``` | eval IP = replace(IP, ";", ",") ``` Separate the IP addresses ``` | eval IP = split(IP, ",") ``` Give each IP address its own event ``` | mvexpand IP ``` Write the results back to the lookup, if desired ``` ``` | outputlookup mylookup.csv ``` ... View more

I left out some key wildcards.  I've updated my answer. Like I said in the answer, the query searches your saved searches, reports, and alerts (all the same thing, really) for references to the lookup file.  It also searches for the outputlookup command to help narrow the scope to those searches that write to the lookup. ... View more

What exactly are you getting for results?  What does "parse properly" mean to you? I can see that the TIME_PREFIX setting is incorrect.  Remove the angle brackets and it should work. Also, the TIMESTAMP_FIELDS setting only applies when INDEXED_EXTRACTIONS is used. ... View more

Search your saved searches for the name of the lookup file.   | rest /servicesNS/-/-/saved/searches splunk_server=local | search search="*computers.csv*" search="*outputlookup*" | table title   ... View more

The Monitoring Console is just a search head so the normal SH ports apply. ... View more

Here's one possible query. index=myapp_prod sourcetype="service_log" MyCriticalAPI Status=* | eventstats count as Total | stats count as Count, values(Total) as Total by Status | eval Pct = Count * 100 / Total | table Status Count Pct ... View more

I'm not sure it's safe to say the app was busy all 5 minutes, but it's better than nothing. index=foo app=* currentUseCount=* | eval busyMinutes=currentUseCount * 5 | timechart span=1h sum(busyMinutes) as busyMinutes by app ... View more

I use this command after first making a backup of the app's lookups directory. tar -zxf <<app file>>.tgz -C /opt/splunk/etc/apps   ... View more

If you untar a new version of an app into $SPLUNK_HOME/etc/apps, it will overwrite all existing files except those in the local directory.  Note that any changes to files in the lookups directory will be replaced with files from the tarball. ... View more

The only assumption we can make about uptime based on success rate is that every time a status code is returned then the application must be up.  The converse is not true - if no status code is returned then we don't know if the application is down or if no request was received. ... View more

I'm glad to hear you solved your problem. I think it is not correct to say that 1 day of data is not good practice.  It can be good practice, depending on your needs.  Many sites use that practice to help ensure their data freezes in a timely fashion.  If a bucket contains multiple days of data then old data in that bucket will remain searchable until the newest event in the bucket expires.  That could violate the site's data retention policy. ... View more

There's a difference between "availabilty" and "success rate".  The sample logs appear to show only responses, from which we can calculate the success rate.  You've done that. Availability, however, must include all of the times a request was sent by the user and NOT received by the server so there is nothing in the logs for it.  That includes times when the application was down or the network too congested, etc.  We can't compute that based on what we have now. If the application logs its own state then we can compute uptime based on how long it spends in something other than the "Up" or "Ready" state. ... View more

Based on the sample events shown, we know the apps were busy during the midnight hour on 6 Feb.  What we don't know is how much time the apps spent being doing work so we can't say how many hours were spent by each app. You can use this query to see how much use each app received each hour.   index=foo app=* currentUseCount=* | timechart span=1h sum(currentUseCount) as hourlyUseCount by app   ... View more

There is no automatic script, but you can write one.  Don't have it issue the set deploy-server command, however.  Have it delete $SPLUNK_HOME/etc/system/local/deploymentclient.conf.  Put this in an app called "fix_deployclient", but don't deploy it, yet. Create a new app called "all_deployment_client" and add to it the existing $SPLUNK_HOME/etc/system/local/deploymentclient.conf file from one of your forwarders.  Set the app to restart splunkd when it deploys. Now deploy all apps to all forwarders.  This will move the deployment server config to a custom app.  The next time you need to make a config change to your forwarders, modify the all_deployment_client app and re-deploy it. Never use the set deploy-server command again.  Why?  Because that command puts entries in $SPLUNK_HOME/etc/system/loca/deploymentclient.conf, which overrides all other deploymentclient.conf files, making it impossible to make a change from the DS. Another option is to use your company's management software (Ansible, Puppet, etc.) to change all of the forwarders. ... View more