That specific error is usually caused by you having a Search Head Cluster, then trying to edit configs on a Search Head Member instead of via the Deployer then deploying it. See this for more information. https://docs.splunk.com/Documentation/Splunk/9.2.0/DistSearch/PropagateSHCconfigurationchanges If that does not seem to be the problem here, then reply back with a few more specifics!   ... View more

My first reasonable thought is that you just need to rewrite the two searches and combine them into one. We can help with this!  What do you have for search 1 and search 2 right now? (Don't forget to use the <code> button to paste searches, and if you have to obfuscate a bit of it, feel free - but try to keep the same structure to the searches!) ... View more

First, A couple of links that you should probably have around.  This is a great flowchart for the whole pile o' stuff to do around and upgrade. https://community.splunk.com/t5/Installation/What-s-the-order-of-operations-for-upgrading-Splunk-Enterprise/m-p/408003 And of course the official docs https://docs.splunk.com/Documentation/Splunk/latest/Installation/HowtoupgradeSplunk And there are a few gotchas with the upgrades you'll have to go through.  While we don't know all the problems you'll have in your environment, the big ones that I'm aware of that you'll have to address is the upgrade to Python3 and the migration of kvstore to wired tiger.  The former is documented here: https://docs.splunk.com/Documentation/Splunk/9.0.0/Installation/AboutupgradingREADTHISFIRST The latter is found in here, though I think it doesn't make it prominent enough - https://docs.splunk.com/Documentation/Splunk/9.0.0/Installation/AboutupgradingREADTHISFIRST (It ends up pointing you to here: https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/MigrateKVstore) There may be others, you should read through the About Upgrading docs for each version with an eye on your environment and how it's set up.  For instance, here's the one for the 8.1.14 upgrade.  https://docs.splunk.com/Documentation/Splunk/8.1.14/Installation/HowtoupgradeSplunk I know, it seems boring.  But doing so will pay off in the end!   Happy Splunking! -Richfez       ... View more

Maybe it's as easy as adding it in the where clause: | mstats latest_time(application_ready_time.value) as latest_ts WHERE index=my-metrics-index host=some-host app.name IN ("appname1", "appname2") BY app.name | eval past_threshold=if(now() - latest_ts >= 30, "Y", "N") | eval latest=strftime(latest_ts, "%Y-%m-%d %H:%M:%S") | table app.name latest past_threshold x Give that a try. If it doesn't work, you can just post filter it with something like | mstats latest_time(application_ready_time.value) as latest_ts where index=my-metrics-index host=some-host by app.name | search app.name IN ("appname1", "appname2") | eval past_threshold=if(now() - latest_ts >= 30, "Y", "N") | eval latest=strftime(latest_ts, "%Y-%m-%d %H:%M:%S") | table app.name latest past_threshold x The former with the app.names as part of the WHERE clause is probably preferable.  Splunk can often push these sorts of search terms down into the search itself, but I'm not sure if it does that with mstats.  Meaning, if it CAN do that, it'll perform the same as the first one (more or less).  But if it can't do that, it'll run quite a bit slower because it'll get all those stats off disk, then throw away all but the two sets you want to keep. But it would work, either way.  🙂 ... View more

Ah - sometimes the easy answer are the answer, but sometimes they're not! So, from what I can see of fill_summary_index.py, the dedup option isn't actually magic.  That means there's no reason you can't just make a few minor modifications (mostly to timeframes) to just backfill the summary index manually. Indeed, there's no magic here anyway.  If fill_summary_index.py is not filling in your blank areas in the summary index correctly using the saved search from the "regular" collector for the summary index, then it seems to me that it's likely that the main search simply isn't working right anyway. The reasoning here is that when it runs 'normally', it's running over a time period and dumping its output to that summary index. This is exactly what the backfilling version does, with the only difference being that it sets a different start/end time.  Again, no magic, just searches running over time periods. So, a couple of ways forward. 1) You could provide the search and maybe  we can spot why it doesn't work right for backfilling. 2) You could craft up a "deduplication search" that you can pass to the backfill function to tell it *how* to identify which periods need backfilling.  (I don't know how to do this, but the notes for the backfill function says you can do this, so I believe it.  And of course, just because I don't know how to do it right now doesn't mean we can't help figure it out, or someone else might!) 3) Or maybe you can just manually run the search that would do the backfilling, only manually selecting the timeframes so that you don't get duplication.  I mean, I'd guess it's just a standard saved search that ends up with `| collect...` at the end.  🙂 Anyway, I do hope this helps, and maybe this bump will get someone else who does this a lot to chime in - we'll see!   ... View more

Generally speaking the 403 error is on the server side.  You say you are trying to download a trial, and that you "Managed to register it. "  I don't know what that latter means?  If it's a trial, you download it, install it and it just works.  They're *all* trials until you put in a real license key (or connect it to your license master).[1] You could try a) just logging into a different place in Splunk, like docs.splunk.com.  I think once you are logged in there you can then go back to splunk.com and see if it all still works. b) Have you tried just creating a new account and seeing if you can download it then?  I'm not aware of any actual process to this, you just "create new account" and fill in a few pieces of information then off to the downloads you go. Anyway, hope one of these gets you working again!     [1] LOL, this has actually been a complaint of mine for a decade now.  I don't want to download a trial and convert it, I want to feel like I'm downloading the actual paid-for version that $company owned.  But it's just words, so 'whatever'.  🙂   ... View more

Well, tstats works with more than just time fields.  The limitation is that it only works with fields that are created *at* the time the events are indexed. https://docs.splunk.com/Documentation/Splunk/9.1.3/Indexer/Indextimeversussearchtime I honestly think some of that information about performance or whatever is outdated, but most of that's all still fine documentation. Index time fields are those created when the data's indexed.  By default it's just the built-in fields, like _time, sourcetype, and so on. In some cases it's all the fields, for instance with INDEXED_EXTRACTIONS=<json/csv/whatever>. But otherwise Splunk generally relies on search time fields - fields that are built "on the fly" as you run your search.  It's more flexible and doesn't go 'out of date' as events change or your needed fields change around. The docs above should have links off them to explain more, but that's the gist of it. ... View more

tstats only works with index-time fields, and those fields are all likely to be search-time. Alternatives in this case to try to improve performance: 1) Report acceleration, which should let you create a saved report that keeps most of the details of the search up-to-date, then you refer to that in your dashboard and it should drop the time *way* down. 2) Or build a data model (if one doesn't exist in the CIM add on already) for this data, and accelerate the data model.  Similar to above in overall speed, but quite a bit different under the hood and more flexible. Both of these (and some other options) are in the docs, well explained, here: https://docs.splunk.com/Documentation/Splunk/9.1.3/Knowledge/Aboutsummaryindexing I hope that gets you started! Happy Splunking!   ... View more

Did you enable HEC?  https://docs.splunk.com/Documentation/Splunk/9.1.3/Data/UseHECfromtheCLI says it's something like   splunk http-event-collector enable -uri https://localhost:8089   (It's unclear from the docs if enabling a specific token enabled the global HEC settings or not, but this definitely will do that)   ... View more

I was expecting your props.conf to have INDEXED_EXTRACTIONS = CSV You are also using a TIME_PREFIX instead of TIMESTAMP_FIELDS...  And you have a PREAMBLE_REGEX set, which looks like it's set to the first fieldname in the header (which would remove the header line) though you don't provide FIELD_NAMES... Putting that all together, it looks like you aren't really treating those files as CSV files.  I'm not sure what's going on, but I wonder if it would work right if you treated them as CSV. If that doesn't help, it'd be useful to see the contents of a file that doesn't work, and one that does.  ... View more

Have you tried the -dedup option for the fill_summary_index.py? Run your fill_summary_index.py script with '-h', like $ splunk cmd python fill_summary_index.py -h There's all sorts of options in there, including dedup and timeframe changes.  It might be useful to spend a few minutes reading that carefully. You may also find it useful to review the fine docs on this: https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Managesummaryindexgapsandoverlaps   Happy Splunking!  ... View more

A subsearch seems like the right answer here.  A subsearch is enclosed in [] brackets inside your main search, runs first, and the results of that subsearch get fed back into the main search as search terms. So you have two searches here, the search that finds the cyberark data, and the one that finds the AD data.  You didn't provide either of those separate searches, so I'm just making up some pseudosearches for those. Let's say your cyberark search is something like index=cyberark action=doAnImportantThing | dedup user Which would return a short list of users involved in ... well, whatever doAnImportantThing is in this case.  Let's say 'Mary" and "John" So, at its simplest, you just use that as your subsearch. index=ad [search index=cyberark action=doAnImportantThing | dedup user] Don't forget to add "search" to the subsearch, it's automatic in the main search, but not anywhere else. So your subsearch runs, returns its data formatted like (( user=Mary ) OR (user=John) ), which means your outer search ends up being index=ad ( ( user=Mary ) OR (user=John) ) And there you go. You'll want to refer to here for more and more examples: https://docs.splunk.com/Documentation/Splunk/9.1.3/Search/Aboutsubsearches Some other comnmands/stuff to know - 'earliest=...' and 'latest=...', and also check out the 'format' command which can alter how the subsearch get returned (to do things like AND, or whatever else if you want). ... View more

@Jaki001  You will probably get better responses if you open a new question with your problem. The question was probably asked fine, it's only that you need your own set of answers to this and that will happen by asking the question as a question instead of as a comment to this 6 year old thread. So, copy all that out, and start up a new "Ask a Question" and I'll bet you'll get some great answers! ... View more

It's mostly in the docs: https://docs.splunk.com/Documentation/Forwarder/8.2.5/Forwarder/Installtheuniversalforwardersoftware and maybe https://docs.splunk.com/Documentation/Forwarder/8.2.5/Forwarder/Configuretheuniversalforwarder As to logrotate - that's all outside of Splunk.  Basic Linux maintenance on log files.  Any internet search engine is your friend here and all will lead to results that are useful. Best search may be though to get some "Splunk specific" search results, using a search like "splunk syslog" and read through a few of those best practices that will come up. ... View more

| inputlookup <lookup or filename> | stats max(<field that's the date field>) or | inputlookup <lookup or filename> | stats max(<field that's the date field>) BY displayName   And if you were wanting a more useful way base search here that returns regular data | lookup <lookupName> <fieldInLookup> AS <fieldInData> OUTPUT lastUpdate Which assuming you fix up the lookup name, double-check the fieldInLookup vs. fieldInData order (I always get those backwards!) and change the fieldname 'lastUpdate' to whatever it is in your lookup, will output the lastUpdate for each ... "field" you match on.  Perhaps displayName or something, whatever it is that should match. This might help: https://docs.splunk.com/Documentation/SplunkCloud/8.2.2107/SearchReference/Lookup   ... View more

Well, this is really a linux question, not a splunk question. You should be able to find a forwarder's installed splunk executable by a command similar to $ sudo find / -name splunk -type f Look for a line like /opt/splunk/bin/splunk or /opt/splunkforwarder/bin/splunk. And come to think of it, it's a rarity that it's outside of those two paths in a small environment like what yours sounds like. As to why it's not running?  That's a question with a zillion different answers. Possibly it's just not running (e.g. crashed, not set to autostart and the system got rebooted, someone issued a "splunk stop" command, etc...) and you can just start it up. Possibly it's not able to connect to your indexers  (upgrades to idx and the forward is just too old to talk to it, networking changes, gremlins. Maybe someone uninstalled the entire forwarder from it.  Disabled it.  Deleted the user it runs as. As mentioned, the reasons it's not running are nearly infinite, you'll have to just do a little basic troubleshooting.    ... View more

Well, that doesn't actually have "website" in it as a field. Still.  If you have that data ingested, and the fields that appear like they should be extracted are (total_time, content_md5, etc...), then ... OK, so I'm looking even closer at this.  How would you, like as a regular person using words in English, describe how you would manually use these 4 rows events to know if the site/page was up at the time?  Because, what I see is that it might be more effective to count where field timed_out=True (assumption that's it's value when it's not false). Or where response_code is 400 or higher (assuming these are http status codes, or similar, and that 300-level ones are redirects.  And if this is server code, my guess is problems will be in the error codes at 500 and above in that case. Either way... for a count of "not timed out" vs. "timed out"   <your base search here> | timechart span=1h count by timed_out   Or maybe you ONLY want the ones that timed_out, this way you can reserve the "by" clause in the timechart for "by title" to split it based on ... well, title.  You wanted server or web page, but I don't see that directly so this is my proxy for it.    <your base search here> timed_out=False | timechart span=1h count by title   Or maybe only where the status codes are 400+? <your base search here> status>400 | timechart span=1h count by status There are quite a few options. A lot of the options are pretty simple ones, leading me to suggest that you take Splunk Fundamentals 1. It's a free 6-10 hour on-line course from Splunk that covers a lot of fairly simple use cases like that, and a lot more.  Just search for it in ... oh they've changed things in Splunk education recently.  Try here: https://education.splunk.com/single-subject-courses and look at the ones that offer "free e-learning".  Many of those map to Splunk Fundamentals Part 1.     ... View more

Dumb question! Is this right, or should the minus and plus be reversed? | eval end=if(info_max_time=="+Infinity",now(),info_max_time)-432000 | eval beginning=if(info_min_time=="0.000",1604260193,info_min_time)+432000 Because a quick read of that seems to say you are taking the end and *subtracting* 432000 from it, and the beginning and *adding* 4320000 to it, then trying to find where _time was between those two. I think it might need to be | eval end=if(info_max_time=="+Infinity",now(),info_max_time)+432000 | eval beginning=if(info_min_time=="0.000",1604260193,info_min_time)-432000 Hopefully it's that simple of a facepalm, but let us know if it's not!   Also, you could try hardcoding some times in there for those to make sure it works right when you do that.  Might help narrow down where it might be going wrong. ... View more

This might be possible with the stats command, transaction command, or in a variety of other ways. What does your data look like?  Can you provide a few rows of that raw data as you see it in Splunk? ... View more

You could possibly just stick a | head 1000 or some other number right after the base search (before the first rex) and run it over a somewhat longer period (like 20 or 30 minutes in this case), if you had a reasonably predicable "count" you could go back to consistently. How would you, as an actual person looking at the data, describe what you mean by "only showing the last data".  I mean, how would you do that if you had this data in excel?  What would define the "last data," and how would that be split apart from the data that came before? I think with this answer we might be able to come up with something better.   On a side note, this search looks both really fragile, and also could be faster and better. Was the replace for 1H and 1E with UC1 skipped on purpose with the appendcols search? I think if that was just an omission or isn't important, that you could do this index=centreon check_command="Cisco-SNMP-Interfaces-Global-Status" service_description="Status_All-Interfaces" src_interface!="Ethernet*.*" src_interface!="Vlan*" src_interface!="mgmt*" src_interface!="port*" src_interface!="Null*" src_interface!="loopback*" | head 1000 | rex field=host "ZSE-(?<loc>\w+)-(?<room>\w+).*" | replace "1H" WITH "UC1"| replace "1E" WITH "UC1"| replace "2H" WITH "UC2"|replace "2F" WITH "UC2"| replace "6E" WITH "C6"| replace "6F" WITH "C6"| replace "6T" WITH "C6"| replace "4B" WITH "C4"| replace "4T" WITH "C4"| replace "4E" WITH "C4" | eval site=loc+"-"+room | stats count(src_interface) as tot_int, sum(eval(if(state_interface="up",1,0))) as tot_up by site | eval tot_free = tot_int - tot_up   I'm sure that probably needs tweaking, but the advantages in speed and non-duplicated SPL is very large - it's probably worth some time doing a thing like this.   ... View more

Could you explain what it is you are trying to actually accomplish?  I'm pretty sure we can find a way to get done what it is you need to get done if we only knew what it was you were trying to do, but you are still starting at the end of the process.  But maybe this is the answer you were looking for - There's no way to branch like that in SPL.  It is a linear pipeline from the beginning to the end (with a few weird fan-outs that aren't branches for the map command and a few others).  If you do a thing in step N-1 that removes field X, then in step N you can't reference field X because it's not there.  The *only* output it gives will be the output from step N, and that won't include what was removed in step N-1. So if outputlookup doesn't have a way to filter to a specific set of fields, the only way to filter it to a specific set of fields is in the step before the outputlookup, which means you lose that context. So either you run the search twice - once to get the alert and once to update the threshold, or you rethink *how you are doing this whole process* and make it so it'll all work like you want.  Which we'd love to help with, but we need to know what it is you are doing in order to help you with that.                 ... View more

If you only want "last_file_time" back in the MyFileThresholds.csv, then only *put* last_file_time in there. ... | table last_file_time | outputlookup append=false MyFileThresholds.csv  Of course, you didn't structure your search to only GIVE you a single last_file_time, but instead to give one per event, and an event in this case is ... probably ... possibly?... a list of filenames and last_file_time and some other fields? If you only wanted to update the last_file_time with the new information from your search, then it seems like that's what you are doing - though you hid that giant search inside the "it has to be a join" so we can't really see what it's doing.  Anyway! If you would like a better answer, please let us know what it is you are trying to do in English.  Just explain it. It feels like you are starting at the end of the process instead of at the beginning.   ... View more

OK, so assuming you strictly interpret the need here, and assume that it's only values of 600 and 900 and no others, and I took out one 600-value from the test data I built so I could confirm that if there's a 900 and NOT a 600, that it does what I expected. Then, with those caveats and this fragility, one could do the last two lines here. | makeresults | eval testdata = "cod:5678,status:600XXcod:9012,status:600Xcod:1234,status:600Xcod: 1234,status:900Xcod:4987,status:600Xcod:4987,status:900Xcod:3655,status:900" | makemv testdata delim="X" | mvexpand testdata | rex field=testdata "cod:\s*(?<cod>[^,]*),status:\s*(?<status>\d*)" | chart values(status) over cod by status | rename 600 AS Status1, 900 AS Status2 WHERE - the first 5 lines (run them all by themselves!) are just to build a test data set as a run-anywhere search. The chart command and the rename command - the last two lines - are the ones that do the work. And it assumes you have this data ingested in a way there are two fields for each record - a cod field and a status field.  Otherwise, really, you should go back and reingest the data and set that up. ... View more

Good News! It appears that in Splunk Enterprise 8.1 they have added this. Direct quote from authorize.conf for version 8.1+ (just search for it in there): srchIndexesDisallowed = <semicolon-separated list> * A list of indexes that this role does not have permission to search on or delete. * 'srchIndexesDisallowed' takes precedence over 'srchIndexesAllowed', 'srchIndexesDefault' and 'deleteIndexesAlowed'. If you specify indexes in both this setting and the other settings, users will be unable to search on or delete those indexes. * Follows the same wildcarding semantics as the 'srchIndexesDefault' setting. * If you make any changes in the "Indexes" Settings panel for a role in Splunk Web, those values take precedence, and any wildcards you specify in this setting are lost. * No default.   ... View more