If this only happens upon a restart of the server, you will need to ensure that the variable is declared in your startup script. Splunk will be configured to boot-start in a best practice scenario. However, if the boot-start script does not declare that variable, it will not exist when Splunk is started by init or systemd. Your shell RC file does not execute upon boot or init/systemd starting services; only upon interactive login. Keep it there. It is important for when Splunk is restarted by a human running the shell. Or automation for that matter. Solution: declare the variable. For init, you can simply declare it just like you already do in your shell RC in the start) f. For systemd, you will declare it like this in the [Settings] stanza:     Environment="REQUESTS_CA_BUNDLE=/full/path/to/your/internal/ca/pem/file/with/no/variables"       ... View more

Great, thanks a lot @bowesmana ,..much appreciated !  ... View more

The section of the troubleshooting guide your refer to is wrong in fixing this app's issue with respect to the certificates.  That section refers to splunk server side authentication not the app.  ... View more

here's what I landed on, in case it's helpful for folks in the future.     | eval isIPV4 = if(match(IP,"^((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4}$"),"True","False")   The best regex for validating IPV4 is an ever-evolving conversation on stack overflow.  So, I used the latest from there, but from this highest rated answer, not the accepted one: https://stackoverflow.com/a/36760050/6376311   Since someone mentioned it, in order to validate private/public IPv4, I made myself an eval-based macro with the following:   case( cidrmatch("10.0.0.0/8",$IP$),"False", cidrmatch("172.16.0.0/12",$IP$),"False", cidrmatch("192.168.0.0/16",$IP$),"False", isnull($IP$) OR like($IP$,""), "False", match($IP$,"^((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4}$"),"True")   Then, I can use it in any query like:   | eval Remote_Address_isExternal = `isExternalIPv4(Remote_Address)`     ... View more

Finally figured this one out thanks to a similar issue someone had 8 years ago lol. you will need to bypass the first-time run script by doing a few things. 1. rm -rf $splunkforwarder_home/ftr 2. cp $splunkforwarder_home/etc/auth/cacert.pem.default $splunkforwarder_home/etc/auth/cacert.pem 3. cp $splunkforwarder_home/etc/auth/ca.pem.default $splunkforwarder_home/etc/auth/ca.pem 4. cp $splunkforwarder_home/etc/myinstall/splunkd.xml.cfg-default $splunkforwarder_home/etc/myinstall/splunkd.xml 5. create a user-seed.conf file for the default admin user. 6. $splunkforwarder_home/bin/splunk start This should start the UF then just proceed to stop, enable boot-start, and finally start the UF service for the final time. ... View more

Thank you Johnhua for the note about inclusion vs. exclusion. Although i think SEDCMD might work we have decided to go with dropping the event into null queue using props and transforms. Thanks again. ... View more

if you don't have ip filed in raw data and want use rex to take out IP address , plus want know how much data coming from that ip for a day. index=<YourIDXname>earliest=-1d@d sourcetype =<"if any"> |rex "^[^\t\n]*\t(?P<srcip>[^\t]+)" | eval size=len(_raw) | stats sum(size) as bytes by srcip | eval KB=round(bytes/1024,2) |lookup iptest.csv local=true ManagementIP as srcip OUTPUT SiteIdentifier HostName DeviceManufactorer DeviceType | search HostName=* | dedup srcip | table SiteIdentifier HostName DeviceManufactorer DeviceType srcip KB | rename srcip as "Matched IPAddress" |sort by SiteIdentifier DeviceManufactorer ... View more

In my case, index=example | table SID Auto Manual Total Everything looks fine on web. However in email as inline, it shows: Auto Total SID Manual ... View more

@BernardEAI  this is a known issue with 7.3.3 with some workaround documented in the following answer: https://community.splunk.com/t5/Dashboards-Visualizations/How-to-include-CSS-in-Dashboards-with-the-latest-Splunk-version/m-p/465917 Also, ideally after testing CSS Style you should try to migrate the same from XML to a CSS static file. ... View more

Perfect, thank you @shivanshu1593  ... View more

Since you already have that data in Splunk, have you considered copying over the buckets from the test index to the production index? I believe the ServiceNow TA tracks the last update from a given table to avoid duplicates. That is why when you change the index it just continues from the most recent update from that table. ... View more

You need to use export PYTHONPATH so that this variable get passed to the child process called (to run program_name) in your script. Export PYTHONPATH instead just setting it in .bash_profile. export PYTHONPATH=/apps/splunk/etc/apps/splunk-sdk-python ... View more

it is visible via btool as soon as i add an index to our indexes.conf ... View more

Some knowledge objects have a "Move" option, but many do not. For those that do not, you'll need to edit the respective .conf files and restart Splunk. ... View more

Hi @brdr , This message is usually an indication of improperly configured storage for indexing operations. If you're running into a situation where your indexers have less than 5GB (the default threshold for this message) of free disk space for the hot/cold storage volumes, you probably have one of the following situations: 1. You have not properly configured indexes.conf settings for volume management that allows Splunk to clean up space as needed for the hot/cold volumes. 2. You have not provided the supported default minimum disk space for /opt/splunk (or wherever Splunk is installed) of 300GB and search operations are overfilling that space causing this message. From your desription, it sounds like #2 in this case. If it were me, I would investigate the server(s) in question to determine which folder is causing the issue (most likely something in /opt/splunk/var). From the CLI (Linux) I would use the following command: df -sh /opt/splunk This will show how much storage is being used by each directory in /opt/splunk. If it's var, then I would check var as well: df -sh /opt/splunk/var Depending upon which directory below that is causing the issue, there are different steps to take, but you would have an idea of where the offending data resides. If you have NOT allocated the default minimum of 300GB for /opt/splunk, I would highly recommend that you do that. If the hot/cold data shares the same mount point as /opt/splunk then I would recommend that you review your indexes.conf to implement volume/index management that does not allow your disk to fill up, and rolls data appropriately to do this. Typically, I recommend that configurations be made to leave a 5% - 10% chunk of free space on the volume. I hope this helps. ... View more

When your search head processes increased searches, the load slows down its searches. If a longer search takes too long to complete, you will see the error: Unknown sid The search job 'name of the search' was cancelled remotely or expired. Click on "Share" search at the start to save it for 7 days and avoid the error. Also, if you tell splunk to load an sid that has expired, you will get the same error. This can happen with loadjob , especially if the search job was not saved/shared. ... View more

awesome koshyk! thank you. ... View more

answer is here: https://answers.splunk.com/answers/118581/splunk-search-that-returns-all-the-user-roles-assigned-to-all-the-specific-indexes.html ... View more

Perfect. Thank you!!! ... View more

Thanks for the assist! ... View more

Thank you! ... View more