About deepakc

deepakc

Data rolls off due to a few reasons. Data arrives in Splunk, it then needs to move through HOT/WARM>COLD/FROZEN, otherwise data will build up and you will run out of space. 1.Warm buckets move cold when either the homePath or maxWarmDBCount reach their limits. 2.Cold buckets are deleted when either the frozenTimePeriodInSecs or maxTotalDataSizeMB reach their limits. This may help show why it’s moving – see the event_message field index=_internal sourcetype=splunkd component=BucketMover | fields _time, bucket, candidate, component, event_message, from, frozenTimePeriodInSecs, host, idx,latest, log_level, now, reason, splunk_server, to | fieldformat "now"=strftime('now', "%Y/%m/%d %H:%M:%S") | fieldformat "latest"=strftime('latest', "%Y/%m/%d %H:%M:%S") | eval retention_days = frozenTimePeriodInSecs / 86400 | table _time component, bucket, from, to, candidate, event_message, from, frozenTimePeriodInSecs, retention_days, host, idx, now, latest, reason, splunk_server, log_level You apply config via indexes.conf for the index for disk constrains by configuring the various options: Settings: frozenTimePeriodInSecs (Retention Period in seconds - Old bucket data is deleted (option to freeze it) based on the newest event - maxTotalDataSizeMB = (Limits the overall size of the index - (hot, warm, cold moves frozen) maxVolumeDataSizeMB = (limits the total size of all databases that reside on this volume) maxWarmDBCount = (The maximum number of warm buckets moves to cold) maxHotBuckets = (The number of actively written open buckets - when exceeded it moves to warm state) maxHotSpanSecs = (Specifies how long a bucket remains in the hot/warm state before moving to cold) maxDataSize = (specifies that a hot bucket can reach before splunkd triggers a roll to warm) maxVolumeDataSizeMB = (Overall Volume Size limit) homePath.maxDataSizeMB = (limit the individual index size) coldPath.maxDataSizeMB = (limit the individual index size) maxVolumeDataSizeMB = (limits the total size of all databases that reside on this volume) See the indexes.conf for details https://docs.splunk.com/Documentation/Splunk/9.2.1/Admin/Indexesconf

deepakc

It sounds like you have created a custom syslog app with custom application type of data and its not one of the common NETWORK syslog sources...this means it’s not going to be parsed and formatted and handled by SC4S, therefore your options are: Option 1. See if the SC4S community can create one for you (As this sounds like it’s NOT network data then you might have issues as it sounds like a custom application data. SC4S is not designed to handle OS or Application data. You can log an issue here https://github.com/splunk/splunk-connect-for-syslog and maybe they can help. You will need to send a PCAP file. (I doubt if this is feasible, so then look at option 2) Option 2. Install a normal syslog server (syslog-ng or R-syslog) and configure it as opposed to using SC4S as its primarily designed to handle common network syslog data sources. Send your custom syslog app data to the server running normal (syslog-ng or r-syslog) and configure it log the data into text files into a folder. Install a Splunk UF and configure it to monitor (inputs.conf) your log files and send to Splunk cloud via outputs.conf. The Splunk UF will pick those up and then using outputs.conf send that data to Splunk cloud. You then need to create a TA to parse the custom syslog raw data, so apply metadata, sourcetype, fields, extraction and ensure the timestamp etc are all correct, then install the custom TA in Splunk cloud.

deepakc

So the the echo works - you can see data in Splunk, but your syslog APP which sends syslog data is not visable in Splunk and tcpdump shows that the APP is sending data to SC4S. Things to check: 1. Check the “No data in Splunk" section - https://splunk.github.io/splunk-connect-for-syslog/main/troubleshooting/troubleshoot_SC4S_server/#hectoken-connection-errors-aka-no-data-in-splunk Restart sc4s and look at the logs /usr/bin/<podman|docker> logs SC4S 2. Is your syslog APP a common syslog source and supported by SC4S? 3. Is your syslog APP in the known SC4S vendors list? 4. Check if it need some special enviromental config for the /opt/sc4s/env_file (Example look at the McAfee known source, it has a number of configuration options, indexes, ports, TA's env file config see this example - https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/McAfee/epo/) 5. Check the /opt/sc4s/env_file ensure the settings for your syslog APP are set here. 6. Check /opt/sc4s/local/context/splunk_metadata.csv Ensure the keyname (You App source), and ensure its mapped to the correct index in cloud 7. Have you deployed the correct TA's for your syslog APP onto Splunk cloud.

deepakc

From your the logs it shows: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events (So run this and check the main index - if you can see this then your the connection is working. In terms of the /opt/sc4s/local/context/splunk_index.csv follow all the steps from the below run time configuration, there are a number of stpes and you need to complete them all. https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/getting-started-runtime-configuration/ As you can send curl test events to cloud you don't need whitelist (BUT its best practise to have them in place for security reasons.)

deepakc

Check some of the app permissions settings using the below, this may help troubleshoot - it sounds like a permissions issue. | rest splunk_server=local servicesNS/nobody/search/configs/conf-macros | search eai:acl.app=my_new_app

deepakc

A few things to check: 1. Have you enabled Whitelisting for HEC as this is cloud or are firewalls blocking. 2. Check logs journalctl -b -u sc4s 3. Check your all your indexs have been created in Splunk cloud. 4. Check the indexes are mapped /opt/sc4s/local/context/splunk_index.csv 5. Try basic testing using curl - create a token and use the below, may need some tuning https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/HTTPEventCollectortokenmanagement Use below example and change to your stack name curl "https://http-inputs.mysplunkserver.splunkcloud.com:8088/services/collector" \ -H "Authorization: Splunk CF179AE4-3C99-45F5-A7CC-3284AA91CF67" \ -d '{"event": "Hello, world!", "sourcetype": "manual"}'

deepakc

Try this | inputlookup userinfo | eval fourth_result=if(ExamID>=120 AND ExamID<=125,"GOOD","OTHER")

deepakc

This is an example using makeresults and rex | makeresults | eval _raw="Test1=101,Test2=102,Test3=103,Test4=104,Test5=105,Test6=106,Test7=107,Test8=108,Test9=109,Test101=110" | makemv _raw delim="," | rex field=_raw "(?<field>Test7)=(?<value>\d+)" | table field value

deepakc · ‎02-08-2018

Funny after logging onto the search heads, the app is installed and the add-on removed...

deepakc · ‎02-08-2018

Hi I have configured two search head in a cluster and Im trying to deploy the app - splunk-app-for-unix-and-linux_523.tgz bundle to them, but Im getting the error below. Dev Enviroment: Splunk 7.2 /RHEL 6.5 1 x master (runs deployer) 2 x indexers (clustered) 2 x search heads (clustered) I have deployed the ad-on splunk-add-on-for-unix-and-linux_524.tgz to all the splunk servers and can see the add-on in in the web console - looks OK. I have tested the search head cluster and can see the status as and looks OK. The command from the deployer: splunk apply shcluster-bundle -target https://omi10lindp1.ops.com:8089 -auth admin:password http://docs.splunk.com/Documentation/Splunk/7.0.2/DistSearch/PropagateSHCconfigurationchanges Error and in the splunkd.log is: 02-08-2018 09:25:59.510 +0000 WARN AppsDeployHandler - Error while deploying apps to first member: Error while deleting app=Splunk_TA_nix on target=https://192.168.0.130:8089: Non-200/201 status_code=500; {"messages":[{"type":"ERROR","text":"\n In handler 'localapps': Cannot update application info: /nobody/Splunk_TA_nix/app/install/state = disabled: Could not find writer for: /nobody/Splunk_TA_nix/app/install/state [0] [/opt/splunk/etc]"}]} Anyone come across this or similar issue? Dee

deepakc · ‎02-06-2018

Hi P_gurav Thanks for the tip. For some reason and after the overwrite of the files, the owner and group would show 506, the actual folder permissions looked the same when I compared them to the other peer node, so I ran chown (splunkuser) on the folders and and this time it went through. Again thanks 🙂

deepakc · ‎02-05-2018

Hi, In my dev environment I have (all 7.1 enterprise running on RHEL6.5) 1 x Master server 2 x Peer (index servers / cluster) 2 x Search heads I'm trying to upgrade them to 7.2. I have followed the upgrade documentation, everything was ok on all the other nodes in terms of upgrade steps, but one of the peer nodes failed with the below search. So I was wondering if anyone else has seen this or come across the same problem? splunk start --accept-license --answer-yes This appears to be an upgrade of Splunk. -------------------------------------------------------------------------------- ) Splunk has detected an older version of Splunk installed on this machine. To finish upgrading to the new version, Splunk's installer will automatically update and alter your current configuration files. Deprecated configuration files will be renamed with a .deprecated extension. You can choose to preview the changes that will be made to your configuration files before proceeding with the migration and upgrade: If you want to migrate and upgrade without previewing the changes that will be made to your existing configuration files, choose 'y'. If you want to see what changes will be made before you proceed with the upgrade, choose 'n'. Perform migration and upgrade without previewing configuration changes? [y/n] y -- Migration information is being logged to '/opt/splunk/var/log/splunk/migratio n.log.2018-02-05.17-41-23' -- Migrating to: VERSION=7.0.2 BUILD=03bbabbd5c0f PRODUCT=splunk PLATFORM=Linux-x86_64 Copying '/opt/splunk/etc/myinstall/splunkd.xml' to '/opt/splunk/etc/myinstall/sp lunkd.xml-migrate.bak'. An unforeseen error occurred: Exception: <type 'exceptions.IOError'>, Value: [Errno 13] Permission den ied: '/opt/splunk/etc/myinstall/splunkd.xml-migrate.bak' Traceback (most recent call last): File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1290 , in sys.exit(main(sys.argv)) File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1143 , in main parseAndRun(argsList) File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 998, in parseAndRun retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True) File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call return self.func(args, fromCLI) File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", l ine 30, in wrapperFunc return func(dictCopy, fromCLI) File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/_internal.py", lin e 183, in firstTimeRun migration.autoMigrate(args[ARG_LOGFILE], isDryRun) File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/migration.py", lin e 3091, in autoMigrate comm.copyItem(PATH_SPLUNKD_XML, PATH_SPLUNKD_XML_BAK, dryRun) File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", li ne 1008, in copyItem shutil.copy(src, dst) File "/opt/splunk/lib/python2.7/shutil.py", line 119, in copy copyfile(src, dst) File "/opt/splunk/lib/python2.7/shutil.py", line 83, in copyfile with open(dst, 'wb') as fdst: IOError: [Errno 13] Permission denied: '/opt/splunk/etc/myinstall/splunkd.xml-mi grate.bak' Please file a case online at http://www.splunk.com/page/submit_issue Any thoughts on this would be great. Regards Dee

deepakc · ‎01-02-2018

ok many thanks

deepakc · ‎01-02-2018

I was following the guidelines from this page on how to estimate the storage sizing as a learning excercise, but I do not seem to get any totals and I have uploaded some sample data into a index. The document is http://docs.splunk.com/Documentation/Splunk/7.0.1/Capacity/Estimateyourstoragerequirements Im using Splunk 7.1/RHEL6.5. Command output: cd /opt/splunk/var/lib/splunk/defaultdb sudo du -ch hot_v* du: cannot access `hot_v*': No such file or directory 0 total Any thoughts?

deepakc · ‎12-22-2017

I have Splunk 7.1 / RHEL65 / Test enviroment (New to splunk) I see you have Splunk Add-on for Apache Web Server, but do you still need a forwarder to forward the apache logs? Rgds Dee

deepakc · ‎12-20-2017

I have a lab Splunk 7.1 setup and I noticed in the help > about that hadoop is listed as a product. What does this mean, can anyone explain? rgds Dee

deepakc · ‎12-15-2017

I should have rebooted after uninstalling version 7.0 and then perform a clean install of version 7.1

deepakc · ‎12-15-2017

thanks for the pointers, by uninstalling it and reinstalling, then checking the permissions, its working now

deepakc · ‎12-15-2017

thanks I did checked the permissions, and it was correct as I could write to the folder as the user , I uninstalled it all, rebooted, reinstalled 7.1 and after it was ok... (so I must have made a mistake somewhere and did have version 7.0 installed, which I uninstalled but did not reboot) thanks for the pointers Dee

deepakc · ‎12-15-2017

Hi (I'm new to splunk ) Environment: Splunk 7.1 / RHEL6.5 I have create my own log file for test purposes / learning (its like a syslog log) and I can see the data in splunk. But when I try to create an index for it I get the error below, does anyone know what could be causing this ? ERROR MESSAGE: Data could not be written: /nobody/search/indexes/testosboot/thawedPath: $SPLUNK_DB/testosboot/thaweddb Rgds Dee

Posts	20
Solutions	0
Karma Given	0
Karma Received	0
Member Since	‎12-15-2017

Online Status	Offline
Date Last Visited	yesterday

App Bundle Install Error

Why am I encountering an issue on peer server when...

How do you estimate the storage sizing?

Splunk Add-on for Apache Web Server: If I have thi...

List of products - Hadoop

Cannot create Index

Re: Index rolling off data before retention age

Re: Splunk Connect for Syslog

Re: Splunk Connect for Syslog

Re: Splunk Connect for Syslog

Re: How do use the Splunk REST API to update macro...

Re: Splunk Connect for Syslog

Re: How to retrieve value from lookup for multival...

Re: Filed extraction

Re: App Bundle Install Error

App Bundle Install Error

Re: Why am I encountering an issue on peer server ...

Why am I encountering an issue on peer server when...

Re: How do you estimate the storage sizing?

How do you estimate the storage sizing?

Splunk Add-on for Apache Web Server: If I have thi...

List of products - Hadoop

Re: Cannot create Index

Re: Cannot create Index

Re: Cannot create Index

Cannot create Index