"bucket" is a search command and could be skewing your results. try renaming your "bucket" to "mybucket" and see if that helps. ... View more

There is a great sizing estimator over at https://splunk-sizing.appspot.com. Based on the information you have provided, you would need a minimum of 13.7 TB of storage per indexer (see here). ... View more

It seems to work pretty consistently for me for both values, but I'm using pretty small samples (~150 events). You may need to tweak the time_window, for the counting, but give it a try. ... View more

This is a great guide to setting up syslog with Splunk. Enjoy! Splunk Success with Syslog ... View more

I would recommend adding more CPUs to your search head before you go adjusting those limits. You'll be happier in the long run. ... View more

Streamstats seems to work the way you are thinking. <your search> | reverse| streamstats time_window=5m count by id | table _time id count I put the reverse in to get the events in the right order. It produces this outcome which looks like it lines up with your expectations. _time id count 2020-01-09 12:00:00 A 1 2020-01-09 12:01:00 B 1 2020-01-09 12:02:00 A 2 2020-01-09 12:03:00 A 3 2020-01-09 12:04:00 A 4 2020-01-09 12:05:00 A 4 2020-01-09 12:06:00 A 5 2020-01-09 12:07:00 A 5 2020-01-09 12:08:00 A 5 2020-01-09 12:09:00 A 5 2020-01-09 12:10:00 B 1 2020-01-09 12:11:00 A 4 ... View more

The forwarders use an internal load balancing mechanism to determine the target for sending their data. The load balancing mechanism is not configurable. Ideally, your indexers should all be identical -- same amount of memory; same number of CPUs; same amount of disk capacity for OS, splunk, hot/warm, and cold. For best planning, based on those storage specifications, you should plan your indexes and retention for the lowest storage amount of 4.3TB, or a total cluster capacity of 21.5 TB (minus the needed storage for your replication factor). You are already seeing the impact of the inconsistent sizing with the errors noted in question 3. It's not possible to determine if the retention of 1 year is problematic without the details of how your indexes are stored. I would strongly recommend talking this over with 1) your Splunk Sales Engineer, 2) your local user group experts, or 3) Splunk Professional Services. In the meantime, please read up on how Splunk clustering works here. ... View more

Yes, this is possible. I copy/pasted your small snippet of log and set up an ingestion into Splunk and extracted the fields you had identified (the field extraction might need some tweaking depending on your usage). Props.conf is as follows: [phonelog] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Custom pulldown_type = true EXTRACT-transaction_id,user_id,state = ^[^\(\n]*\((?P<transaction_id>[^\)]+)[^:\n]*:\s+(?P<user_id>\w+)[^\(\n]*\((?P<state>\w+) One you have the data extraction you should be able to find the duration of calls with the transaction command. It will look something like: your_search | transaction state startswith="OffHook" endswith="OnHook" Hope that helps! ... View more

with that many forwarders, you should be (IMHO) using GIT for managing your configs. and then you deploy the entire set of configs to all of the DS. think of managing them as a whole. And with a deployment of that size, you should be talking to your Splunk Field Architect. ... View more

The DS is a single-threaded application and does not scale well vertically. Adding more CPU/Memory to a DS is not going to help much. You'll need about 1 DS per (approximately) 5000 hosts. and you'll want to put them behind a load balancer. the hardware requirements are pretty low. we have our sized at 8CPU/32GB of memory, but you can likely use smaller servers. I would recommend testing various sizing configs to see what works well for your environment. ... View more

You can use syslog to forward those Linux OS logs to an external syslog server (which could be hosted on your HF or elsewhere). Once that is done, set up a UF/HF to foward those logs to your intermediate forwarding layer. ... View more

Put the server.conf into an app and deploy it from your cluster master just like any other app you’d deploy. However it’s generally not recommended to run a kvstore on your indexers. ... View more

Try this: | rex field=_raw "Dest\s*:\s(?P<myfield>.*)" ... View more

Adding the Splunk Blog post from Feb 12, 2018. https://www.splunk.com/blog/2018/02/12/meltdown-patch-and-the-impact-on-infrastructure-supporting-splunk-solutions.html ... View more

There are several solutions for sending SNMP traps from Splunk. Here's one over on Splunkbase that is pretty basic. Here is another. ... View more

Adding a link to RedHat's Speculative Execution Exploit Performance Impacts - Describing the performance impacts to security patches for CVE-2017-5754 CVE-2017-5753 and CVE-2017-5715 https://access.redhat.com/articles/3307751 ... View more

It looks like a new line is the event breaker. Splunk should detect that on it's own, but you can try: LINE_BREAKER = ([\r\n]+) ... View more

We saw something similar with the EMC Isilon app. we had to make an update to correct the scoping of the authhandlers.py python script to just the app. in line 38 of the authhandlers.py script it was: entities = entity.getEntities(['admin', 'passwords'], namespace=myapp, owner='nobody', sessionKey=sessionKey) we changed it to: entities = entity.getEntities(['admin', 'passwords'], search=myapp, namespace=myapp, owner='nobody', sessionKey=sessionKey) and now it works. See if that helps. Jim ... View more

Have you looked in the _internal logs to see what's happening? Are there any events like: 04-14-2017 10:03:46.851 -0400 ERROR TcpOutputFd - Connection to host=Indexer1:9997 failed I also noticed that Indexer2 has the FQDN whereas Indexer1 does not. can your HF resolve both hostnames as you have them specified? And are there any firewalls in between the HF and the indexers? ... View more

I do charge back for Splunk at $WORK. This is the process I've developed to present the information to the respective data owners and to my administration team. I have a lookup table which contains the name of the index with the expected daily ingestion (a.k.a. "what they paid for"). The lookup table looks like this: "daily_limit",idx ,"_audit" ,"_internal" ,"_introspection" "0.002",bidata ,"anomaly_detection" "1.5",app 1,array "1.4",avaya Then I have a scheduled report run every morning to generate a report with the daily usage versus what their limits. index=_internal source=*license_usage.log type="Usage" idx=* pool="Production" | bucket span=1d _time | stats sum(b) as b by _time idx | eval daily_usage_mb=b/1024/1024 | lookup index_limits idx OUTPUT daily_limit as daily_limit_gb | eval daily_limit_mb=daily_limit_gb*1024 | eval warning=daily_limit_mb*.8 | eval Alert=case(daily_usage_mb>=daily_limit_mb, "1 overage", daily_usage_mb>=warning, "2 warning", 1==1, "3 ok") | table idx Alert daily_limit_mb daily_usage_mb | sort Alert, idx | fieldformat daily_usage_mb=tostring(daily_usage_mb,"commas") | fieldformat daily_limit_mb=tostring(daily_limit_mb,"commas") In order to provide the same data to the data owners, I create a summary search of the license usage that I can present to the data owners. index=_internal source=*license_usage.log type="Usage" idx=* | eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | stats sum(b) as volumeB by idx, pool| eval volumeMB=round(b/1024/1024,3) I also provide an alert to the data owners if their usage the previous day exceeds 80% of their usage. index=summary search_name="Summarize Daily License Usage by Index" idx=app| eval volumeMB=round(volumeB/1024/1024,3) | lookup index_limits idx OUTPUT daily_limit | timechart span=1d sum(volumeMB) avg(eval(daily_limit*1000)) as daily_limit_MB And finally I also provide a report to each group for them to monitor their 30 day usage against their expected daily ingestion: index=summary search_name="Summarize Daily License Usage by Index" idx=app| eval volumeMB=round(volumeB/1024/1024,3) | lookup index_limits idx OUTPUT daily_limit | timechart span=1d sum(volumeMB) avg(eval(daily_limit*1000)) as daily_limit_MB Hope that helps. Jim ... View more

I think the "sendemail" command is what you're looking for. https://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Sendemail ... View more