About m_zandinia

m_zandinia

Hi everyone, I have 3 indexers (in a cluster) located on Site A. The current replication factor (RF) is set to 3. I need to move operations to Site B. However, for technical reasons, I cannot physically move the existing data — I will simply have 3 new indexers at Site B. Here’s the plan I’m considering: Launch 1 new indexer at Site B. Add the new indexer to the existing cluster. Increase the RF to 4 (so that all raw data is fully replicated across the available indexers). Shut down all 3 indexers at Site A. Decrease the RF back to 3. (I understand there is a risk of some data loss.) Add 2 additional new indexers to the cluster at Site B. My main concern is step 5 — decreasing the RF — which I know is not best practice, but given my situation, I don't have many options. Has anyone encountered a similar situation before? I'd appreciate any advice, lessons learned, or other options I might not be aware of. Thanks in advance!

m_zandinia · ‎03-11-2024

Do you find any solution for this? I have some UFs that run with local system and they can send sysmon logs but I have some UFs that run with virtual account and therefore they can't send sysmon logs. I have the same message as you.

m_zandinia · ‎02-18-2024

I was able to fix my issue with symbolic links, thanks to the following topic. https://community.splunk.com/t5/Deployment-Architecture/How-to-move-index-from-one-hard-drive-to-another-in-Splunk/m-p/170733 Here is the steps I did: I created two directories on each volume, like this mkdir /Splunk-Storage/HOT/HOT1 mkdir /Splunk-Storage/HOT/HOT2 mkdir /Splunk-Storage/COLD/COLD1 mkdir /Splunk-Storage/COLD/COLD2 I stopped Splunk on one Indexer. Then moved the indexes to the appropriate directories as desired mv /Splunk-Storage/HOT/testindex1 /Splunk-Storage/HOT/HOT1/testindex1 mv /Splunk-Storage/COLD/testindex1 /Splunk-Storage/COLD/COLD1/testindex1 mv /Splunk-Storage/HOT/testindex2 /Splunk-Storage/HOT/HOT2/testindex2 mv /Splunk-Storage/COLD/testindex2 /Splunk-Storage/COLD/COLD2/testindex2 It took no time of course. Then I created symbolic links just like this ln -s /Splunk-Storage/HOT/HOT1/testindex1 /Splunk-Storage/HOT/testindex1 ln -s /Splunk-Storage/COLD/COLD1/testindex1 /Splunk-Storage/COLD/testindex1 ln -s /Splunk-Storage/HOT/HOT2/testindex2 /Splunk-Storage/HOT/testindex2 ln -s /Splunk-Storage/COLD/COLD2/testindex2 /Splunk-Storage/COLD/testindex2 Then I started Splunk. At this point, Splunk remained unaware of the changes occurring on the underlying file system, yet it continued to function, with the actual data now residing in the correct path. After repeating this process on all indexers, I proceeded to modify the indexes.conf on CM and pushed the changes. After checking that everything is correct, I removed the soft links.

m_zandinia · ‎11-19-2023

Yes. That is correct. I forgot about how I must manage the new indexes.conf and push it because I faced many issues when the indexes.conf on indexers doesn't match with indexes.conf on CM. I'll be very grateful If also @woodcock gives me an advice

m_zandinia · ‎11-18-2023

Hi Splunkers Currently, I have 8 indexers and about 100 indexes! Here is a sample of my indexes.conf: # volumes [volume:HOT] path = /Splunk-Storage/HOT maxVolumeDataSizeMB = 2650000 [volume:COLD] path = /Splunk-Storage/COLD maxVolumeDataSizeMB = 27500000 ### indexes ### [testindex1] repFactor = auto homePath = volume:HOT/testindex1 coldPath = volume:COLD/testindex1 thawedPath = /Splunk-Storage/COLD/testindex1/thaweddb summaryHomePath = /Splunk-Storage/HOT/testindex1/summary frozenTimePeriodInSecs = 47520000 [testindex2] repFactor = auto homePath = volume:HOT/testindex2 coldPath = volume:COLD/testindex2 thawedPath = /Splunk-Storage/COLD/testindex2/thaweddb summaryHomePath = /Splunk-Storage/HOT/testindex2/summary frozenTimePeriodInSecs = 47520000 I don't restrain my indexes by size, only by time. The current median age of all data is about 180 days. Regarding my fstab file: /dev/mapper/mpatha-part1 /Splunk-Storage/HOT xfs defaults 0 0 /dev/mapper/mpathb-part1 /Splunk-Storage/COLD xfs defaults 0 0 Now, for compliance reasons, I want to separate two of my indexes to preserve them for a longer duration (at least two years). I have considered two possible methods to accomplish this: 1. a. Create a different path and volume. b. Stop all indexers. c. Move the two indexes to the new path. d. Start all indexers. If I'm correct, the issue is that I can't move just two indexes because I didn't mount different paths in the OS. Therefore, I would have to move all other indexes to another path. Essentially, this means creating two paths and volumes in both my OS and indexes.conf. 2. a. Decrease the frozenTimePeriod for all indexes except the two to, for example, 150 days. b. Wait for Splunk to free up some disk space. c. Increase the frozenTimePeriod for those two indexes to, for example, 730 days. The second solution may seem more straightforward, but I'm uncertain if it is a best practice or a good idea at all. Could you please guide me on how to implement the first solution with minimal downtime? Thank you in advance for your assistance!

m_zandinia · ‎04-09-2023

Thanks @PickleRick Yes, I know the tricky part is how I shrink my underlying filesystem because at the end of the day it's about the space must be free and available at the hypervisor level.

m_zandinia · ‎04-09-2023

Thanks I do know this isn't gonna be easy! The RF=3 and SF=2. There is another way I am thinking. If I can acquire 160TB for COLD data maybe this approach is gonna work. 1. Down HOT to 1 TB (Because I just have 4 TB temporary and it will not work) 2. Letting Cluster to move data from HOT to COLD 3. Bringing up 4 new indexers with HOT=1TB and COLD=40TB 4. Rebalance the data to spread COLD data on 8 indexers. 5. Then down COLD to 20TB This is much more like @woodcock suggested. But because another problem I mentioned, the weird thing is the COLD area in just one Vmware file and the file itself can't be shrink! although I decrease the volume size in Splunk and shrink it in Linux, the infra team still have 40TB used space, so after all of this I have to delete the COLD space one by one and create it again with the size=20TB! Anyway, I suppose this is the safest approach. I appreciate your suggestion @PickleRick @woodcock

m_zandinia · ‎04-08-2023

Thanks for replies I actually missed some facts. First of all, my daily data ingestion is something about 3-4 TB per day. Total events, for example, on Wednesday, are around 3,500,000,000. There isn’t any free space in the HOT & COLD spaces. and there is only 80 TB temporarily that I can designate for COLD, and 4 TB for HOT. On the other hand, the current COLD space is just one raw Vmware file. Because of the type of COLD space, the Infra team says we can’t decrease the COLD space, and the server must go down, destroying the file, and designating again. Let’s assume we have 4 indexers with these names: indexer1, indexer2, indexer3, indexer4, and want to add indexer5, indexer6, indexer7, and indexer8. So, this is the procedure that I think about 1. Bringing up indexer5 to indexer8 and designate HOT=1TB and COLD=20TB from the temporary space 2. Bringing down indexer1 3. Rebalance the data and waiting to cluster being established 4. Bringing up indexer1 with HOT=1TB and COLD=20TB 5. Rebalance the data and waiting to cluster being established Repeat these steps until all the indexers are up. So the question is at what step I must reconfigure my volumes (in indexes.conf) down to HOT=1TB and COLD=20TB to avoid data loss? And do you think using the frozen path is a good idea to avoid data loss?

m_zandinia · ‎04-04-2023

Hi Splunkers I have 4 indexers in my cluster environment and 8 TB for hot volume, 4 TB for summaries, and 160 TB for cold buckets, totally. I don't have a freeze path. This is my indexes.conf as well. [volume:HOT] path = /Splunk-Storage/hot maxVolumeDataSizeMB = 1900000 # ~2 TB [volume:COLD] path = /Splunk-Storage/cold maxVolumeDataSizeMB = 40000000 # ~40 TB [volume:_splunk_summaries] path = /Splunk-Storage/splunk_summaries maxVolumeDataSizeMB = 950000 # ~1 TB Now, I want to add 4 indexers but I can't increase my volumes due to some limitations. So, I have to split my total space between 8 indexers instead of 4 indexers. What is your suggestion to avoid possible data loss and minimum down time?

m_zandinia · ‎12-24-2022

Hi, I collected the cisco deviceslog with "Cisco Networks Add-on for Splunk Enterprise". And install "Cisco Networks App for Splunk Enterprise" on my search heads. Now I want to know which user at what time execute what commands. In the App, Audit, "Configuration change transactions" shows me the time and user but for the "cmd" (command) just shows "!exec: enable" . It just shows the main command that enables user to enter other commands. Is it something wrong with my configs in Splunk or it's just about log level on Cisco devices? Tanks in advance

m_zandinia · ‎03-02-2022

Thanks @gcusello Is works perfectly when I set it on indexers. I set those settings on UF and because of that it didn't work.

m_zandinia · ‎03-02-2022

Hi Splunkers! I have a problem with props.conf and tranforms.conf I face with this error in Linux Servers. multipathd[212317]: sdb: failed to get sgio uid: No such file or directory multipathd[212317]: sdb: add missing path So I set props.conf and transforms.conf to get rid of this messages. It seems correct and I can't figure out why this doesn't work?! props.conf [syslog] TRANSFORMS-null = setnull transorms.conf [setnull] REGEX = multipathd DEST_KEY = queue FORMAT = nullQueue I also tried these REGEXs REGEX = .*multipathd.* REGEX = (.+multipathd.+) But nothing happened So where did I make a mistake? appreciate for your time

m_zandinia · ‎01-09-2022

Hi Splunkers. I have an indexer cluster and all of sudden all of them goes up and down and stuck in BatchAdding status. I have 4 indexers. These are my settings: [clustering] cluster_label = IndexerCluster mode = master rebalance_threshold = 0.95 replication_factor = 3 search_factor = 2 restart_timeout = 180 service_interval = 90 heartbeat_timeout = 180 cxn_timeout = 300 send_timeout = 300 rcv_timeout = 300 max_peer_build_load = 20 max_peer_rep_load = 50 max_fixup_time_ms = 0 maintenance_mode = false I increase max_peer_build_load to improve my fixup tasks but it doesn't work. I've followed the amount of buckets and it increases very slowly. I have this error in my splund.log file on indexers ERROR ProcessTracker - (child_581__Fsck) BucketBuilder - BucketBuilder::error: Event data size is 0. Raw and Meta data may be missing for bucket="/Splunk-Storage/HOT/eventlog-online-index/db_1641702441_1641656220_301" WARN ProcessTracker - (child_601__Fsck) Fsck - Repair entire bucket, index=eventlog-online-index, tryWarmThenCold=1, bucket=/Splunk-Storage/HOT/eventlog-online-index/db_1641702441_1641656220_301, exists=1, localrc=3, failReason=(entire bucket) Rebuild for bkt='/Splunk-Storage/HOT/eventlog-online-index/db_1641702441_1641656220_301' failed: BucketBuilder::error: Event data size is 0. Raw and Meta data may be missing for bucket="/Splunk-Storage/HOT/eventlog-online-index/db_1641702441_1641656220_301" On the other hand I face with crash.log file on my indexers continuously Received fatal signal 8 (Floating point exception). Cause: Integer division by zero at address [0x0000557E03DBB1D9]. Crashing thread: indexerPipe Registers: RIP: [0x0000557E03DBB1D9] _ZN12HotDBManager19computeBucketMapKeyERK15CowPipelineData + 121 (splunkd + 0xEF91D9) RDI: [0x00007F43D73836D0] RSI: [0x00007F43ABDAA72D] RBP: [0x00007F43C022EB40] RSP: [0x00007F43C07FD5A0] RAX: [0x07AC58C70206CAB3] RBX: [0x07AC58C70206CAB3] RCX: [0x0000000000000000] RDX: [0x0000000000000000] R8: [0x00000000000000B8] R9: [0x00007F43C8F3E060] R10: [0x00007F43D73867D0] R11: [0x00007F43D6200080] R12: [0x00007F43D7385E08] R13: [0x00007F43C07FD5F0] R14: [0x00007F43C02148E0] R15: [0x00007F43B6C2B500] EFL: [0x0000000000010246] TRAPNO: [0x0000000000000000] ERR: [0x0000000000000000] CSGSFS: [0x002B000000000033] OLDMASK: [0x0000000000000000] OS: Linux Arch: x86-64 Backtrace (PIC build): [0x0000557E03DBB1D9] _ZN12HotDBManager19computeBucketMapKeyERK15CowPipelineData + 121 (splunkd + 0xEF91D9) [0x0000557E03DBCFDA] _ZN12HotDBManager15_suitableBucketERK15CowPipelineDatalRblR3Str + 410 (splunkd + 0xEFAFDA) [0x0000557E03DBF018] _ZN12HotDBManager10suitableDbERK15CowPipelineDatalRblR3Str + 24 (splunkd + 0xEFD018) [0x0000557E03E1AF53] _ZN11IndexWriter11_dbLazyLoadERK15CowPipelineDatall + 131 (splunkd + 0xF58F53) [0x0000557E03E1C054] _ZN11IndexWriter14write_internalER15CowPipelineDatalRP8DBBucketb + 308 (splunkd + 0xF5A054) [0x0000557E03E1C8D7] _ZN11IndexWriter10write_implER15CowPipelineDatalb + 103 (splunkd + 0xF5A8D7) [0x0000557E03E1CC43] _ZN11IndexWriter5writeER15CowPipelineDatal + 19 (splunkd + 0xF5AC43) [0x0000557E03E1404F] _ZN14IndexProcessor7executeER15CowPipelineData + 3951 (splunkd + 0xF5204F) [0x0000557E0433F585] _ZN9Processor20executeMultiLastStepER18PipelineDataVector + 101 (splunkd + 0x147D585) [0x0000557E03B2ABCA] _ZN8Pipeline4mainEv + 1418 (splunkd + 0xC68BCA) [0x0000557E048FD9D8] _ZN6Thread8callMainEPv + 120 (splunkd + 0x1A3B9D8) [0x00007F43D67D6609] ? (libpthread.so.0 + 0x2609) [0x00007F43D66FD263] clone + 67 (libc.so.6 + 0xFD263) Linux / indexer1-datacenter / 5.4.0-92-generic / #103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021 / x86_64 /etc/debian_version: bullseye/sid Last errno: 2 Threads running: 72 Runtime: 8.643140s argv: [splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd] Regex JIT enabled RE2 regex engine enabled using CLOCK_MONOTONIC Thread: "indexerPipe", did_join=0, ready_to_run=Y, main_thread=N First 8 bytes of Thread token @0x7f43c2118e10: 00000000 00 e7 7f c0 43 7f 00 00 |....C...| 00000008 x86 CPUID registers: 0: 00000016 756E6547 6C65746E 49656E69 1: 00050657 08400800 7FFEFBFF BFEBFBFF 2: 76036301 00F0B5FF 00000000 00C30000 3: 00000000 00000000 00000000 00000000 4: 00000000 00000000 00000000 00000000 5: 00000040 00000040 00000003 00002020 6: 00000AF7 00000002 00000009 00000000 7: 00000000 00000000 00000000 00000000 8: 00000000 00000000 00000000 00000000 9: 00000000 00000000 00000000 00000000 A: 07300404 00000000 00000000 00000603 B: 00000000 00000000 0000002F 00000008 C: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 E: 00000000 00000000 00000000 00000000 F: 00000000 00000000 00000000 00000000 10: 00000000 00000000 00000000 00000000 11: 00000000 00000000 00000000 00000000 12: 00000000 00000000 00000000 00000000 13: 00000000 00000000 00000000 00000000 14: 00000000 00000000 00000000 00000000 15: 00000002 000000F0 00000000 00000000 16: 00000BB8 00000FA0 00000064 00000000 80000000: 80000008 00000000 00000000 00000000 80000001: 00000000 00000000 00000121 2C100800 80000002: 65746E49 2952286C 6F655820 2952286E 80000003: 6C6F4720 32362064 20523834 20555043 80000004: 2E332040 48473030 0000007A 00000000 80000005: 00000000 00000000 00000000 00000000 80000006: 00000000 00000000 01006040 00000000 80000007: 00000000 00000000 00000000 00000100 80000008: 0000302E 00000000 00000000 00000000 terminating... My OS is Ubuntu server 20.04. Any suggestion? Can I bring up one indexer outside of my cluster to prevent log drop and after the cluster will be stable join it to cluster?

m_zandinia · ‎12-05-2021

Hi everyone, I have 3 indexers with this specification 1.7 TB local ssd for HOT & WARM Buckets 1.7 TB local ssd for data models 27 TB SAN Storage for COLD buckets replication factor: 3 search factor: 2 Now I want to add 2 indexers which have local ssd just like other servers but for now I can't provide SAN Storage for COLD Buckets. So as I mentioned and because of my RF which is 3, my cold buckets spread exactly as my settings. My question is: Is there a problem to add my new indexers now and after a while add SAN Storage to them?

m_zandinia · ‎12-05-2021

It's a good question. If you find out how can find which user have stopped a service please let me know. Thanks

m_zandinia · ‎09-29-2021

Hi Splunkers! I hope you all are doing well. This is my indexes.conf My problem is that the COLD volume was fulled. This is the output of df command The fs of COLD volume is xfs Do you know that the total maxsize of both COLD and splunk_summareis must not exceed from total space or Just setting the COLD volume is enough because the splunk_summaries volume is part of that? I mean in my case Splunk set the addition of both volume:COLD and volume:_splunk_summaries for total space for storing buckets or just set the maxVolumesize of volume:COLD config? Thanks in advance for any advice PS: I know Splunk do recommend that the summaries must be stored in HOT volume!

m_zandinia · ‎09-18-2021

Thanks for your time. It's worked perfectly. Just I have a misspelling in my post. I've corrected it so you do. MAX_TIMESTAMP_LOOAKAHEAD = 16 MAX_TIMESTAMP_LOOKAHEAD = 16

m_zandinia · ‎09-18-2021

Hi Splunkers! I have a problem with line breaking in Splunk add-on F5-bigip. I've tried some regex to break the line correctly but I'm not successful. First of all for simplicity I changed my outputs.conf in Heavy Forwarder. outputs.conf [indexAndForward] index = true In fact the indexing is false on this node and this HF forward data to my indexer cluster and I also have search head cluster. But as I mentioned just for simplicity I turned mu indexing to true in this HF. Then I used these regexes to break the lines props.conf [f5:bigip:syslog] # LINE_BREAKER = ^()\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2} # LINE_BREAKER = ^\w{3}\s\d*\s\d{2}\W\d{2}\W\d{2} LINE_BREAKER = ([\r\n]+)\w{3}\s\d+\s\d{2}\W\d{2}\W\d{2} # LINE_BREAKER = ([\r\n]+) # LINE_BREAKER = \n MAX_TIMESTAMP_LOOKAHEAD = 16 # ADD_EXTRA_TIME_FIELDS = subseconds NO_BINARY_CHECK = true # EVENT_BREAKER_ENABLE = false # TIME_FORMAT = %b %d %H:%M:%S TIME_PREFIX = SHOULD_LINEMERGE = false TRUNCATE = 1000000 This is some of my data that I can't break the line correctly. Sep 18 19:12:27 192.168.1.1 Sep 18 14:42:27 F5-LTM-3.company.local info logger[25169]: [ssl_req][18/Sep/2021:14:42:27 +0000] 1.1.1.1 TLSv1.2 ECDHE-RSA-AES128-SHA "/mgmt/shared/inflate/available" 2 Sep 18 19:12:28 192.168.1.1 Sep 18 14:42:28 F5-LTM-3.company.local ASM:unit_hostname="F5-LTM-3.company.local",management_ip_address="192.168.1.1",management_ip_address_2="",http_class_name="/Common/Online",web_application_name="/Common/Online",policy_name="/Common/Online",policy_apply_date="2021-08-26 10:13:52",violations="Illegal redUSection attempt",support_id="13616148673914804247",request_status="alerted",response_code="302",ip_client="1.1.1.1",route_domain="0",method="GET",protocol="HTTP",query_string="",x_forwarded_for_header_value="1.1.1.1",sig_ids="",sig_names="",date_time="2021-09-18 14:42:28",severity="Error",attack_type="Other Application Activity",geo_location="US",ip_address_intelligence="N/A",username="N/A",session_id="0",src_port="44180",dest_port="80",dest_ip="1.1.1.1",sub_violations="",vUSus_name="N/A",violation_rating="3",websocket_dUSection="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="",staged_threat_campaign_names="",blocking_exception_reason="N/A",captcha_result="not_received",microservice="",vs_name="/Common/OnlineShared3-HTTP",uri="/account/login",fragment="",request="GET /Account/Login HTTP/1.1\r\nConnection: keep-alive\r\nHost: example.com\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.8,en-US;q=0.5,en;q=0.3\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: Mozilla/5.0 (Android 7.0; Mobile; rv:68.0) Gecko/68.0 FUSefox/68.0\r\nUpgrade-Insecure-Requests: 1\r\nX-Forwarded-For: 1.1.1.1\r\n\r\n",response="Response logging disabled" Sep 18 19:12:28 192.168.1.1 Sep 18 14:42:28 F5-LTM-3.company.local ASM:unit_hostname="F5-LTM-3.company.local",management_ip_address="192.168.1.1",management_ip_address_2="",http_class_name="/Common/Online",web_application_name="/Common/Online",policy_name="/Common/Online",policy_apply_date="2021-08-26 10:13:52",violations="Illegal redUSection attempt",support_id="13616148673951684370",request_status="alerted",response_code="302",ip_client="1.1.1.1",route_domain="0",method="GET",protocol="HTTP",query_string="",x_forwarded_for_header_value="1.1.1.1",sig_ids="",sig_names="",date_time="2021-09-18 14:42:28",severity="Error",attack_type="Other Application Activity",geo_location="US",ip_address_intelligence="N/A",username="N/A",session_id="0",src_port="19338",dest_port="80",dest_ip="1.1.1.1",sub_violations="",vUSus_name="N/A",violation_rating="3",websocket_dUSection="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="",staged_threat_campaign_names="",blocking_exception_reason="N/A",captcha_result="not_received",microservice="",vs_name="/Common/OnlineShared3-HTTP",uri="/account/login",fragment="",request="GET //Account/Login HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 FUSefox/92.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: keep-alive\r\nCookie: GuidedTourVersion=1; SiteVersion=3.7.6; __utma=226054936.2062308401.1625890970.1631960683.1631966584.238; __utmz=226054936.1625890970.1.1.utmcsr=(dUSect)|utmccn=(dUSect)|utmcmd=(none); crisp-client%2Fsession%2Fbb1636a8-4b45-4fbb-971e-d5e50e2a1e1f=session_230233c6-895e-42d0-b257-4ae4c1903150; _hjid=b846f33d-e2e6-4c9a-a757-f9ab405b0193; Token=6abe8980-5856-4d6f-b05a-2915b970983e; lastmessage-6=87696; lastmessage-4=1; lastmessage-2=undefined; text0_1567617252=true; text0_496056564=true; .ASPXAUTH=4A5473E3674D47ED86E8EA52D6A4613C2F30F1D31A41DF7F8BEDBAB120DE5ACEB8E3DD46D71 Sep 18 19:12:29 192.168.1.1 Sep 18 14:42:29 F5-LTM-3.company.local ASM:unit_hostname="F5-LTM-3.company.local",management_ip_address="192.168.1.1",management_ip_address_2="",http_class_name="/Common/Online",web_application_name="/Common/Online",policy_name="/Common/Online",policy_apply_date="2021-08-26 10:13:52",violations="Illegal redUSection attempt",support_id="13616148673926887289",request_status="alerted",response_code="302",ip_client="1.1.1.1",route_domain="0",method="GET",protocol="HTTP",query_string="",x_forwarded_for_header_value="1.1.1.1",sig_ids="",sig_names="",date_time="2021-09-18 14:42:29",severity="Error",attack_type="Other Application Activity",geo_location="US",ip_address_intelligence="N/A",username="N/A",session_id="0",src_port="46453",dest_port="80",dest_ip="1.1.1.1",sub_violations="",vUSus_name="N/A",violation_rating="3",websocket_dUSection="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="",staged_threat_campaign_names="",blocking_exception_reason="N/A",captcha_result="not_received",microservice="",vs_name="/Common/OnlineShared3-HTTP",uri="/account/login",fragment="",request="GET //Account/login HTTP/1.1\r\nHost: example.com\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Linux; Android 7.1.1; SM-J510F Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/64.0.3282.137 Mobile Saenri/537.36 AgentWeb/4.1.3 UCBrowser/1.1.1.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en,en-US;q=0.9,en-GB;q=0.8,en-US;q=0.7\r\nX-Requested-With: com.sefryekcompany.mobiletradingpro\r\nX-Forwarded-For: 1.1.1.1\r\n\r\n",response="Response logging disabled" Sep 18 19:12:29 192.168.1.1 Sep 18 14:42:29 F5-LTM-3.company.local ASM:unit_hostname="F5-LTM-3.company.local",management_ip_address="192.168.1.1",management_ip_address_2="",http_class_name="/Common/Online",web_application_name="/Common/Online",policy_name="/Common/Online",policy_apply_date="2021-08-26 10:13:52",violations="Illegal redUSection attempt",support_id="13616148673919202912",request_status="alerted",response_code="301",ip_client="1.1.1.1",route_domain="0",method="GET",protocol="HTTPS",query_string="37419741",x_forwarded_for_header_value="1.1.1.1",sig_ids="",sig_names="",date_time="2021-09-18 14:42:29",severity="Error",attack_type="Other Application Activity",geo_location="US",ip_address_intelligence="N/A",username="N/A",session_id="f8689163755118a6",src_port="44760",dest_port="443",dest_ip="1.1.1.1",sub_violations="",vUSus_name="N/A",violation_rating="3",websocket_dUSection="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="",staged_threat_campaign_names="",blocking_exception_reason="N/A",captcha_result="not_received",microservice="",vs_name="/Common/OnlineShared1-HTTPS",uri="/serviceworker.js",fragment="",request="GET /serviceworker.js?37419741 HTTP/1.1\r\nHost: example.com\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nAccept: */*\r\nSave-Data: on\r\nService-Worker: script\r\nSec-Fetch-Site: same-origin\r\nSec-Fetch-Mode: same-origin\r\nSec-Fetch-Dest: serviceworker\r\nReferer: https://mobile.bmibourse.com/serviceworker.js?37419741\r\nUser-Agent: Mozilla/5.0 (Linux; Android 10; SM-A207F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Mobile Saenri/537.36\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: en-US,en;q=0.9,en-US;q=0.8,en;q=0.7\r\nCookie: basket-warning-readed=1; basket-option-visited=true; tag-market-map-visited=true; index-technical-visited=true; stock-technical-visited=true; AppVersion=1.1.2; TS01e42c80=0180bb6f222b77a4b3dd30e3eddfc570acb1a0674cc23f80304088a610b57e5e43c686eb7415c18bc949724b74a1f77b7746en6cd8\r\nX-Forwarded-For: 5.116.208 Sep 18 19:12:29 192.168.1.1 Sep 18 14:42:29 F5-LTM-3.company.local ASM:unit_hostname="F5-LTM-3.company.local",management_ip_address="192.168.1.1",management_ip_address_2="",http_class_name="/Common/Online",web_application_name="/Common/Online",policy_name="/Common/Online",policy_apply_date="2021-08-26 10:13:52",violations="Illegal redUSection attempt",support_id="13616148673963109971",request_status="alerted",response_code="301",ip_client="1.1.1.1",route_domain="0",method="GET",protocol="HTTPS",query_string="37418741",x_forwarded_for_header_value="1.1.1.1",sig_ids="",sig_names="",date_time="2021-09-18 14:42:29",severity="Error",attack_type="Other Application Activity",geo_location="US",ip_address_intelligence="N/A",username="N/A",session_id="80b2664635b96eeb",src_port="41628",dest_port="443",dest_ip="1.1.1.1",sub_violations="",vUSus_name="N/A",violation_rating="3",websocket_dUSection="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="",staged_threat_campaign_names="",blocking_exception_reason="N/A",captcha_result="not_received",microservice="",vs_name="/Common/OnlineShared3-HTTPS",uri="/serviceworker.js",fragment="",request="GET /serviceworker.js?37418741 HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Android 7.0; Mobile; rv:68.0) Gecko/68.0 FUSefox/68.0\r\nAccept: */*\r\nAccept-Language: en-US,en;q=0.8,en-US;q=0.5,en;q=0.3\r\nAccept-Encoding: gzip, deflate, br\r\nService-Worker: script\r\nConnection: keep-alive\r\nCookie: _ga=GA1.2.1098137509.1594471619; basket-warning-readed=1; basket-option-visited=true; AppVersion=1.1.2; index-technical-visited=true; tag-market-map-visited=true\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nX-Forwarded-For: 1.1.1.1\r\nSSLcompany: 1\r\n\r\n",response="Response logging disabled" Sep 18 19:12:29 192.168.1.1 Sep 18 14:42:29 F5-LTM-3.company.local ASM:unit_hostname="F5-LTM-3.company.local",management_ip_address="192.168.1.1",management_ip_address_2="",http_class_name="/Common/Online",web_application_name="/Common/Online",policy_name="/Common/Online",policy_apply_date="2021-08-26 10:13:52",violations="Illegal redUSection attempt",support_id="13616148673952377578",request_status="alerted",response_code="301",ip_client="1.1.1.1",route_domain="0",method="GET",protocol="HTTPS",query_string="37418741",x_forwarded_for_header_value="1.1.1.1",sig_ids="",sig_names="",date_time="2021-09-18 14:42:29",severity="Error",attack_type="Other Application Activity",geo_location="US",ip_address_intelligence="N/A",username="N/A",session_id="3915b37e523c6d41",src_port="55434",dest_port="443",dest_ip="1.1.1.1",sub_violations="",vUSus_name="N/A",violation_rating="3",websocket_dUSection="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="",staged_threat_campaign_names="",blocking_exception_reason="N/A",captcha_result="not_received",microservice="",vs_name="/Common/OnlineShared2-HTTPS",uri="/serviceworker.js",fragment="",request="GET /serviceworker.js?37418741 HTTP/1.1\r\nHost: example.com\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nAccept: */*\r\nService-Worker: script\r\nX-Requested-With: com.sefryekcompany.mobiletradingpro\r\nSec-Fetch-Site: same-origin\r\nSec-Fetch-Mode: same-origin\r\nSec-Fetch-Dest: serviceworker\r\nReferer: https://mobile.mobinsb.com/serviceworker.js?37418741\r\nUser-Agent: Mozilla/5.0 (Linux; Android 10; SM-A600G Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/81.0.4044.138 Mobile Saenri/537.36\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-US,en;q=0.9,en-CA;q=0.8,en-US;q=0.7,en;q=0.6\r\nCookie: companyRLCUrl=////////////////////////////////////core.companyrlc.com/; companyRLApiUrl=//rlcchartapi.companyrlc.com/; BrokerId=777; ThemeName=MobinSarmayeh; DisabledModules=changebroker; PushSubDomainName=push2v7.company.co Thanks in advance

m_zandinia · ‎08-14-2021

I could solve my problem with this solution navigate to $SPLUNK_HOME\etc\apps\splunk_httpinput\local Edit the inputs.conf file and increase the maxEventSize That's it!

m_zandinia · ‎08-11-2021

Hi Splunkers I've tried to read some data from MS SQL Server. The data is json like. It works for a while and then I encounter with this message: ERROR HttpInputDataHandler - Failed processing http input, token name=db-connect-http-input, channel=n/a, source_IP=127.0.0.1, reply=6, events_processed=802, http_input_body_size=11904838 ERROR HttpInputDataHandler - Parsing error : While expecting event's raw text: String value too long. valueSize=5246755, maxValueSize=5242880, totalRequestSize=11904838 After that no data getting in. Is there any way to increase the maxValueSize? or my problem is originated from elsewhere Thanks in advance

m_zandinia · ‎07-25-2021

Thank you for your suggestion. You're right. I do prefer to have more indexers instead of just having 2 huge servers!

m_zandinia · ‎07-25-2021

Hi Splunkers! I currently use 3 indexers in order to ingest my data and respond to search jobs. We use ES in our deployment. My indexers' hardware is 3 DL38 G7 with 12 physical core and 128GB of RAM. The daily ingested data is 500GB/day although sometimes the ingested data was been over 1.5 TB/day! but this has happened very few times. I have problems with my ES as the correlation searches always get delayed because of the lack of CPU on my indexers. Now my company has decided to upgrade the indexers. They suggest me 2 DL560 G10 with 192 physical core and 1.5TB of RAM. That's great! Isn't it? My only concern is that in my current deployment if one of my indexers goes down I have 2 indexers to ingest data and responding to search jobs but if I replace the old servers with new servers then if one of my indexers goes down I just have one indexer. So what's your professional recommendation in my circumstances?

Posts	22
Solutions	1
Karma Given	11
Karma Received	1
Member Since	‎07-25-2021

Online Status	Offline
Date Last Visited	2 weeks ago

Moving Indexers to a New Site with RF Adjustments

Move some indexes to separate volume

How to reduce hot and cold volume safely?

Getting in the commands from cisco devices

How to use config to get rid of multipath error?

Indexers BatchAdding problem

indexers cold buckets path

Splunk summery volume

F5 BIG-IP linebreaking

DB Connect Failed processing http input

Moving Indexers to a New Site with RF Adjustments

Re: Sysmon events not getting indexed

Re: Move some indexes to separate volume

Re: Move some indexes to separate volume

Move some indexes to separate volume

Re: How to reduce hot and cold volume safely?

Re: How to reduce hot and cold volume safely?

Re: How to reduce hot and cold volume safely?

How to reduce hot and cold volume safely?

Getting in the commands from cisco devices

Re: config to get rid of multipath error

How to use config to get rid of multipath error?

Indexers BatchAdding problem

indexers cold buckets path

Re: How to find users who stopped a windows servic...

Splunk summery volume

Re: F5 BIG-IP linebreaking

F5 BIG-IP linebreaking

Re: DB Connect Failed processing http input

DB Connect Failed processing http input

Re: Hardware recommended for indexers

Hardware recommended for indexers

Are you a member of the Splunk Community?