I have an issue where I have set up a Universal Forwarder on a Windows Azure server to monitor data stored on an Azure file share server. This is my inputs.conf: [monitor://\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20*.xml]
disabled = 0
index = kana
sourcetype = kana_xml
crcSalt = <SOURCE> The issue I have is Splunk thinks the CRC has changed each time the file is written to and re-ingests the whole file. The header of the file does not change, so I'm not sure why this happens. I read some other posts referring to how Azure file share caches data that changes metadata involved in the CRC calculation, but I'm not sure if that is definitely the case. Each file generates approx 6,000 events, but due to the re-ingestion this can amount to over a million events per file. Our license would get eaten up pretty quickly if I left the feed enabled constantly. Another knock on issue to this is when the log fills and a new file is created, Splunk doesn't see the new file and the data feed stops until the Splunk forwarder is restarted. It does however stop ingesting the previous file. Splunk's internal log shows the following details confirming it thinks the file is new: 10-05-2022 11:37:12.556 +0100 DEBUG TailReader [8280 tailreader0] - Defering notification for file=\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml by 3.000ms
10-05-2022 11:37:12.556 +0100 DEBUG TailReader [8280 tailreader0] - Finished reading file='\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3.000
10-05-2022 11:37:12.556 +0100 DEBUG WatchedFile [8280 tailreader0] - Reached EOF: fname=\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml fishstate=key=0x8908643efe7e891f sptr=865145 scrc=0x77aadaaeb3af22ee fnamecrc=0xbd1b79bedeae4211 modtime=1664963939
10-05-2022 11:37:12.556 +0100 DEBUG WatchedFile [8280 tailreader0] - seeking \\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml to off=857837
10-05-2022 11:37:12.524 +0100 DEBUG TailReader [8280 tailreader0] - About to read data (Reusing existing fd for file='\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml').
10-05-2022 11:37:12.524 +0100 INFO WatchedFile [8280 tailreader0] - Will begin reading at offset=0 for file='\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml'.
10-05-2022 11:37:12.524 +0100 INFO WatchedFile [8280 tailreader0] - Checksum for seekptr didn't match, will re-read entire file='\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml'.
10-05-2022 11:37:12.478 +0100 DEBUG TailReader [8280 tailreader0] - Will attempt to read file: \\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml from existing fd.
10-05-2022 11:37:12.478 +0100 DEBUG TailReader [8280 tailreader0] - Start reading file="\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml" in tailreader0 thread
10-05-2022 11:37:00.394 +0100 INFO Metrics - group=per_source_thruput, series="\\********.file.core.windows.net\kanaresponse\respshare\logs\log20221005_101607.xml", kbps=0.622, eps=0.064, kb=19.295, ev=2, avg_age=1134.000, max_age=2268
10-05-2022 11:36:48.567 +0100 DEBUG TailReader [5484 MainTailingThread] - Enqueued file=\\********.file.core.windows.net\KanaResponse\RespShare\logs\log20221005_101607.xml in tailreader0 If anyone has any ideas how to circumvent this issue, I'd be hugely grateful. I have tried using MonitorNoHandle, but that doesn't work as (a) Splunk wants the network drive location to be mapped to a drive, which we aren't able to do and (b) it requires individual files to be monitored, which we can't do easily as the new file uses the timestamp of when it is created in it's filename. Thanks
... View more
Hi @Smashley, did you ever manage to resolve this issue as I'm experiencing very similar behaviour with an XML KANA log. Splunk is continually re-ingesting the whole file every time a new entry is written to it. The internal log reports "seek crc didn't match" and "Checksum for seekptr didn't match, will re-read entire file" Thanks
... View more
UPDATE - I tried installing an older version of the installation software I had - 18.104.22.168 and it installed and Splunk started successfully. So while I haven't figured out the issue, I do at least have a working forwarder on my Solaris 10 server. I would still be interested to know if anyone knows why the older version works and the new one doesn't? According to the download page all versions of the UF software should work on Solaris 10 up to version 7.3.6. FURTHER UPDATE - 7.3.5 also fails on start up with the same error, however 22.214.171.124 installs and starts up fine.
... View more
Hi, We have an issue installing the Universal Forwarder software on Solaris 10 SPARC servers. splunkforwarder-7.3.6-47d8552a4d84-SunOS-sparc.tar.Z The following error occurs directly after accepting the license agreement and entering the Splunk admin username. ld.so.1: splunkd: fatal: libstdc++.so.6: open failed: No such file or directory OS info: SunOS 5.10 Generic_150400-65 sun4v sparc The libc.so version is SUNW_1.23 (as per Splunk Docs: universal forwarders on a Sun SPARC system that runs Solaris need a patch level of SUNW_1.22.7 or later of the C library (libc.so.1)). We have tried updating our LD_LIBRARY_PATH to include the location of the “missing” library. libstdc++.so.6 is located in /usr/sfw/lib on our servers, so we ran: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/sfw/lib but it didn't work. Any help or guidance would be appreciated. Thanks
... View more
We have 10 Saved Searches that we want to schedule to run in chronological order. We can do this using cron, but it is important that each search doesn't start until the previous search completes. This is because these searches output to the same lookup files, so we don't want to end up with different saved searches trying to update the same file at the same time.
In order to mitigate this scenario, I have been trying to create a simple control lookup table, which consists of a single column 'SS' with a single variable starting with '1', this will then update with a value of 1 through to 10.
When saved search 1 starts I want it to check the control lookup table for the value '1'. If it is '1' the search will run and update the output lookup file, otherwise, if the value isn't '1' the search won't run, or it can run but it won't find any events to update as the saved search value didn't match. Once the search has completed the control lookup table will update to the next saved search number (this part I can do quite easily with eval and outputlookup).
The problem I have is getting the control value passed into the main search, so far it won't recognise it. This is what I have tried:
| inputlookup append=t scheduleLookup.csv
| eval run=if(SS=1,"yes","no")
| search run=yes
| table _raw
This returns nothing, however, if I change the search to run=no, the event is found. The actual search does more data manipulation with evals than the above, but I changed it to table _raw to hopefully simplify the problem.
I'm sure I'm missing something simple, but any advice would be appreciated.
... View more
I have some BIG-IP data that I am ingesting as plain text files, as I can't directly connect to the BIG-IP servers due to security rules.
I have used regex field extractions to extract various data items at index-time, e.g. vs_pkts_in from props.conf:
EXTRACT-vs_pkts_in = VIRTUAL\s\w+\s+\w+\s+\w+\s+\|\s+\w+\s+\w+\s+\w+\s+\|\s+\(\w+,\s+\w+,\s+\w+,\s+\w+\)\s+=\s+\([\w.]+,\s+[\w.]+,\s+[\w.]+,\s+[\w.]+\)\s+\|\s+\(\w+,\w+\)\s+\w+\s+=\s+\((?P<vs_pkts_in>[\w.]+)
Unfortunately, this field value is recorded in various formats: 123, 1.23M, 1.23G and 1.23T. I have used the below search to convert the values into bytes, but so far I have been unable to work out how to apply this conversion at index-time, with the above extraction.
| makemv vs_pkts_in
| mvexpand vs_pkts_in
| rex field=vs_pkts_in "^(?<Value>[\d.]*)(?<Unit>[\w.]*)$"
| eval factor=case(Unit="B",1,Unit="K",1024,Unit="M",1024*1024,Unit="G",1024*1024*1024,Unit="T",1024*1024*1024*1024,true(),1)
| eval vs_pkts_in_bytes=Value*factor
I have approx 20 fields that this applies to, hence why I'd like to apply the conversion at index-time rather than search-time, as my search queries are very large.
Any help or advice on how to do this would be appreciated.
... View more