I am trying to track file transfers from one location to another. Flow: Files are copied to File copy location -> Target Location Both File copy location and Target location logs are in the same i...
See more...
I am trying to track file transfers from one location to another. Flow: Files are copied to File copy location -> Target Location Both File copy location and Target location logs are in the same index but each has it own sourcetype. File copy location events has logs for each file but Target location has a logs which has multiple files names. Log format of filecopy location: 2024-12-18 17:02:50 , file_name="XYZ.csv", file copy success 2024-12-18 17:02:58, file_name="ABC.zip", file copy success 2024-12-18 17:03:38, file_name="123.docx", file copy success 2024-12-18 18:06:19, file_name="143.docx", file copy success Log format of Target Location: 2024-12-18 17:30:10 <FileTransfer status="success> <FileName>XYZ.csv</FileName> <FileName>ABC.zip</FileName> <FileName>123.docx</FileName> </FileTransfer> Desired result: File Name FileCopyLocation Target Location XYZ.csv 2024-12-18 17:02:50 2024-12-18 17:30:10 ABC.zip 2024-12-18 17:02:58 2024-12-18 17:30:10 123.docx 2024-12-18 17:03:38 2024-12-18 17:30:10 143.docx 2024-12-18 18:06:19 Pending I want to avoid join.
Thanks, @bowesmana . Q - "When you say fuzzy, do you mean it should match based on similarity using something like Levenshtein distance? Do you want 123 main street 123 maine street 123 cain st...
See more...
Thanks, @bowesmana . Q - "When you say fuzzy, do you mean it should match based on similarity using something like Levenshtein distance? Do you want 123 main street 123 maine street 123 cain street all to match." A - No. I know about Levenshtein ; however, the similarity would have to disregard (not the correct word) the street numbers in counting/calculating. 123 main street and 124 main street would never be a match. 123 main street and 123 main street apt 2 would be a match. It is assumed, and probably incorrectly, the property owner of 123 main street apt 4 and 123 main street apt 6 are the same for the building. Of course condos knock this idea out. Q - "What size is your lookup - you may well be hitting the default limits defined (25MB)" A - csv: 1 million records - 448,500 bytes // kvstore: 3 million records - 2,743.66 MB Q - "What are you currently doing to be 'fuzzy' so your matches currently work or are you really looking for exact matches somewhere in your data?" A - I stripped off any non-numeric characters at the beginning of the address on the lookup and use that field for the as in my lookup command with my kvstore | lookup my_kvstore addr as mod_addr output owner Q - Is your KV store currently being updated - and is it replicated? A - No replication. The data would be refreshed yearly, or possibly every quarter. Q - Also, if you are just looking at some exact match somewhere, then the KV store may benefit from using accelerated fields - that can speed up lookups against the KV store (if that's the way you're doing it) significantly. A - Using the above code, the addr would be the accelerated field, correct? Thanks again for your help and God bless. Genesius
I recommend using a HF only if necessary. In addition to the factors listed previously, HFs add a layer of complexity, are something else to manage, and introduce another point of failure. I distin...
See more...
I recommend using a HF only if necessary. In addition to the factors listed previously, HFs add a layer of complexity, are something else to manage, and introduce another point of failure. I distinct advantage of HFs in a Splunk Cloud environment is better control over how your data is parsed. It's much easier to manage the apps in a HF than it is to do so in Splunk Cloud - even with the Victoria experience. Of course, you should have at least 2 HFs for redundancy.
Hi @tmcbride17 , the correct question is what's the protocol that uses Splunk Universal Forwarder to forward logs to the Indexers? An add-on is a configuration on the UF. To send logs, the UF usua...
See more...
Hi @tmcbride17 , the correct question is what's the protocol that uses Splunk Universal Forwarder to forward logs to the Indexers? An add-on is a configuration on the UF. To send logs, the UF usually uses TCP http or https, it depends if TLS is enabled or not and by default it uses the 9997 port but it can also use HEC, that's less efficient than the other. Forwarders are managed by the Deployment Server using TCP https on port 8089. Ciao. Giuseppe
Hi @azer271 , if you're speking of an Indexer Cluster, best practices hint to deisable web interface on Search Peers (Indexers) and maintain it on the Cluster Manager. If you are speaking of a Sear...
See more...
Hi @azer271 , if you're speking of an Indexer Cluster, best practices hint to deisable web interface on Search Peers (Indexers) and maintain it on the Cluster Manager. If you are speaking of a Search Head Cluster, you have to use the Deployer to deploy Apps to the SHs, and the common configurations like the connection with the Indexer Cluster. If you don't have a Search Head Cluster but a stand-alone Search Head, you can run the command on the Search Head to connect it to the Cluster Manager and the Search Peers. The Deployer isn't a Search Head and cannot be configured as a SH. In conclusion, what's your requirement: you need an Indexer Cluster? if yes, mono site or multi site? you need a Search Head Cluster or a stand-alone Search Head? Remember that you cannot use the Deployment Server to deploy apps to the Indexer Cluster and to the Search Head Cluster. for more information, see at https://docs.splunk.com/Documentation/Splunk/9.3.2/Indexer/Aboutclusters and https://docs.splunk.com/Documentation/Splunk/9.3.2/DistSearch/AboutSHC Ciao. Giuseppe
Hello there. I would like to ask about Splunk best practices, specifically regarding cluster architecture. One suggested practice is to configure all Splunk servers running Splunk Web (aka: a search ...
See more...
Hello there. I would like to ask about Splunk best practices, specifically regarding cluster architecture. One suggested practice is to configure all Splunk servers running Splunk Web (aka: a search head) as members of the indexer cluster, (at least that is what I hear from the architecture lesson). For example, there is a Splunk deployer. I need to use this command or achieved through web: splunk edit cluster-config -mode searchhead -manager_uri https://x.x.x.x:8089 (indexer cluster manager IP) -secret idxcluster Another one suggested practice is adding the Splunk servers (mention above such as deployers) to distributed search > search peers as well in manager. I would like to know why these are good practice and what are the benefits of doing these. (The deployer is not really a search head?) Thank you.
Thanks, @PickleRick I understand about the linear search nature of lookups. I was hoping there were perhaps some new commands on the horizon with the most recent (or future) versions of Splunk Ente...
See more...
Thanks, @PickleRick I understand about the linear search nature of lookups. I was hoping there were perhaps some new commands on the horizon with the most recent (or future) versions of Splunk Enterprise. Or someone might have experience with MLTK, or another Splunk product, to handle this use case. Thanks and God bless, Genesius
Hi @CHAUHAN812,
In that case, in the indexes.conf file, you just need to adjust the frozenTimePeriodInSecs parameter in the 2 index stanzas.
[index01]
frozenTimePeriodInSecs = 34187400
[index02]...
See more...
Hi @CHAUHAN812,
In that case, in the indexes.conf file, you just need to adjust the frozenTimePeriodInSecs parameter in the 2 index stanzas.
[index01]
frozenTimePeriodInSecs = 34187400
[index02]
frozenTimePeriodInSecs = 34187400
Restart Splunk after that
Yes , I have an individual indexer which is installed on Linux machine. And I need to increase the frozenTimePeriodInSecs only for 2 of the indexes. So to increase the Frozen Time Period from 12 mon...
See more...
Yes , I have an individual indexer which is installed on Linux machine. And I need to increase the frozenTimePeriodInSecs only for 2 of the indexes. So to increase the Frozen Time Period from 12 months to 13 months then I just need to update the frozenTimePeriodInSecs values to the indexes.conf file from the indexer server right ?
If you have individual indexers then it is correct place. After change do reload for it. If you have indexer cluster then you must do this change on CM. Edit correct indexes.conf file somewhere under...
See more...
If you have individual indexers then it is correct place. After change do reload for it. If you have indexer cluster then you must do this change on CM. Edit correct indexes.conf file somewhere under master-apps or manager-apps. After that apply cluster-bundle, when it has distributed into search peers.
I want to increase one of my index frozen Time Period from 12 months to 13 months. I have increased the Max Size of Entire Index from the Splunk indexer > Settings. But I know this is not enough as m...
See more...
I want to increase one of my index frozen Time Period from 12 months to 13 months. I have increased the Max Size of Entire Index from the Splunk indexer > Settings. But I know this is not enough as my index frozen Time Period is set on 12 months period. So where should I update this value ? Should I need to update 'Indexes.conf' file for required indexes to the indexer server itself which is installed on Linux machine. What things I need to take care while updating this frozen Time Period.
@samy335 How did you register for the Splunk Cloud free trial account? Did you use your business email or personal email? If you used a business email, check with your IT team to see if they are...
See more...
@samy335 How did you register for the Splunk Cloud free trial account? Did you use your business email or personal email? If you used a business email, check with your IT team to see if they are blocking any external emails.
I am assuming due to the way the query is being evaluated, it doesn't just take latest value, and hence due to using max, it gets the largest value stored of that field for however long you store dat...
See more...
I am assuming due to the way the query is being evaluated, it doesn't just take latest value, and hence due to using max, it gets the largest value stored of that field for however long you store data in analytics.
You can either try and change the max to min which should get the lowest value always but better would be to append the following clause to the query to ensure that only the last 5 minutes of data gets used to get the value - SINCE 5 minutes
@refahiati Are you experiencing high resource usage on the Splunk Heavy Forwarder? If so, I suggest configuring syslog-ng or rsyslog on the Heavy Forwarder to collect logs and store them in a separa...
See more...
@refahiati Are you experiencing high resource usage on the Splunk Heavy Forwarder? If so, I suggest configuring syslog-ng or rsyslog on the Heavy Forwarder to collect logs and store them in a separate directory. You can then monitor that directory to forward the events to Splunk indexers. Additionally, review the queues in the metrics.log file for any potential issues.