Thanks @PickleRick I have very less visibility and access issues on the source side, while I understand it is the easiest way to do this on the client side, I am trying to understand what are the possibilities at I have or can do on my HF on which I have full control. ... View more

Hi @isoutamo    I am currently using Splunk ingest actions feature to route the logs to S3 bucket and it doesn't have the capability to include <host>:<original sourcetype> for the events.  Thank you for taking time to reply to my query. ... View more

Hi @VatsalJagani    Yes I raised a case with Splunk support and they confirm they do not have such capability in place and I advised the to add it to their future enhancements list.  I hope this will be considered.  Appreciate your response. ... View more

I completely agree with what you've stated below @gcusello @isoutamo @ITWhisperer @PickleRick , and I'm on the same page. However, as you know, compliance principles operate on the premise that whether an issue is present or not, it's best to assume it is and address it accordingly. In my situation, we mainly deal with network-related data where the likelihood of finding PII is very low. Nonetheless, as a security requirement, we want to establish controls that ensure any sensitive information, if present, is masked. ... View more

Exactly what I was looking for, thank you so much !! @ITWhisperer  ... View more

Appreciate your response @marnall .  My questions comes from our recent scenario, where we did a splunk upgrade using Infrastructure as code and we are using smartstore for indexing.  We were in the opinion that the data get moved to external storage once it hits the warm bucket but unfortunately, we lost some of the data during the migration.  The only reason we could think of is the hot buckets which are stored locally did not get rolled over to warm bucket which could have been available in the external storage and are available for later searches.  We have another migration scheduled for this weekend, so I want to be cent percent sure we don't have any data loss. ... View more

Hi @yuanliu  Thank you for your feedback! I will definitely look into the performance issue and plan for further improvements. Regarding the query, I tried it out, and here’s how it’s working: Index A contains around 70k assets, which serves as our asset inventory. Some hosts in this index have multiple IP addresses assigned to them. Index B has just the hostname, but this can include a mix of IP addresses, FQDNs, and hostnames. When I ran the query, the first lookup compared the Reporting_Host with the hostnames in Index A and determined whether there was a match. The second lookup compared the Reporting_Host against the IP addresses in Index A to check for matches. However, when we combined these lookups as shown in the query you shared, the results only reflected matches from the second lookup, meaning only the IP addresses were being compared. Additionally, since a host in Index A has multiple IP addresses, the query gives a match for the IP address that corresponds, but for the remaining IP addresses associated with that host, it shows them as missing. ... View more

Something like this  index=B | stats count by Reporting_Host | search NOT [| inputlookup inventory.csv ] | table Hostname ] inventory.csv has the table pickup from index A.  Lookup query is - index=B | stats values(IP address) by Hostname Operating_system PS This query is not working, just a thought. ... View more

Hi @yuanliu  Unfortunately none of the below queries are working for me.  First one is crashing splunk so unable to test it.  Second one, I don't get any results.  Could be because the field "Reporting_Host" is present only in index B and since we are excluding index B in the next step, the results are 0.  However I tried renaming the Hostname field in index A and try running the query but no results.   Can we test this scenario using a look up table, that might improve the search performance.  Can you give me something in this regards? ... View more

@yuanliu Thank you for your response.  I tried below query but it doesn't seem to be working.  When  I further cut down the query for testing, looks like "|where index!=B" is not working.  Everything before this query is working but when I add this condition, I get 0 results. also, the query seems to be very aggressive.  My index A has almost close to 70k events and index B has around 10k events.  Splunk was crashing few times when I try to run the query.   Any suggestions, how to address this ? ... View more

Thank you @yuanliu  @jawahir007  Both of your solutions are working absolutely fine. @yuanliu  yes, index A always has larger number of hosts compared to index B. I would like to further expand this query to match the IP address aswell.  Can you provide some guidance around that. index A data  Hostname IP address OS xyz 190.1.1.1,  101.2.2.2, 102.3.3.3, 4.3.2.1 Windows zbc 100.0.1.0 Linux alb 190.1.0.2 Windows cgf 20.4.2.1 Windows bcn 20.5.3.4, 30.4.6.1 Solaris   Index B Hostname zbc 30.4.6.1 alb 101.2.2.2   Results Hostname IP address OS match xyz 190.1.1.1,  101.2.2.2, 102.3.3.3, 4.3.2.1 Windows ok(because IP address 101.2.2.2 is matching) zbc 100.0.1.0 Linux ok alb 190.1.0.2 Windows ok cgf 20.4.2.1 Windows missing(neither hostname is present nor the IP is matching) bcn 20.5.3.4, 30.4.6.1 Solaris yes(IP is matching) In my initial use case, I compared the hostnames in index A with those in index B. Now, I want to check if the hosts in index A are reporting their IP addresses in index B. If there’s a match, I will mark the corresponding hostname in index A as "ok." ... View more

@ITWhisperer Appreciate your help on this but the query doesn't seem to be working.  For count=1 / count =2 I see the events appear in both lookup and indexed events.   ... View more

@ITWhisperer Could you explain how this works ? do I need to append this to my original query ? I don't see the syslog_farm index used anywhere in your search query. ... View more

| inputlookup cmdb_asset_inventory.csv | rename Generic_Hostname as Reporting_Host | eval Reporting_Host=lower(Reporting_Host) | stats count by Reporting_Host Hostname Environment Tier3 Operating_System | join type=left [ search index=prod_syslogfarm | eval Reporting_Host=lower(Reporting_Host) | stats values(Reporting_Host) as Exists by Reporting_Host ] | fillnull Exists value=0 | search Exists = 0   FYI, query to obtain the cmdb_asset_inventory index=cmdb | eval Generic_Hostname=mvappend(Hostname, IP_Address) .   In the cmdb_asset_inventory, a hostname may contain multiple IP addresses(as you see below). Output -  Reporting_Host Hostname Environment Tier3 Operating_System count Exists 1.11.12.13 xyz Production Server Windows Server 2022 1 0 1.0.1.1 xyz Production Server Windows Server 2022 1 0 xyz xyz Production Server Windows Server 2022 1 xyz xyz.abc.com xyz Production Server Windows Server 2022 1 0   I've been able to achieve partial success with the query where Exists=xyz. The challenge I'm facing is that the host "xyz" is reporting with the hostname "xyz", and I'm able to look up this hostname in the inventory. Once a match is found, it should ignore all other combinations since "xyz" in the syslog host is present in the inventory lookup.   I tried my best to explain my requirement, apologies if something above doesn't make sense.  I will try to be more clear. ... View more

@ITWhispererI attempted to execute your search by my goal is to identify and output the assets that are present in `myinventory` lookup but absent from the `syslog_farm` index. ... View more

Hi @yuanliu  It seems that the current query is only retrieving results from the 'myinventory' lookup without performing the intended comparison with the 'asset_inventory' data. It appears that you need to modify the query such that it compares both datasets ('myinventory' and 'asset_inventory') and returns only the discrepancies between the two. ... View more

Thank you @richgalloway  But this does not meet my requirement as my syslog data contains combination of hostname, fqdn and IP address and I have to match all these three fields with the respective asset inventory data which has these fields (Hostname, IP address, FQDN).  So I have to check if the syslog hostname or IP or fqdn is present in the aset inventory data and output only if the syslog data doesn't match with any of these three fields in the asset inventory data.  I have posted this question with examples in this link.  Although I have accepted the answer here https://community.splunk.com/t5/Splunk-Search/How-to-compare-a-look-up-field-with-multivalued-indexed-data-in/m-p/691717#M235509 after further testing, it doesn't seem to be working as expected. ... View more

That worked!! One last thing, how do I display only specific sourcetype out of (A B C D E) for where the events for all the days=0. reword this statement ... View more