hello. I use splunk db connect
3.1.3 connect mongodb database. it is working now. and I can use SQL statement query data from mongodb
|dxquery connection="testmongodb" query="select * from result"
.
We all know that mongodb does not contain a self-incrementing column. So mongodb's data is similar to the following format:
info ip port task_date task_id time vul_info
Unauthorized Access 172.16.10.9 6379 2019-6-6 d40617172258939a57fdb5617724fc55 2019-6-6 {"vul_type":"Weak password",vul_name:"Redis Weak password",vul_level:"High"}
SMB Remote Overflow 10.10.2.8 445 2019-6-6 cfab842aa0e8166cabb2f4548477756b 2019-6-6 {"vul_type":"Remote Overflow",vul_name:"SMB Remote Overflow",vul_level:"High"}
MySQL Weak password 10.10.2.7 3306 2019-6-13 2389ccda6788fc124d1cec7a951f7089 2019-6-13 {"vul_type":"Weak password",vul_name:"MySQL Weak password",vul_level:"High"}
Firstly, it does not have an rising column, for example id
, secondly, it does not have a timestamp.
If I use input with batch to index these data into splunk, there will be a lot of duplicate data. So I hope. Every time there is new data in the result collections, it can automatically index to Splunk .
If this mongo collection have a rising column, it will be easy to implement this requirement, unfortunately not.
So is there a clever way to index new data from mongodb to splunk?
Hi @bestSplunker,
Have you considered using the kv-store...it's also a mongodb data base so you can just dbquery your data from your DB into the kv-store and then use the results from there.
Cheers,
David
Hi @bestSplunker,
Have you considered using the kv-store...it's also a mongodb data base so you can just dbquery your data from your DB into the kv-store and then use the results from there.
Cheers,
David
@DavidHourani
what should I do? Is there a link to the documentation? These data are vulnerability information, meaning that a host has discovered a new vulnerability, so I want alert it if result
collection adding a new recorder
sure, have a look here for how to set up the kv-store : https://docs.splunk.com/Documentation/Splunk/7.3.0/Knowledge/ConfigureKVstorelookups
It's very easy to use, and once the data is there you can use it as a lookup, which is perfect for vulnerability data. You'll be able to use that data to enrich your logs using the lookup
command.
Let me know when you try setting it up if you find any difficulty, i'd be happy to help.
Thank you, I solved this problem through kvstore. firstlly, I query the result from mongodb, and then search for it in Kvstore. If the result does not exist in kvstore, an alert is triggered, finally fill all the results into the kvstore..
|dbxquery connection="testmongodb" query="select * from Result"
|search NOT [|inputlookup resultcollections]
|outputlookup resultcollections