I have a folder which stores .txt files. I need to compare the data that is inside the files. Not only accumulate the data for each search
1234.txt
4563.txt
8523.txt
The 3 files have similar data but at the same time new data.
Example
Three different .txt files are generated during the month, all month.
What we want is to collect data compare equal as a single vulnerability if they have the same IP and same vulnerability.
Do not take that as a cumulative total would be 6, but 3 are repeated with the same data only date difference.
The process would stay on 3 vulnerabilities and compared in total
All files is in the same folder in splunk.
2014-01-04.txt
response_datetime="2014-01-04 20:46:15" scan_date="2014-01-04 20:46:15" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2
2014-01-25.txt
response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2
response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=45863214 ip=65.196.59.130 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.3
Target Hostname: 192.168.1.3
response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=4D2F5C05 ip=192.168.1.4 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.4
2014-01-18.txt
response_datetime="2014-01-18 20:45:14" scan_date="2014-01-18 20:45:14" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2
response_datetime="2014-01-18 20:45:14" scan_date="2014-01-18 20:45:14" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.3 testid=772 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name=Services port=23 protocol=tcp results="A telnet server seems to be running on this port
Target IP: 192.168.1.3
The simplest way would be to just do a dedup on the keys/fields which uniquely identify a vulnerability - host/ip combination. An example search would be like this
your base search from all there files | dedup host ip vuln_type vuln_name results
This will remove all the duplicates/repeating vulnerability from all there files and give only the unique ones. After that you can perform your post process.
The simplest way would be to just do a dedup on the keys/fields which uniquely identify a vulnerability - host/ip combination. An example search would be like this
your base search from all there files | dedup host ip vuln_type vuln_name results
This will remove all the duplicates/repeating vulnerability from all there files and give only the unique ones. After that you can perform your post process.
This is a type of code that we use to analyze the .txt files
As the commentary would add this code to compare files that are inside the month.
sourcetype="---" org_id="25336984" NOT vuln_risk=0 | chart count over scan_id by vuln_risk | rename 8 as High, 4 as Medium, 1 as Low | replace 78963512 with "Prueba" in scan_id | rename scan_id as "Scan Name" | fields "Scan Name", High, Medium, Low | addtotals
Try this,
index=<your index> sourcetype=<sourcetype> | stats count as vulnerabilities by ip,vuln_type,source | sort -count | table vuln_type,vulnerabilities
Hi vasanthmss
Do you know?
As I can use the code that you registered with this code is the use for this part specifically vulnerabilities
sourcetype="---" org_id="25336984" NOT vuln_risk=0 | chart count over scan_id by vuln_risk | rename 8 as High, 4 as Medium, 1 as Low
i need any suggestion.
Hi Nidet,
Share some some sample logs / details which are available in the files, and kindly share what you want to compare exactly.
I shared information, about the question. Thanks for your support
Have you gone through the Search Tutorial? I think you'll find your answer by the end. Its kind of hard to try an explain so much information on the answers site.