Splunk Search

Compare field in search

nidet
Explorer

I have a folder which stores .txt files. I need to compare the data that is inside the files. Not only accumulate the data for each search

1234.txt
4563.txt
8523.txt

The 3 files have similar data but at the same time new data.

What code can be used to compare 3 files?

Example

Three different .txt files are generated during the month, all month.
What we want is to collect data compare equal as a single vulnerability if they have the same IP and same vulnerability.

Do not take that as a cumulative total would be 6, but 3 are repeated with the same data only date difference.
The process would stay on 3 vulnerabilities and compared in total

All files is in the same folder in splunk.

2014-01-04.txt

response_datetime="2014-01-04 20:46:15" scan_date="2014-01-04 20:46:15" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2

2014-01-25.txt

response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2

response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=45863214 ip=65.196.59.130 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.3
Target Hostname: 192.168.1.3

response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=4D2F5C05 ip=192.168.1.4 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.4

2014-01-18.txt

response_datetime="2014-01-18 20:45:14" scan_date="2014-01-18 20:45:14" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2

response_datetime="2014-01-18 20:45:14" scan_date="2014-01-18 20:45:14" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.3 testid=772 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name=Services port=23 protocol=tcp results="A telnet server seems to be running on this port
Target IP: 192.168.1.3

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

The simplest way would be to just do a dedup on the keys/fields which uniquely identify a vulnerability - host/ip combination. An example search would be like this

your base search from all there files | dedup host ip vuln_type vuln_name results 

This will remove all the duplicates/repeating vulnerability from all there files and give only the unique ones. After that you can perform your post process.

View solution in original post

somesoni2
Revered Legend

The simplest way would be to just do a dedup on the keys/fields which uniquely identify a vulnerability - host/ip combination. An example search would be like this

your base search from all there files | dedup host ip vuln_type vuln_name results 

This will remove all the duplicates/repeating vulnerability from all there files and give only the unique ones. After that you can perform your post process.

View solution in original post

nidet
Explorer

This is a type of code that we use to analyze the .txt files
As the commentary would add this code to compare files that are inside the month.

sourcetype="---" org_id="25336984" NOT vuln_risk=0 | chart count over scan_id by vuln_risk | rename 8 as High, 4 as Medium, 1 as Low | replace 78963512 with "Prueba" in scan_id | rename scan_id as "Scan Name" | fields "Scan Name", High, Medium, Low | addtotals

0 Karma

vasanthmss
Motivator

Try this,

 index=<your index> sourcetype=<sourcetype> | stats count as vulnerabilities by ip,vuln_type,source | sort -count | table vuln_type,vulnerabilities

nidet
Explorer

Hi vasanthmss

Do you know?
As I can use the code that you registered with this code is the use for this part specifically vulnerabilities

sourcetype="---" org_id="25336984" NOT vuln_risk=0 | chart count over scan_id by vuln_risk | rename 8 as High, 4 as Medium, 1 as Low

i need any suggestion.

0 Karma

vasanthmss
Motivator

Hi Nidet,

Share some some sample logs / details which are available in the files, and kindly share what you want to compare exactly.

0 Karma

nidet
Explorer

I shared information, about the question. Thanks for your support

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

Have you gone through the Search Tutorial? I think you'll find your answer by the end. Its kind of hard to try an explain so much information on the answers site.

0 Karma