topic Re: Help creating a sourcetype for this data in Getting Data In

Help creating a sourcetype for this data

roayers — Tue, 29 Sep 2020 17:26:45 GMT

I've been trying to figure out a way to create a sourcetype and extract data like this.
Can someone help? It appears to be 3 goups.

Here is the ideal break out of the fields required

The first 2 lines can be ignored

Group 1

Conventional - system type the fields for this group are as follows The first word conventional is not required, a space then there are 2 empty fields then these fields,
name,avoid,system_type,fl_qk,tag_number,hold,time,ana_dgc,dig_agc,dig_wait_time,dig_thr_mode,dig_thr_lvl

I don't need the contents of the DQKS_Status field or its values

Group 2

C-Group
The first word conventional is not required, there are 2 empty fields then these fields,
name,avoid,latitude,longitude, range,location_type,sys_qk

Group 3

C-Freq and its values are a subset of the C-Group with these fields,
name, avoid,frequency,modulation,audio,dept,service_type,attenuator,delay,alert_tone,alert_light,vol_offset,num_tag, priority

Sample data

TargetModel BCDx36HP

FormatVersion 1.00
Conventional Ft IndianTown Gap Off Conventional 1 Off 0 Off Off 400 Auto 8
DQKs_Status Off On Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off
C-Group Helo's Comm Card Off 0.000000 0.000000 0.0 Circle 1
C-Freq Harrisburg N/E Off 118250000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq CXY Tower Off 119500000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq ABE Approach Off 119650000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Reading Tower Off 119900000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq LNS Tower Off 120900000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq MUIR Ground Off 121625000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq ZER CTAF Off 123075000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Harrisburg S/W Off 124100000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq MUIR ASOS Off 124175000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq MDT Tower Off 124800000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq MUIR Tower Off 126200000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Harrisburg S/E Off 126450000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Reading Approach Off 127100000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq NTA Off 141500000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Balky Off 142450000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq NTA Off 227300000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Balky Off 239150000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq LNS Tower Off 251100000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq CXY Tower Off 257800000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq MDT Tower Off 269350000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Harrisburg N/E Off 269450000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq MUIR Ground Off 269525000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Harrisburg S/W Off 273525000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Harrisburg S/E Off 281525000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq MUIR Tower Off 290500000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq HQ ZER CTAF Off 300050000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq HQ Reading App Off 375825000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq Reading Tower Off 375925000 AUTO 208 Off 2 0 Off Auto Off On Off Off
C-Freq HQ ABE App Off 376125000 AUTO 208 Off 2 0 Off Auto Off On Off Off
Conventional Gap Range Active Off Conventional 2 Off 0 Off Off 400 Auto 8
DQKs_Status Off On On Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off
C-Group Air to Air Comms Off 0.000000 0.000000 0.0 Circle 1
C-Freq Angry Off 139150000 AM TONE=Srch 208 Off 2 0 Off Auto Off On Off Off
C-Freq 233.4500Mhz Off 233450000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq MD ANG A10 A/A Off 271400000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq 290.5000Mhz Off 290500000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq MD ANG A10 A/A Off 293200000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq 104Th TFS Off 354800000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Group Miscellaneous Off 0.000000 0.000000 0.0 Circle 2
C-Freq ANG Flight Following On 40900000 FM 208 Off 2 0 Off Auto Off On Off Off
C-Freq ANG Op's On 41500000 FM 208 Off 2 0 Off Auto Off On Off Off
C-Freq ANG Op's On 49950000 FM 208 Off 2 0 Off Auto Off On Off Off
C-Freq KMUI Ground Off 121625000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq HBG Approach SW Off 124100000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq Cleveland VHF Off 124325000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq KMDT Tower On 124800000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq KMUI Tower Off 126200000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq Muir Departure Off 126450000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq 133.9700Mhz Off 133970000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq NJ ANG Air to Air Off 139625000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq 139.7000Mhz On 139700000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq Air to Air Off 142300000 FM 208 Off 2 0 Off Auto Off On Off On
C-Freq Bollen Alternate Off 232700000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq 233.4500Mhz Off 233450000 AM 208 Off 2 0 Off Auto Off On Off On
C-Freq 238.4000Mhz Off 238400000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq 239.1500Mhz Off 239150000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq Steel A/R Off 259400000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq MD ANG A10 A/A Off 266600000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq NY UHF Fighters Off 269100000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq MD ANG A10 A/A Off 271400000 AM 208 Off 2 2 Off Auto Off On Off On
C-Freq 290.5000Mhz Off 290500000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq MD ANG A10 A/A Off 293200000 AM 208 Off 2 0 Off Auto Off On Off On
C-Freq Steel Tankers Off 293700000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq Steel A/R Off 301600000 AM 208 Off 2 0 Off Auto Off On Off On
C-Freq Huntress Off 338300000 AM 208 Off 2 0 Off Auto Off On Off Off
C-Freq Cleveland UHF Off 353850000 AM 208 Off 2 0 Off Auto Off On Off On
C-Freq 362.9500Mhz Off 362950000 AM 208 Off 2 0 Off Auto Off On Off Off

Re: Help creating a sourcetype for this data

micahkemp — Wed, 27 Dec 2017 22:17:51 GMT

This seems to extract the fields you want, though I don't know if you need multiple lines per event:

[<sourcetype name>]
SHOULD_LINEMERGE = false
EXTRACT-conventional = ^Conventional (?<name>.*) (?<avoid>[^ ]+) (?<system_type>[^ ]+) (?<fl_quick_key>[^ ]+) (?<tag_number>[^ ]+) (?<hold>[^ ]+) (?<time>[^ ]+) (?<ana_dgc>[^ ]+) (?<dig_agc>[^ ]+) (?<dig_wait_time>[^ ]+) (?<dig_thr_mode>[^ ]+) (?<dig_thr_lvl>[^ ]+)$  
EXTRACT-cgroup = ^C-GroupHelo's (?<name>.*) (?<avoid>[^ ]+) (?<latitude>[^ ]+) (?<longitude>[^ ]+) (?<range>[^ ]+) (?<location_type>[^ ]+) (?<sys_qk>[^ ]+)$
EXTRACT-cfreq = ^C-Freq (?<name>.*) (?<avoid>[^ ]+) (?<frequency>[^ ]+) (?<modulation>[^ ]+) (?<audio>[^ ]+) (?<dept>[^ ]+) (?<service_type>[^ ]+) (?<attenuator>[^ ]+) (?<delay>[^ ]+) (?<alert_tone>[^ ]+) (?<alert_light>[^ ]+) (?<vol_offset>[^ ]+) (?<num_tag>[^ ]+) (?<priority>[^ ]+)$

Re: Help creating a sourcetype for this data

roayers — Wed, 27 Dec 2017 22:25:49 GMT

Micahkemp,

Yes

Group 1 - conventional would only have 1 event
Group 2 - cgroup would only have 1 event
Group 3 - cfreq would have multiple events

Re: Help creating a sourcetype for this data

micahkemp — Wed, 27 Dec 2017 22:58:20 GMT

What I'm asking is if they are multiline events, and if so, could you group the lines into how you would like them to exist within each event.

I'm not sure your specific grouping needs will be easily accomplished in Splunk. The ability to group events relies on a single definition of when to break into a new event. It may be possible with some regex, but it isn't the typical use case.

Re: Help creating a sourcetype for this data

roayers — Wed, 27 Dec 2017 23:16:35 GMT

Micahkemp,

I created a props.conf file in splunk\etc\apps\search\local and pasted the code into it. I added 1 line, pulldown_type = true, i then restarted splunk. I then used the web interface to import the file into a new test index but it only parsed out 1 field, tone. It did not parse out any of the other fields.

Re: Help creating a sourcetype for this data

micahkemp — Wed, 27 Dec 2017 23:25:18 GMT

Did you run the search in verbose mode? tone isn't one of the fields my regexes made available, so I wonder if your sourcetype is making use of KV_MODE=auto to get that one.

Re: Help creating a sourcetype for this data

roayers — Wed, 27 Dec 2017 23:58:37 GMT

micahkemp

Yes, the search was run in verbose mode. I just noticed that tone was not in the regex so you're right KV_MODE=auto must have got that field

Re: Help creating a sourcetype for this data

roayers — Thu, 28 Dec 2017 16:06:23 GMT

sure

ideally

Group1 - Convention would contain all of those fields in 1 event
|_________
Group2 - C-Group would contain all of those fields in 1 event
|__________
Group3 - C-Freg would contain all of those fields as individual events
|______________________
|______________________
|______________________
|______________________

Re: Help creating a sourcetype for this data

roayers — Thu, 28 Dec 2017 16:19:07 GMT

micahkemp,

Let me rethink this, if you had to ingest that data how would you do it? I'm not sure of all of the best practices regarding ingesting data. I also thought a bout a global replace in the original source file to break up the groups.

Re: Help creating a sourcetype for this data

micahkemp — Thu, 28 Dec 2017 18:16:09 GMT

I think you want to keep each line as a separate event. At reporting time you may want to group them logically (via stats or other search commands). Have you tried implementing the above configurations to see what they get you?

Re: Help creating a sourcetype for this data

roayers — Thu, 28 Dec 2017 19:28:21 GMT

Yes, I agree each line should be a separate event. My data file appears to have issues, there are other fields that are not displayed unless options are selected in the program that generated the files. Now that I believe have all of the fields, I want to focus on the field extractions then I can use the the search commands to get what I'm looking for.