I've got data coming in from Meraki APs. It's mostly good, but I can't get rid of the single quotes at index time.
Data:
Nov 15 17:08:27 192.168.255.1 1 1542301707.870571288 main_office events content_filtering_block url='https://baddymcbadface.com/...' category0='Spyware and Adware' server='123.123.123.123:443' client_ip='192.168.204.80'
Here's what I have set up in my props and transforms.conf
Transforms:
[meraki-singlequotes]
REGEX = \s*([^=]+)='([^']*)'
FORMAT = $1::$
Props:
[source:meraki]
REPORT-singlequotes = meraki-singlequotes
I've also tried changing this with [sourcetype:meraki-hq], but that doesn't work as well.
I'm bringing this in via syslog. inputs.conf for this is in the SPLUNK_HOME/etc/apps/launcher/local/ I've added them there, but no dice.
I've also put them in the TA-meraki as well, but it's not taking. Does anyone have any suggestions?
Here are my updated conf files. Looks like it's still not taking. Thanks @laurie_gellatly and @FrankVl
Props.conf
[source::meraki]
SEDCMD-singlequotes=s/\s*([^=]+)='([^']*)'/g
Transforms.conf (though, I think if I'm doing SEDCMD command, I don't need transforms anymore)
[meraki-singlequotes]
REGEX = \s*([^=]+)='([^']*)'
FORMAT = $1::$2
Anonymized output:
Nov 19 18:42:58 192.168.200.15 1 1542652978.961504488 Board_Room events type=disassociation radio='1' vap='0' client_mac='E4:2B:DD:DD:DD:DD' channel='36' duration='141.595751334' auth_neg_dur='0.839737584' last_auth_ago='140.746016874' is_8021x='1' full_conn='1.309590753' ip_resp='1.309590753' ip_src='192.168.207.2' arp_resp='0.839737584' arp_src='192.168.207.2' dns_server='222.222.222.222'
Here are my updated conf files. Looks like it's still not taking. Thanks @laurie_gellatly and @FrankVl
Props.conf
[source::meraki]
SEDCMD-singlequotes=s/\s*([^=]+)='([^']*)'/g
Transforms.conf (though, I think if I'm doing SEDCMD command, I don't need transforms anymore)
[meraki-singlequotes]
REGEX = \s*([^=]+)='([^']*)'
FORMAT = $1::$2
Anonymized output:
Nov 19 18:42:58 192.168.200.15 1 1542652978.961504488 Board_Room events type=disassociation radio='1' vap='0' client_mac='E4:2B:DD:DD:DD:DD' channel='36' duration='141.595751334' auth_neg_dur='0.839737584' last_auth_ago='140.746016874' is_8021x='1' full_conn='1.309590753' ip_resp='1.309590753' ip_src='192.168.207.2' arp_resp='0.839737584' arp_src='192.168.207.2' dns_server='222.222.222.222'
That’s not how SEDCMD works. It should be s/<regex matching what needs to be replaced>/<what it needs to be replaced with>/g
(the g is to keep applying it, rather than replacing just the first match).
In your case, that could be as simple as s/'//g
.
That did it. Thanks.
And remember it won't fix what you've already ingested into the index. Only new ingestions will be altered!
I now have a moment to take a closer look and provide a more extensive answer (apart from my earlier comment about REPORT not being performed at indextime), there are several problems with your attempts:
[source::<source>]
, note the double :
.[<sourcetype>]
, no sourcetype:
in front of the actual sourcetype.=
; see also: https://regex101.com/r/1LJ1Wl/12
at the end of your FORMAT string.Also your transform needs to $1::$2
And as already stated it happens at search not index time.
If it MUST happen at index then you could look at using sedcmd.
...Laurie:{)
A REPORT transform works at searchtime, not at indextime.
Can you perhaps explain a bit further what is and what isn't working, what output do you get now and what is still wrong with that which your are trying to fix?