Hi all,
rex "WifiCountryDetails\W+(?<WifiCountryDetails>[\w*\s*]+)"
We r using the above Rex for getting the Wi-Fi country details... But the problem is while fetching the data, if the Wi-Fi country name is empty it automatically gathers the next field value in it..
But if the wificountrydetails are empty it has to show empty in the data,
please let me know how to achieve it.
Hi @sateesh250795 ..
1)Can you provide some sample logs..
2) After the "WifiCountryDetails" field, do you have a "," or ":", etc ?!?!?
let me assume it is a comma (",")...then, lets try this rex:
rex "WifiCountryDetails\W+(?<WifiCountryDetails>[\w*\s*]+)\,"
178.197.202.134 SECURITY_MONITORING_AUTOMATION CAMERA_SYSTEMS VML4 VML4030 1697109816450 INFO 51D29C77A022A KU5TFBKH-1700-316-114351851 - statusInfo {"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable","ChargerTech":"QuickCharger","ChargingState":"Off","Bat1Volt":7.931,"Temperature":24,"Battery1CaliVoltage":7.931,"Streamed":53395,"UserStreamed":1650,"MotionStreamed":7561,"IRLEDsOn":21054,"PoweredOn":11029385,"CameraOnline":11029337,"CameraOffline":48,"WifiConnectionCount":2,"WifiConnectionAttempts":2,"PIREvents":66,"FailedStreams":1,"FailedUpgrades":0,"SnapshotCount":0,"LogFrequency":2,"CriticalBatStatus":0,"ISPOn":57564,"TimeAtPlug":11025490,"TimeAtUnPlug":10979418,"PercentAtPlug":95,"PercentAtUnPlug":100,"ISPWatchdogCount":0,"ISPWatchdogCount2":0,"SecsPerPercentCurr":0,"SecsPerPercentAvg":0,"PirOorEvents":0,"DdrFailCnt":0,"carrierFw":"Generic","SignalStrengthIndicator":2,"SystemVersionString":"1.096.3.2_1.8.65.0_22","ID":534,"SystemFirmwareVersion":"1.096.3.2_30_0bad0df","HardwareVersion":"H6","SystemSerialNumber":"51D29C77A022A","TimeStamp":"2023-10-12T11:23:36+0000"}
Hi @sateesh250795 .... may we know if you got answer for your question please, thanks.
As I always warn people, do not treat structured data as text. Using Splunk's built-in function with JSON, XML, and so on is much more robust and saves you tons of headaches in future maintenance.
All you need to do is to extract the conformant JSON into its own field. Then apply spath. (An alternative new syntax is fromjson.)
| rex "^[^{]+\s*(?<json>{.+})"
| spath input=json
It's that simple. Your sample log will give you everything in that JSON.
fieldname | fieldvalue |
Bat1Volt | 7.931 |
BatPercent | 100 |
BatTech | Rechargeable |
Battery1CaliVoltage | 7.931 |
CameraOffline | 48 |
CameraOnline | 11029337 |
ChargerTech | QuickCharger |
ChargingState | Off |
CriticalBatStatus | 0 |
DdrFailCnt | 0 |
FailedStreams | 1 |
FailedUpgrades | 0 |
HardwareRevision | H6 |
HardwareVersion | H6 |
ID | 534 |
IRLEDsOn | 21054 |
ISPOn | 57564 |
ISPWatchdogCount | 0 |
ISPWatchdogCount2 | 0 |
LogFrequency | 2 |
MotionStreamed | 7561 |
PIREvents | 66 |
PercentAtPlug | 95 |
PercentAtUnPlug | 100 |
PirOorEvents | 0 |
PoweredOn | 11029385 |
SecsPerPercentAvg | 0 |
SecsPerPercentCurr | 0 |
SignalStrengthIndicator | 2 |
SnapshotCount | 0 |
Streamed | 53395 |
SystemFirmwareVersion | 1.096.3.2_30_0bad0df |
SystemSerialNumber | 51D29C77A022A |
SystemVersionString | 1.096.3.2_1.8.65.0_22 |
Temperature | 24 |
TimeAtPlug | 11025490 |
TimeAtUnPlug | 10979418 |
TimeStamp | 2023-10-12T11:23:36+0000 |
Type | status |
UserStreamed | 1650 |
WifiConnectionAttempts | 2 |
WifiConnectionCount | 2 |
WifiCountryDetails | |
carrierFw | Generic |
json | {"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable","ChargerTech":"QuickCharger","ChargingState":"Off","Bat1Volt":7.931,"Temperature":24,"Battery1CaliVoltage":7.931,"Streamed":53395,"UserStreamed":1650,"MotionStreamed":7561,"IRLEDsOn":21054,"PoweredOn":11029385,"CameraOnline":11029337,"CameraOffline":48,"WifiConnectionCount":2,"WifiConnectionAttempts":2,"PIREvents":66,"FailedStreams":1,"FailedUpgrades":0,"SnapshotCount":0,"LogFrequency":2,"CriticalBatStatus":0,"ISPOn":57564,"TimeAtPlug":11025490,"TimeAtUnPlug":10979418,"PercentAtPlug":95,"PercentAtUnPlug":100,"ISPWatchdogCount":0,"ISPWatchdogCount2":0,"SecsPerPercentCurr":0,"SecsPerPercentAvg":0,"PirOorEvents":0,"DdrFailCnt":0,"carrierFw":"Generic","SignalStrengthIndicator":2,"SystemVersionString":"1.096.3.2_1.8.65.0_22","ID":534,"SystemFirmwareVersion":"1.096.3.2_30_0bad0df","HardwareVersion":"H6","SystemSerialNumber":"51D29C77A022A","TimeStamp":"2023-10-12T11:23:36+0000"} |
Here is an emulation you can play with and compare with real data
| makeresults
| eval _raw="178.197.202.134 SECURITY_MONITORING_AUTOMATION CAMERA_SYSTEMS VML4 VML4030 1697109816450 INFO 51D29C77A022A KU5TFBKH-1700-316-114351851 - statusInfo {\"Type\":\"status\",\"HardwareRevision\":\"H6\",\"WifiCountryDetails\":\"\",\"BatPercent\":100,\"BatTech\":\"Rechargeable\",\"ChargerTech\":\"QuickCharger\",\"ChargingState\":\"Off\",\"Bat1Volt\":7.931,\"Temperature\":24,\"Battery1CaliVoltage\":7.931,\"Streamed\":53395,\"UserStreamed\":1650,\"MotionStreamed\":7561,\"IRLEDsOn\":21054,\"PoweredOn\":11029385,\"CameraOnline\":11029337,\"CameraOffline\":48,\"WifiConnectionCount\":2,\"WifiConnectionAttempts\":2,\"PIREvents\":66,\"FailedStreams\":1,\"FailedUpgrades\":0,\"SnapshotCount\":0,\"LogFrequency\":2,\"CriticalBatStatus\":0,\"ISPOn\":57564,\"TimeAtPlug\":11025490,\"TimeAtUnPlug\":10979418,\"PercentAtPlug\":95,\"PercentAtUnPlug\":100,\"ISPWatchdogCount\":0,\"ISPWatchdogCount2\":0,\"SecsPerPercentCurr\":0,\"SecsPerPercentAvg\":0,\"PirOorEvents\":0,\"DdrFailCnt\":0,\"carrierFw\":\"Generic\",\"SignalStrengthIndicator\":2,\"SystemVersionString\":\"1.096.3.2_1.8.65.0_22\",\"ID\":534,\"SystemFirmwareVersion\":\"1.096.3.2_30_0bad0df\",\"HardwareVersion\":\"H6\",\"SystemSerialNumber\":\"51D29C77A022A\",\"TimeStamp\":\"2023-10-12T11:23:36+0000\"}"
``` data emulation above ```
While I wholehaeartedly agree about not using text-based tools for structured data, the problem is that you can't use spath for defining extractions - you can only use it in search. And json or xml kv-extractions work only if the whole event is a well-formed structure. You can't use it if you have - for example - a json event with a syslog header.
There is an open idea about it somewhere on ideas.splunk.com
extractions work only if the whole event is a well-formed structure. You can't use it if you have - for example - a json event with a syslog header.There is an open idea about it somewhere on ideas.splunk.com
I remember seeing you posting a link in Slack but can't find it in a hurry. Maybe more people could vote for such. Be honest, though, seeing all kinds of "smart" schemes developers come up to mix JSON with plain text in this board, it can be tricky to implement a generally application solution.
One workaround for situations like this is to make a data-specific regex field extraction to extract the compliant part for each sourcetype so this regex doesn't have to be included in search commands. This makes easier code maintenance.
Found it.
https://ideas.splunk.com/ideas/EID-I-208
And I don't think it even needs to cover so many use cases. So far we can't do the one, most popular - a structured data blob forwarded wthin a syslog message.
I wiuldn't expect such feature to be too automatic. It could just go with "if the sourcetype has KV_MODE set the STRUCTURED_PART (by default (.*) ) defines which part of the raw event is subjected to KV extraction". And that's it. You could do your own regex to capture everything after the syslog header or whatever you need. Simple yet flexible. Like LINE_BREAKER or TOKENIZER
That STRUCTURED_PART could be entered as a comment. I think it is more practical than a general solution.
the rex command, in my previous reply.. works perfect. pls check it.
As you are a new member, karma points / upvotes are appreciated, pls help those who help you, with your karma points. if any reply solved your issue, pls accept it as "Accept as solution", thanks.
Lets try this (i have tested this on my laptop, its working fine)
source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty"
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>\w+)\"" | table _raw WifiCountryDetails
If we r doing it for stats its not working.... giving the same results as before .... can u pls try for stats count by wificountrydetails instead of table
source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty"
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>\w+)\""
| stats count by WifiCountryDetails
working fine, as suggested previously, check screenshot below.. (pls note the rex command closely.. i have edited ur rex little bit.. i added the ("\"\:\") before rex and (\") after the rex).
I did exactly same but the again its fetching the next field.. pls look into the screenshot
please take the screenshot along with the SPL command also seen inside the image (zoom down or keep the browser window small, so big portion can be copied to the image)
source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty"
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>(\w+|\S))\""
| stats count by WifiCountryDetails
Pls check the above SPL.. i checked it and its working fine.
to troubleshoot further.. can you pls copy paste two sample logs, one with country and one without country.
(pls remove ip address, important details from the sample logs.. just copy paste the line like these..
{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable"
{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"india","BatPercent":100,"BatTech":"Rechargeable"
)
Hi Splunk New beginners:
For absolute beginners...Splunk newbie learning videos..
https://www.youtube.com/@SiemNewbies101/playlists
I have added 25 small videos of rex... Completely for Splunk newbies and beginners.
hope this helps somebody, thanks.
Pls check this SPL:
source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty"
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>(\w+|\S))\""
| eval WifiCountryDetails = if(isnull(WifiCountryDetails) OR len(WifiCountryDetails)==0, "No Country Name", WifiCountryDetails)
| stats count by WifiCountryDetails
No. its not working😑. again its taking the next field value... and we r gathering the count by using stats count by wificountrydetails...
there is : in the place of wificountry details ..... its has to fetch as it is.. but it is fetching next field value batpercent in the place of wificountry details.