Getting Data In

Would like to reformat event at ingestion time to avoid rex/rename in queries

elumpkinTnaa
Explorer

Currently the app I'm working on generates log events in the following (simplified/obfuscated) format before they are ingested into Splunk:

2021-09-24 19:00:00.016 +00:00 [Warning] Something.SomethingElse.YetAnotherThing: jsonData={ "alice": "Alison", "bob": "Bobby", "group" : {"joe": "Joseph", "jane": "Janet"}}

The only bits of those log events which are important are the timestamp at the leftmost end, and the well-formed json data after the equal sign. 

What I wish was possible is to change the event being created by the application to be only a well-formed JSON object that included the timestamp.  In other words something like this....

{"_time":"2021-09-24 19:00:00.016 +00:00", "alice": "Alison", "bob": "Bobby", "thing" : {"joe": "Joseph", "jane": "Janet"}}

But that is it's own challenge (outside of Splunk) which will take me time to make happen.  In the meantime I wonder if there is something I could setup in Splunk so that, at ingestion time, the original log event was transformed into that latter format.  This would save me from having to do rex & rename commands like this as part of each an every splunk query I want to run.  Which is not only annoying, but I'm guessing slows down the queries as well.

host="something"

| rex "jsonData=(?<jsonData>.+})"

| rename jsonData as _raw

| spath

| search event="*Exception*" 

Is this possible?  Furthermore is this possible given that the events are ingested from Azure via the plugin: Splunk Add-on for Microsoft Cloud Services?

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try using SEDCMD.  Put this in your props.conf file (in the appropriate stanza):

SEDCMD-keepJSON = s/\[.*jsonData/jsonData/
---
If this reply helps you, Karma would be appreciated.

elumpkinTnaa
Explorer

It seems I left out a relevant fact, we are using Splunk Cloud and therefore don't have access to that file.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That is key information to not leave out, but you still can update props.conf.  Put the file in an app and install the app.

---
If this reply helps you, Karma would be appreciated.

jconger
Splunk Employee
Splunk Employee

It's possible to do this type of thing in the UI too.  Go to Settings > Source Types > choose the sourcetype > Advanced > New setting > Save

cloud_sourcetype.png

elumpkinTnaa
Explorer

Due to my lack of familiarity with the props.conf file this is a very attractive solution.  Thanks!

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...