Splunk Enterprise

How do I Insert custom field and value to splunk?

jordilazo
Explorer

Hi,

Im dummy in Splunk and I have one doubt. Maybe you can help me.

I want to insert in an index that I have created some data that I have obtained when executing a script in python, so the result of the script is the following:

 

 

sourcetype="script_emails" mail_sender="jordi@jordilazo.com" mail_recipient="jordilazo2@jordilazo.es" mail_date="10-10-2022" mail_subject="RE: NMXWZFOG< >VSTI" mail_reviewcomment="Comment:ÑC<AZR=@P"&"\A"

 

 

How do I configure the inputs, props and transform so that it is uploaded correctly in Splunk?

- Field - Value
- Source
- Sourcetype

I have this:

 

inputs.conf

 

 

[script://"script.py"]
disabled = 0
index = python_emails
interval = 22 13 * * *
source = ????(I dont know what to insert here)
sourcetype = mytest

 

 

 

transform.conf

 

 

[test_sourcetype]
REGEX = sourcetype="(\w+)"
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[test_comment]
REGEX = mail_reviewcomment="(.+)"
FORMAT = mail_reviewcomment::$1
WRITE_META = true

 

 

 

props.conf

 

 

[mytest]
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
TIME_PREFIX = timestamp=
MAX_TIMESTAMP_LOOKAHEAD = 10
CHARSET = UTF-8
KV_MODE = auto
TRANSFORMS-test_sourcetype = test_sourcetype,test_comment

 

 

Thanks for you help!

Labels (2)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

If you want you can add source and sourcetype to inputs.conf. If not then splunk use script name for those.

f you have several source types returned by this script (which actually means that output of this script is different) then you should use those transforms.conf settings. You can also look INGEST_EVAL command to pick up those values from input stream.

If/when you are not setting sourcetype on inputs.conf you must change in [mytest] to [source::.../<script.py>] or [script.py]  to match those events.  

You have defined that timestamp (_time) has picked from field timestamp, but I don't see it on your script output. If it's missing then ok otherwise change the filename for it (mail_date?) and check that you have correctly formatted TIME_FORMAT for it.

Test the script by running it on source host as "sudo -u<splunk user> /opt/splunk/bin/splunk cmd /path/to/your/script". This must work to use it as an scripted inputs.  Maybe you need to change the path in your inputs.conf stanza if it cannot find it by "script.py".

r. Ismo

View solution in original post

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

If you want you can add source and sourcetype to inputs.conf. If not then splunk use script name for those.

f you have several source types returned by this script (which actually means that output of this script is different) then you should use those transforms.conf settings. You can also look INGEST_EVAL command to pick up those values from input stream.

If/when you are not setting sourcetype on inputs.conf you must change in [mytest] to [source::.../<script.py>] or [script.py]  to match those events.  

You have defined that timestamp (_time) has picked from field timestamp, but I don't see it on your script output. If it's missing then ok otherwise change the filename for it (mail_date?) and check that you have correctly formatted TIME_FORMAT for it.

Test the script by running it on source host as "sudo -u<splunk user> /opt/splunk/bin/splunk cmd /path/to/your/script". This must work to use it as an scripted inputs.  Maybe you need to change the path in your inputs.conf stanza if it cannot find it by "script.py".

r. Ismo

0 Karma

jordilazo
Explorer

Hi isoutamo,

Thanks for your information!!

Thanks to you I have been able to upload the data correctly except for a small error.
In the field: mail_reviewcomment I have a = which makes splunk automatically create a new field for me without me asking it.
Is there any way to be able to insert the = symbol to the splunk and at the same time not create a new field? Thank you.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

I think that you must add search time extraction for that field, otherwise it use = as key value separator. Just add it to search head with props.conf or with gui.

0 Karma

jordilazo
Explorer

Sorry is the first time I am doing this.

Could you explain it with more detail?

Where should I add EXTRACT? as you said in the props.conf but exactly where and with what parameters?

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...