Getting Data In

props.conf

EHariharan
Explorer

Hi All,

can anyone help us to figure out magic six for the below sample log?

SHOULD_LINEMERGE=
LINE_BREAKER=
MAX_TIMESTAMP_LOOKAHEAD=
TIME_PREFIX=
TRUNCATE=
TIME_FORMAT=

VersionNumber=7.2 build 13. Maint HF-005,Priority=N/A,LocalTranNumber=I32790D942,RemoteTranNumber=N/A,TransferStartTime=003940,TransferStartDate=20190327,

Thanks in Advance!

Tags (1)
0 Karma
1 Solution

jnudell_2
Builder

Hi @EHariharan ,
The answer provided by @ragedsparrow is how you should approach data onboarding. However, if you're the one tasked with owning the data, and you're just given some events you might try these values:


SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = TransferStartTime=
TIME_FORMAT = %H%M%S,TransferStartDate=%Y%m%d
TRUNCATE = 10000
MAX_TIMESTAMP_LOOKAHEAD = 40

View solution in original post

0 Karma

jnudell_2
Builder

Hi @EHariharan ,
The answer provided by @ragedsparrow is how you should approach data onboarding. However, if you're the one tasked with owning the data, and you're just given some events you might try these values:


SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = TransferStartTime=
TIME_FORMAT = %H%M%S,TransferStartDate=%Y%m%d
TRUNCATE = 10000
MAX_TIMESTAMP_LOOKAHEAD = 40

0 Karma

EHariharan
Explorer

Thanks, jnudell_2. It worked

Also, I would like to thank ragedsparrow and FrankVl .

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

The "magic six" are usually defined by you or whomever knows the data. Essentially, you are telling Splunk where to break the events and how to identify the timestamps for indexing.

I suggest you do this;

  • Identify what constitutes a new event.
  • Identify what the timestamp for the event is in the event.

Just looking at that event, the TIME_FORMAT might look like this:
TIME_PREFIX = TransferStartTime=
TIME_FORMAT = %H%M%S,TransferStartDate=%Y%m%d

You may not HAVE to use everything form the Magic 6, but you should try to if you can.

What I usually do is bring in sample data into a standalone instance (usually running on my laptop) and use the "Add Data" ability to bring in the data and test out props before I deploy them out.

0 Karma

FrankVl
Ultra Champion

I'd move TransferStartTime= to the TIME_PREFIX setting, other than that I had the same suggestion in mind 🙂

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

Yeah, that is much better. Modified my answer to reflect that.

0 Karma

FrankVl
Ultra Champion

What have you tried so far and what problems are you running into? This is a platform for asking questions, not for asking other people to do your job for you 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...