Getting Data In
Highlighted

What configuration is required to index a single log with one event only, transforms.conf or props.conf?

New Member

Hi,
My query is that Splunk indexer is indexing a single log with two separate events whereas it should be one event only.
The issue is that I am receiving two timestamps in a single log and I need Splunk to index it as a single event only.

Full Event Expected:-

[2018-01-31T15:23:25.470+04:00]...abc.....def...........ghi...........
......................................................................
......................................................................
  <ABC xmlns="http://tempuri.org/">
    <A>
      <ID>1234567</ID>
      <tickets>
        <DEF>
          <ticketNumber>12345</ticketNumber>
          <paidAmount>100</paidAmount>
          <paymentDateTime>2015-02-10T15:25:19Z</paymentDateTime>
          <receiptNumber>987654321</receiptNumber>
        </DEF>
      </tickets>
    </A>
  </ABC>

Received Event 1:-

[2018-01-31T15:23:25.470+04:00]...abc.....def...........ghi...........
......................................................................
......................................................................
  <ABC xmlns="http://tempuri.org/">
    <A>
      <ID>1234567</ID>
      <tickets>
        <DEF>
          <ticketNumber>12345</ticketNumber>
          <paidAmount>100</paidAmount>

Received Event 2:-

      <paymentDateTime>2015-02-10T15:25:19Z</paymentDateTime>
      <receiptNumber>987654321</receiptNumber>
    </DEF>
  </tickets>
</A>

Could anyone please suggest me how to proceed with this and what parameters to use for configuring props.conf or ?transforms.conf(if required)?

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

SplunkTrust
SplunkTrust

Hi @AdsicSplunk,

Please try below configuration in props.conf on Indexer/Heavy Forwarder whichever comes first.

props.conf

[yoursourcetype]
TIME_PREFIX = ^\[
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%:z
MAX_TIMESTAMP_LOOKAHEAD = 29

Restart splunk on Indexer/Heavy Forwarder.

I hope this helps.

Thanks,
Harshil

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

New Member

Hi Harsmarvania,

Thank you for your support!!

I tried this config in props.conf but it got worse for me. Now, my indexer is creating even more events breaking each line and putting each line in a separate event. My question is that my event should not break into 2 events but should create one event only ignoring the second timestamp coming inside the event. Please read my questions, if you need some clarifications on this. please feel free to ask me questions.

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

SplunkTrust
SplunkTrust

Hi @AdsicSplunk,

I tried with below configuration in splunk

props.conf

[mysourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N%:z
TIME_PREFIX=^\[
MAX_TIMESTAMP_LOOKAHEAD=29

And it is working perfectly fine.

Please refer screenshot https://imgur.com/a/BnQJ9

If this does not work for you, can you please let us know whether do you have any whitespace before [2018-01-31T15:23:25.470+04:00] ?

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

New Member

Hi @harsmarvania57,

This is the same config that Adonio provided.
I tried this config but now the events are created not based on [2018-01-31T15:23:25.470+04:00] which was being picked without props.conf config earlier. Now, this config has scattered the events and displaying the results which are not at all good. My requirement is to create events based on [2018-01-31T15:23:25.470+04:00] not based on 2015-02-10T15:25:19Z. Please provide your inputs.

Note:- There is no space in the timestamp. The data begins with "[" only.

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

New Member

The screenshot is not accessible. Could you please share again?

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

SplunkTrust
SplunkTrust

If you look at TIME_FORMAT parameter closely in config which is provided by @adonio and config which I have provided then there are difference . You can see screenshot here as well https://prnt.sc/i8i709

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

New Member

I tried with both configs, still I am not getting what is required. Anyway, Thank you for your support, I will try again to get to the desired requirement.

Really appreciate your support.

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

SplunkTrust
SplunkTrust

hello there:

in inputs.conf:

[monitor://path.to.file]
index = index
sourcetype = your_sourcetype

in props.conf on indexer or heavy forwarder:

 [your_sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m%dT%H:%M%s.%3N:%z
TIME_PREFIX=[
MAX_TIMESTAMP_LOOKAHEAD=30
BREAK_ONLY_BEFORE=\

further reading regarding where to place files and which configurations goes in each file here:
https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

hope it helps

0 Karma
Highlighted

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

New Member

Hi Adonio,

Thank you for your support!!
i tried this config but now the events are created not based on [2018-01-31T15:23:25.470+04:00] which was being picked without props.conf config. Now, this config has scattered the events and displaying the results which are not at all good. My requirement is to create events based on [2018-01-31T15:23:25.470+04:00] not based on 2015-02-10T15:25:19Z. Please provide your inputs.

0 Karma