We have updated our Terms of Service, Code of Conduct, and Addendum.

How can we get cribl to keep the time within the log?

How can we get cribl to keep the time within the log?

«1

Answers

  • Is your data coming into Splunk without a time field?

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Yeah, I also don't really get the question

  • hgoolya
    hgoolya Posts: 27 mod

    sorry, I was going to add more detail but got called away. The time in the event, is the local time for the system where the log is generated. We do not want Splunk or Cribl to adjust the time

  • hgoolya
    hgoolya Posts: 27 mod

    We thought by using auto timestamp and setting it to local would keep the local tme.

  • What is the timestamp you are expecting from the original event in Cribl? I can think of a few ways to maintain that is its in the Cribl _raw

  • hgoolya
    hgoolya Posts: 27 mod

    There are no other timestamps in the event, other than the first one. Here is a snip of the raw event. 2023/03/07 10:52:49.815 stAgentSvc p581c t34f0 info tunnel.cpp:841 nsTunnel TLS [sessId 1] Tunneling flow

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod

    Local means the timezone on the Cribl worker itself. If you want to adjust the time to UTC, then you'll need to specify UTC in the "Default Timezone" box. I would also encourage you to adjust this on your Event Breaker/Source. For Syslog, this is under Advanced Settings.

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Are Cribl and that source running on the same timezone?

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    If not, how should Cribl (or Splunk) crystalbally guess the timezone on the remote box?

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Also, the display in Splunk is based on the Splunk users timezone setting

  • hgoolya
    hgoolya Posts: 27 mod

    The logs are from systems all over the world, and it looks like the log entry uses the system time.

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Yeah, then that's the issue you have

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Syslog is an old and crappy standard that never included timezones because, to be fair, when it was invented nobody thought about worldwide log collection and stuff

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod

    My $0.02 here... use a lookup to determine the sender's timezone. Use this to then feed to the `C.Time.adjustTz()` function. I do this in the PAN pack. https://github.com/criblpacks/cribl-palo-alto-networks/blob/master/default/pipelines/pan_traffic/conf.yml#L227-L228

  • hgoolya
    hgoolya Posts: 27 mod

    these are not syslog, these are collected via the Splunk UF installed on the system

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Ah, you're right, my concentration was disturbed by a 4 year old gremlin :grin:

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Mh, UFs should transport the system time zone but I vaguely remember that that doesn't work super well and/or wasn't supported by Cribl

  • hgoolya
    hgoolya Posts: 27 mod

    yep, that is what is odd. I do not remember really ever running into this before

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    I assume most of the time your logs had timezone info, or the sender and Cribl where in the same time zone

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod

    Default Time Zone for the Splunk TCP source is also `Local` . <@UQA16GHUN&gt; can you run the command `date` on your Stream worker to see the configured time zone?

  • You can use C.Time.strftime and grab the Cribl raw _time and convert all of it to UTC if you're looking for a singular time zone use. The last parameter of True/False is the setting you'll want to look at

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    The issue here is that these logs come from a timezone different than the Cribl timezone and that zone information is missing from the event

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    So you effectively have to keep a lookup or something else to derive the timezone from hostname etc

  • I see what you're saying

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Or, what you can do, if your logs have no delay, determine difference between recognized timestamp and current time. That delay is likely very close to a multiple of 1 hour. If that's the case, use that multiple and add/subtract it from the timestamp

  • hgoolya
    hgoolya Posts: 27 mod

    Brendan, I am having the linux engineer run that when thye get back, but I am almost 100% positive that it is in UTC. All servers/appliances are set to UTC

  • xpac xpac
    xpac xpac Posts: 148 ✭✭✭

    Same for Cribl? :wink:

  • hgoolya
    hgoolya Posts: 27 mod

    I think what I am going to do is just use index time. It is not ideal, but to keep up with all the 1000's of desktops is not sustainable.

  • David Maislin
    David Maislin Posts: 230 mod

    The latest release of Cribl supports S2S v4 which would accept the OS time when the UF connects to Cribl.

  • David Maislin
    David Maislin Posts: 230 mod

    Does that help at least for the Splunk side of things?