would it be possible to use the same encryption key in an Edge fleet and in a Stream worker group ?

rrinehart

Hi there,Question about encryption keys: would it be possible to use the same encryption key in an Edge fleet and in a Stream worker group ?I have a use case where we need to encrypt data at the source (Edge) and send it to a S3 compatible storage. Then I should be able to retrieve data from the S3 bucket using a collector Job that would be ran from a Stream worker group and then use a pipeline for decryption. Does it make sense ?For those of you who are going to tell me to forward data from Edge to Stream and then use Encrypt function before sending to S3 I would answer to them that the Sec team insists to encrypt data the closest to the source

xpac xpac

Are you talking about event-level encryption, like using `C.Crypto.encrypt()` ?

rrinehart

yes i am

xpac xpac

Then, yes, considering you use the same keys on all instances

rrinehart

Ok I thought cribl used different secrets for each and every group/fleet

maybe I missed something ...

xpac xpac

Well there's multiple things, but cribl.secret isn't the same es the encryption key for data

You can even use the Cribl app on Splunk to do the decryption in Splunk while searching

Brendan Dalpe

Playing devil's advocate here... been on the infosec side too... Do they mean encrypting the data in transit (i.e. TLS/SSL)?

That's a huge performance penalty for encrypting the actual data itself and would prevent you from doing granular filtering of data in S3 if you ever wanted to replay it.

And distributing the keys to every Edge node would in theory allow an attacker to just decrypt the data if they were able to obtain the key pairs.

rrinehart

data we are dealing with are sensitive and need to be stored on WORM S3 for 10 years. Replying the data is actually not a big deal because of the unencrypted metadata added to each event. I am encrypting _raw field only.Regarding the last point I am considering using an external KMS such as Hashicorp Vault.Last but not least it is a very specific use case, we are talking about 4 Edge clients here for a daily volume around 100MB

transit is already secured with TLS

Brendan Dalpe

Ah, okay. I feel way better about your use case here.

Hashicorp Vault is good, but your Edge clients would need to be able to speak with it to get the `cribl.secret` file.

We don't support external symmetric key storage yet

xpac xpac

are the encryption keys stored in cribl.secret?

ah :slightly_smiling_face:

Brendan Dalpe

No, but they work together

rrinehart

So if I understand correctly, since each workerGroup/fleet has it's own secret I cannot use the exact same encryption key between my fleet and my worker group

xpac xpac

Per my understanding, the `cribl.secret` is only used to store the encrypted representation of the actually used encryption key

It would make no sense to use some randomly generated secret to encrypt the data, because then you wouldn't be able to decrypt it later (without pulling the cribl.secret from all your boxes)

rrinehart

Ok but when you generate an Encryption key for a Edge fleet you have no way to import it to your Worker Group Security settings. But again maybe I missed something ...

Because the leader node is creating a secret for each group/fleet

xpac xpac

Okay, maybe I'm the one talking bullshit here, lemme try something

rrinehart

If you find a way to do it next round is on me

xpac xpac

Okay, I gotta admit this seems a little extra complicated to me

So what you should be able to do is create a key, and then download the key bundle, and manually place it on the leader for the other group

If that group already has encrypted stuff in its config, you might have to reset that

it's not a very pretty solution, but should work

rrinehart

I don't think you can because the secret also used to generate a randomized user/pwd to communicate with the nodes of the group/fleet when you teleport to them, per my understanding

so if you override the secret file of a group I am pretty sure that you break everything

xpac xpac

maybe not used to generate it, but possibly to store it

It seems to me nobody thought about "encrypt to store in S3 and later decrypt when replaying", where those two actions don't happen on the same worker group...