would it be possible to use the same encryption key in an Edge fleet and in a Stream worker group ?
Hi there,Question about encryption keys: would it be possible to use the same encryption key in an Edge fleet and in a Stream worker group ?I have a use case where we need to encrypt data at the source (Edge) and send it to a S3 compatible storage. Then I should be able to retrieve data from the S3 bucket using a collector Job that would be ran from a Stream worker group and then use a pipeline for decryption. Does it make sense ?For those of you who are going to tell me to forward data from Edge to Stream and then use Encrypt function before sending to S3 I would answer to them that the Sec team insists to encrypt data the closest to the source 
Answers
-
Are you talking about event-level encryption, like using `C.Crypto.encrypt()` ?
0 -
yes i am
0 -
Then, yes, considering you use the same keys on all instances
0 -
Ok I thought cribl used different secrets for each and every group/fleet
maybe I missed something ...
0 -
Well there's multiple things, but cribl.secret isn't the same es the encryption key for data
You can even use the Cribl app on Splunk to do the decryption in Splunk while searching
0 -
Playing devil's advocate here... been on the infosec side too... Do they mean encrypting the data in transit (i.e. TLS/SSL)?
That's a huge performance penalty for encrypting the actual data itself and would prevent you from doing granular filtering of data in S3 if you ever wanted to replay it.
And distributing the keys to every Edge node would in theory allow an attacker to just decrypt the data if they were able to obtain the key pairs.
0 -
data we are dealing with are sensitive and need to be stored on WORM S3 for 10 years. Replying the data is actually not a big deal because of the unencrypted metadata added to each event. I am encrypting _raw field only.Regarding the last point I am considering using an external KMS such as Hashicorp Vault.Last but not least it is a very specific use case, we are talking about 4 Edge clients here for a daily volume around 100MB
transit is already secured with TLS
0 -
Ah, okay. I feel way better about your use case here.
Hashicorp Vault is good, but your Edge clients would need to be able to speak with it to get the `cribl.secret` file.
We don't support external symmetric key storage yet
0 -
are the encryption keys stored in cribl.secret?
ah :slightly_smiling_face:
0 -
No, but they work together
0 -
So if I understand correctly, since each workerGroup/fleet has it's own secret I cannot use the exact same encryption key between my fleet and my worker group
0 -
Per my understanding, the `cribl.secret` is only used to store the encrypted representation of the actually used encryption key
It would make no sense to use some randomly generated secret to encrypt the data, because then you wouldn't be able to decrypt it later (without pulling the cribl.secret from all your boxes)
0 -
Ok but when you generate an Encryption key for a Edge fleet you have no way to import it to your Worker Group Security settings. But again maybe I missed something ...
Because the leader node is creating a secret for each group/fleet
0 -
Okay, maybe I'm the one talking bullshit here, lemme try something
0 -
If you find a way to do it next round is on me
0 -
Okay, I gotta admit this seems a little extra complicated to me
So what you should be able to do is create a key, and then download the key bundle, and manually place it on the leader for the other group
If that group already has encrypted stuff in its config, you might have to reset that
it's not a very pretty solution, but should work
0 -
I don't think you can because the secret also used to generate a randomized user/pwd to communicate with the nodes of the group/fleet when you teleport to them, per my understanding
so if you override the secret file of a group I am pretty sure that you break everything
0 -
maybe not used to generate it, but possibly to store it
It seems to me nobody thought about "encrypt to store in S3 and later decrypt when replaying", where those two actions don't happen on the same worker group...
0
