Updating a worker groups leader auth token

Johan Woger

I have changed the auth token on my leader's distributed settings page and the workers in my worker group can no longer connect. what is the proper way to update the worker node's settings to reflect the new auth key?

Brandon McCombs

Be careful with the ./cribl mode-worker command. That has the benefit of encrypting the token but it's for initial configuration (not ongoing changes) of the instance so the repercussions are that it won't preserve any of the settings that have already been defined and stored in the instance.yml file because it writes an entirely new instance.yml.

Leader Node

If you have leader HA enabled and you want the token to be encrypted then you'll need to:

1. disable HA on both nodes
2. update the token on each node using the GUI (otherwise the token won't get re-encrypted
3. re-enable HA

If you have leader HA enabled and you don't care if the token is encrypted then you can simply:

1. update the instance.yml file for each leader node to change the authToken value to the new plain text version of the token
2. then restart each instance.

If leader HA isn't enabled then you can make the leader node authToken update via the GUI or CLI, whichever you prefer.

Clients
To update the associated workers and Edges, you'll have to access them individually to update the auth token. Be aware:

• Doing so via teleport from leader will work but if you mistype the token and restart the instance then you'll be unable to re-teleport to fix it.
• Doing so via the CLI won't encrypt the string but doing it this way can be utilized if you need to automate the change if you have too many Edges/workers to do it manually via the GUI.

Note that the teleport method doesn't currently work for Edge instances (the page doesn't load via teleport using v4.2.1) but it should.

If you have a complicated new token that you don't want to type manually every time then be sure to copy it from the leader while it's visible as plain text in the GUI. Don't copy the encrypted form of it from instance.yml because it's encrypted via the string in cribl.secret that is unique to the leader node and differs per fleet and group.


If you want to use the API to automate this across workers/Edges then use the following:
1. Login to get an API auth token

$ curl -k -X POST https://<FQDN or IP>:9000/api/v1/auth/login -H "Content-Type: application/json" -d '{"username":"<your_username>","password":"<your_password>"}'
{"token":"<your_new_token_displayed_here>","forcePasswordChange":false}

2. Update the cluster authToken. Note that most of the HTTP payload for this step is unique to your configuration so you'll need to watch your browser submit API requests to your instance to know what the fields and values should be because this will write a new instance.yml using what's defined in this payload. The mode value will differ depending on whether it's Stream or Edge.

$ curl -k 'https://<FQDN or IP>:9000/api/v1/system/instance/distributed' -X PATCH -H 'Content-Type: application/json' -H 'Authorization: Bearer your_token_from_the_first_step' -d '{"mode":"worker","group":"default","envRegex":"/^CRIBL_/","master":{"loadBalanced":true,"compression":"gzip","tls":{"disabled":true},"connectionTimeout":10000,"writeTimeout":60000,"tokenTTLMinutes":60,"port":4200,"host":"192.168.1.247","proxy":{"disabled":true},"authToken":"criblmaster","resiliency":"none"},"id":"distributed"}'

3. Then restart the instance:

$ curl -k 'https://<FQDN or IP>:9000/api/v1/system/settings/restart' -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer <your_token_from_the_first_step>' --data-raw '{}'

Jon Rust

CLI:

./cribl mode-worker -H leader_node_hostname -p 4200 -u newtoken -g groupname [-S true]

-S flag is for TLS

Jon Rust

CLI:

./cribl mode-worker -H leader_node_hostname -p 4200 -u newtoken -g groupname [-S true]

-S flag is for TLS

Brandon McCombs

Be careful with the ./cribl mode-worker command. That has the benefit of encrypting the token but it's for initial configuration (not ongoing changes) of the instance so the repercussions are that it won't preserve any of the settings that have already been defined and stored in the instance.yml file because it writes an entirely new instance.yml.

Leader Node

If you have leader HA enabled and you want the token to be encrypted then you'll need to:

1. disable HA on both nodes
2. update the token on each node using the GUI (otherwise the token won't get re-encrypted
3. re-enable HA

If you have leader HA enabled and you don't care if the token is encrypted then you can simply:

1. update the instance.yml file for each leader node to change the authToken value to the new plain text version of the token
2. then restart each instance.

If leader HA isn't enabled then you can make the leader node authToken update via the GUI or CLI, whichever you prefer.

Clients
To update the associated workers and Edges, you'll have to access them individually to update the auth token. Be aware:

• Doing so via teleport from leader will work but if you mistype the token and restart the instance then you'll be unable to re-teleport to fix it.
• Doing so via the CLI won't encrypt the string but doing it this way can be utilized if you need to automate the change if you have too many Edges/workers to do it manually via the GUI.

Note that the teleport method doesn't currently work for Edge instances (the page doesn't load via teleport using v4.2.1) but it should.

If you have a complicated new token that you don't want to type manually every time then be sure to copy it from the leader while it's visible as plain text in the GUI. Don't copy the encrypted form of it from instance.yml because it's encrypted via the string in cribl.secret that is unique to the leader node and differs per fleet and group.


If you want to use the API to automate this across workers/Edges then use the following:
1. Login to get an API auth token

$ curl -k -X POST https://<FQDN or IP>:9000/api/v1/auth/login -H "Content-Type: application/json" -d '{"username":"<your_username>","password":"<your_password>"}'
{"token":"<your_new_token_displayed_here>","forcePasswordChange":false}

2. Update the cluster authToken. Note that most of the HTTP payload for this step is unique to your configuration so you'll need to watch your browser submit API requests to your instance to know what the fields and values should be because this will write a new instance.yml using what's defined in this payload. The mode value will differ depending on whether it's Stream or Edge.

$ curl -k 'https://<FQDN or IP>:9000/api/v1/system/instance/distributed' -X PATCH -H 'Content-Type: application/json' -H 'Authorization: Bearer your_token_from_the_first_step' -d '{"mode":"worker","group":"default","envRegex":"/^CRIBL_/","master":{"loadBalanced":true,"compression":"gzip","tls":{"disabled":true},"connectionTimeout":10000,"writeTimeout":60000,"tokenTTLMinutes":60,"port":4200,"host":"192.168.1.247","proxy":{"disabled":true},"authToken":"criblmaster","resiliency":"none"},"id":"distributed"}'

3. Then restart the instance:

$ curl -k 'https://<FQDN or IP>:9000/api/v1/system/settings/restart' -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer <your_token_from_the_first_step>' --data-raw '{}'