In the ever-evolving world of data analysis, the ability to interact directly with live API endpoints is a significant advancement for practitioners. Cribl Search offers this capability, enhancing your data analysis toolkit. This feature allows you to gain broader visibility into the periphery of your infrastructure, enabling a more comprehensive analysis of user journeys and operational trends.
By querying live API endpoints, you can seamlessly integrate real-time data into your analysis, ensuring your insights are as current and relevant as possible. This is especially valuable for connecting disparate data points across various platforms and applications. Whether you’re monitoring user interactions, evaluating system performance, or tracking application usage, directly integrating live API data into Cribl Search provides a more dynamic and holistic approach to data exploration.
This guide will walk you through setting up Azure API as a dataset provider to Cribl Search. These steps leverage this enhanced capability to enrich your data analysis and decision-making processes.
Step 1: Registering the Application and Service Principal in Azure
Create an Azure service principal, an identity for your applications and tools to access Azure resources. Follow these steps to register:
- Go to the Azure portal and access ‘App registrations.’
- Select ‘New registration’ and provide the necessary details.
- Note the Application (client) ID and Directory (tenant) ID for later use.
In this image, the Azure app search_api has a Service Provider with a display name search_api
Reference: Create a service principal in Azure.
Step 2: Assigning Roles to the Service Provider
Assign the right roles to your service principal for appropriate access levels. You can opt for the ‘Reader’ role or a custom role for specific permissions.
- In Azure, navigate to ‘Subscriptions’ and select yours.
- Go to ‘Access control (IAM)’ and choose ‘Add role assignment’.
- Select the ‘Reader’ role or create a custom role.
In this image, the search_api Service Provider is assigned the Role Reader
Reference: You can assign the built-in role of Reader to the application so it has read access to all endpoints. To limit access to the current Search endpoints (listed in Cribl Search docs), create a custom role: Tutorial: Create an Azure custom role with Azure PowerShell – Azure RBAC
Step 3: Creating the Azure API Dataset Provider
- In Cribl Search, navigate to Data → Dataset Providers.
- Click ‘Create Provider’.
- If prompted with a drop-down menu for Stream Worker Groups or Data Lake Amazon S3 Destinations, proceed by clicking ‘Create.’
Configuring the New Dataset Provider:
- Set the ID as a unique identifier for the dataset provider.
- The Description field is optional.
- Choose Azure API as the Dataset Provider Type.
- Click ‘Add Configuration’ to enter your Azure account details:
- Account Name: Name of your Azure account.
- Tenant ID: ID of your Azure Active Directory.
- Client ID: ID of the application connecting to Azure Active Directory.
- Client Secret: Secret key for the connection.
- Save your configurations.
Step 4: Creating the Dataset
Adding a New Dataset:
- In Cribl Search, go to Data → Datasets.
- Click ‘Add Dataset’.
Configuring the New Dataset:
- Set the ID as a unique identifier for the dataset.
- The Description field is optional.
- Choose the Azure dataset provider you created earlier as the Dataset Provider.
- Click ‘Add endpoint’ to select your desired endpoints (virtual machines, disks, networkSecurityGroups, web apps).
- Enter the Subscription IDs you wish to query.
- Under Processing, set up Datatypes for data organization and field definition.
Reference: Azure API | Cribl Docs
Step 5: Start Searching
With your dataset provider and dataset configured, you’re now ready to explore your data. Search results can appear in seconds, depending on the volume of data in your account.
Original credit: Emil Mikhailov's Blog
