Getting Started

Motivation

1. ETA (expected turnaround time to market) is very low, which gives a competitive advantage.
2. Growing out of the box integrations with popular SaaS products like Cidaas, AWS, and Microsoft Teams.
3. cnips demands fewer skilled developers, which in turn makes it easy to understand and operate
4. cnips enables quick deployment of new features and integrations, helping businesses stay agile and responsive to market changes.
5. cnips includes robust security features, ensuring data protection and compliance with industry standards.
6. The platform offers real-time monitoring and analytics, providing valuable insights and enabling proactive decision-making.

Introduction to cnips

This section will take you through overall cnips with its introduction to modules such as sources, destination, etc. with their terms and terminology.

Terms	Definition
Pipelines	Manage and monitor data flow processes, connecting sources to destinations.
Sources	Configure and manage the data inputs from various sources.
Destinations	Set up and manage where the processed data is sent or stored.
Transformations	Apply modifications or transformations to the data as it moves through the pipeline.
Apps	Integrate and manage additional applications or services.
Configurations	Customize settings and configurations for the system and modules.
Decision blocks	A feature in a pipeline system that allows you to add custom logic to process or transform data dynamically.

The home page of the cnips will display you all the overview key metrics about the events happening under single dashboard. From here, you can navigate to each module and get an update about each operation.

Below is an example of the cnips home page.

Here's a small description for each card on the dashboard:

• Pipelines: View and manage 37 active pipelines, essential for data flow and processing.

• Sources : Monitor 32 connected data sources feeding information into your system.

• Destinations : Track 47 data destinations where processed data is being sent.

• Transformations: Oversee 25 data transformations, ensuring data is structured and ready for analysis.

Inbound Events:

• Events Received : Track the number of inbound events received in the last 24 hours.
• Events Processed : Shows how many of the inbound events were processed successfully.
• Processed : Displays the percentage of processed events, ensuring no pending items.
• Events Failed : Indicates the number of inbound events that failed to process.

Outbound Events:

• Events Queued : Displays the count of outbound events queued for processing.
• Events Processed : Shows the number of outbound events successfully processed.
• Processed : Indicates the completion rate for outbound events, ensuring efficient data flow.
• Events Failed : Highlights any failed outbound events, ensuring reliability.

Source is nothing but the first data ingestion point

Example: Connecting to an API endpoint or database like MongoDB to extract customer data.

Source Name: A field where you provide a name for the new source. Example: Customer Data API

Response Headers (Key/Value): These fields allow you to specify custom HTTP response headers. The "Key" field is for the header name, and the "Value" field is for the corresponding value.

Example: key: content_type and value : application/json.

Secured: A toggle to indicate whether the source should use secure protocols (e.g., HTTPS).

Select a Configuration: Allows you to choose an existing configuration or add a new one by using the search (+) icon.

Example: Choose a configuration like AWS S3 for accessing Amazon S3 storage or click the + icon to create a new configuration such as Custom OAuth Authentication.

Active: A toggle to activate or deactivate the source.

Capture Remote IP: A toggle to specify if the remote IP should be captured.

Submit Button: Press this button to submit and create the new source.

How to create new source

Destination is the nothing but the address of the processed data that is loaded.

Example: Loading transformed data into a cloud storage platform like AWS S3 or an analytics tool like Google BigQuery.

Destination Name: The name given to the destination.

Example: "DEMO_TEST_DESTINATION"

Configuration: Key-value pairs for destination settings.

Example: API_KEY → 123456

Select Scripting Language: Choose the language for custom scripting.

Example: JavaScript Script: Custom code for setting up, executing, and tearing down the connection.

• Example setup: A function that configures API access.

• Example execute: Code that sends data to the destination.

Active: Toggle to enable or disable the destination.

How to create new Destination

Transformation is the operations that modify or clean the data before sending it to the destination.

Example: Adding customer name to the customer order in a cafe .

Transformation Name: The name given to the transformation.

Example: "TEST_CUSTOMER_NAME"

Configuration: Key-value pairs for transformation settings.

Example: API_KEY → 123456

Select Scripting Language: Choose the language for custom scripting.

Example: JavaScript

• You can add custom logic inside the function to transform or enrich the event object (e.g., modify or add data).
• The function must return the modified (or original) event object so the pipeline can continue processing.

Active: Toggle to enable or disable the transformation.

How to create new Transformation

A configuration for moving data from a source to a destination, potentially applying transformations. It can be automated or triggered based on events or schedules

Example: A pipeline fetching data from an SQL database, transforming it, and sending it to a data warehouse like Snowflake.

Source name: Select or Add the starting point of the data flow, where data is fetched or ingested.

Destination: Select or Add the endpoint of the data flow, where the processed data will be delivered or stored.

Active: A toggle to enable or disable the pipeline.

Scheduled Batch Processing: A toggle to enable or disable scheduled batch runs of the pipeline.

How to create new Pipeline

Decision blocks are the conditional logic used to determine different actions based on data conditions or values.

Example: If a user’s subscription status is "expired," route their data to a retention campaign workflow; otherwise, route it to a reporting tool.

Decision Block Name : A suitable decision block name.

Configuration: Allows you to define additional parameters as key-value pairs

Example:

• Key: api_url

• Value: https://api.example.com/endpoint

The "Secret" checkbox ensures sensitive values (e.g., passwords, API keys) are encrypted and hidden.

Select Scripting Language: Dropdown to choose the scripting language for the custom logic.

Script: An editor where you can write the code to transform or enrich the event data

Integrated services or tools that can be utilized within the platform.

Example: Integrating Salesforce or Slack to trigger notifications based on certain data events.

Configuration is a reusable key-value pair setup that acts as a shared resource for pipelines, sources, destinations, or other blocks within the system.

Example: API keys, database connection strings, or authentication credentials stored securely for data access.

Name: A unique identifier for the configuration. Description: A brief summary of the configuration's purpose.

Select Configuration Type:Specifies the category or type of configuration

Select API Access Type: Defines the authentication method for the API.

Client ID: A unique identifier provided by the API provider for authentication.

Client Secret: A confidential key used in combination with the client ID to authenticate API requests securely.

Scopes: Permissions or access levels required for the API. Scopes specify what the application can do on behalf of the user.

well-known Url: A URL that provides metadata about the OAuth2/OpenID Connect configuration, like token endpoints or authorization endpoints.

Active: A toggle to enable or disable the configuration.