Data Extractor

A data extractor helps pull data into the pipeline for processing, instead of using the push mechanism to send data from your source. You can add Javascript or Go code with required modules to connect to your data source and extract the data for processing. For example, you can use a data extractor to retrieve data from a database and send it to a destination.

Additionally, you can associate a schedule with your extractor script to run the data extraction process at regular intervals. It supports selecting a time zone, defining custom schedules using cron expressions, and providing preset intervals like "Every 15 mins" or "Every hour." Users can customize the schedule further by specifying exact minutes, hours, days, or months for task execution. This ensures precise and efficient scheduling for data ingestion into pipelines.

How to create a Data Extractor

To create a Data Extractor:

1. Follow the steps to create a source
2. Click on Data Extractor

3. Provide a name for the Data Extractor .
4. Select a configuration.
5. Select the scripting language.
6. Modify script to return event data, making sure to send it as an array.
7. Click on sumbit.

How to setup a Data Extractor schedule

1. Click on setup schedule after creating the data extractor.

2. Select timezone.
3. Select interval.
4. Save schedule.