Skip to content

Commit

Permalink
Merge branch 'main' into patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
hhanova authored Jan 4, 2024
2 parents 2a524dc + 6a9572b commit 952f624
Show file tree
Hide file tree
Showing 219 changed files with 2,119 additions and 1,081 deletions.
48 changes: 31 additions & 17 deletions _data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,36 +9,32 @@ items:
title: Getting Started Tutorial
items:
- url: /tutorial/load/
title: Part 1 - Loading Data
title: "Part 1: Loading Data"
items:
- url: /tutorial/load/googledrive/
title: Loading with GoogleDrive Extractor
- url: /tutorial/load/googlesheets/
title: Loading Data from Google Sheets

- url: /tutorial/load/database/
title: Loading with Database Extractor
title: Loading Data from Database

- url: /tutorial/manipulate/
title: Part 2 - Data Manipulation
title: "Part 2: Data Manipulation"
items:
- url: /tutorial/manipulate/sandbox/
title: Using Sandbox
- url: /tutorial/manipulate/workspace/
title: Using Workspace

- url: /tutorial/write/
title: Part 3 - Writing Data
items:
- url: /tutorial/write/gooddata/
title: Writing to GoodData
title: "Part 3: Writing Data"

- url: /tutorial/automate/
title: Part 4 - Automation
title: "Part 4: Flow Automation"

- url: /tutorial/ad-hoc/
title: Part 5 - Ad-Hoc Data Analysis
title: "Part 5: Ad-Hoc Data Analysis"

- url: /tutorial/branches/
title: Part 6 - Development Branches
title: "Part 6: Development Branches"
items:

- url: /tutorial/branches/prepare-tables/
title: Prepare Tables

Expand All @@ -57,11 +53,29 @@ items:
- title: Merge to Production
url: /tutorial/branches/merge-to-production/

- url: /tutorial/onboarding/
title: Keboola Platform Onboarding
items:
- url: /tutorial/onboarding/usage-blueprint/
title: Usage Blueprint

- url: /tutorial/onboarding/architecture-guide/
title: Multi-Project Architecture
items:
- url: /tutorial/onboarding/architecture-guide/bdm-guide/
title: Business Data Model

- url: /tutorial/onboarding/cheat-sheet/
title: Best Practices

- url: /tutorial/onboarding/governance-guide/
title: Governance Guide

- url: /components/
title: Components
items:
- url: /components/extractors/
title: Extractors
title: Data Source Connectors
items:
- url: /components/extractors/communication/
title: Communication
Expand Down Expand Up @@ -301,7 +315,7 @@ items:
title: YourPass

- url: /components/writers/
title: Writers
title: Data Destination Connectors
items:
- url: /components/writers/bi-tools/
title: Business Intelligence
Expand Down
11 changes: 11 additions & 0 deletions _includes/tip.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<div class="clearfix"></div>
<div class="alert alert-info" role="alert">
<i class="fa fa-info"></i>
{% if include.title %}
{% capture title %}{{"&#9432;&nbsp; **Tip: " | append: include.title | append: "**"}}{% endcapture %}
{% else %}
{% capture title %}&#9432;&nbsp; **Tip**{% endcapture %}
{% endif %}
{{ title | markdownify}}
{{include.content | markdownify}}
</div>
5 changes: 4 additions & 1 deletion _includes/warning.html
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
<div class="clearfix"></div>
<div class="alert alert-warning" role="alert">
<i class="fa fa-warning"></i>
<strong>Warning:</strong> {{include.content}}
&#9432; <strong>Important:</strong>
<div>
{{include.content | markdownify}}
</div>
</div>
2 changes: 1 addition & 1 deletion _sass/variables.scss
Original file line number Diff line number Diff line change
Expand Up @@ -870,7 +870,7 @@ $headings-small-color: $gray-light !default;
//** Blockquote small color
$blockquote-small-color: $gray-light !default;
//** Blockquote font size
$blockquote-font-size: ($font-size-base * 1.25) !default;
$blockquote-font-size: $font-size-base !default;
//** Blockquote border color
$blockquote-border-color: $gray-lighter !default;
//** Page header border color
Expand Down
4 changes: 4 additions & 0 deletions assets/css/style.scss
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,10 @@ img {
}
}

blockquote {
color: $gray-light;
}

.footer {
.inside {
text-align: center;
Expand Down
36 changes: 18 additions & 18 deletions components/extractors/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Extractors
title: Data Source Connectors
permalink: /components/extractors/
redirect_from:
- /extractors/
Expand All @@ -9,38 +9,38 @@ redirect_from:
* TOC
{:toc}

Extractors are [Keboola Connection components](/components/) used for **importing data from external sources into Keboola Connection**.
Data source connectors (formerly known as Extractors) are [Keboola components](/components/) used for **importing data from external sources into Keboola**.

## Types of Extractors
Typically, extractors connect to [APIs](https://en.wikipedia.org/wiki/Application_programming_interface#Web_APIs)
external services, like Facebook, Youtube, Gmail and so on.
## Types of Data Source Connectors
Typically, data source connectors connect to [APIs](https://en.wikipedia.org/wiki/Application_programming_interface#Web_APIs)
external services, like Facebook, YouTube, Gmail and so on.
They can also connect directly to an arbitrary database.

Extractors can be grouped by their primary purpose:
They can be grouped by their primary purpose:

- Database extractors: [SQL Databases](/components/extractors/database/sqldb/) and [NoSQL MongoDB](/components/extractors/database/mongodb/) and extractors from generic [Storage services](/components/extractors/storage/)
- Database data source connectors: [SQL Databases](/components/extractors/database/sqldb/) and [NoSQL MongoDB](/components/extractors/database/mongodb/) and data source connectors from generic [Storage services](/components/extractors/storage/)
- [Communication](/components/extractors/communication/),
[Social Networks](/components/extractors/social/) and [Marketing and Sales](/components/extractors/marketing-sales/) extractors
- [Other](/components/extractors/other/) extractors such as Geocoding-Augmentation or Weather Forecast
[Social Networks](/components/extractors/social/) and [Marketing and Sales](/components/extractors/marketing-sales/) data source connectors
- [Other](/components/extractors/other/) data source connectors, such as Geocoding-Augmentation or Weather Forecast

For a definitive list of usable extractors, see your project **Extractors** section or the
For a definitive list of usable data source connectors, see your project **Data Sources** section or the
[public list of components](https://components.keboola.com/components).

## Working with Extractors
Even though extractors are generally designed for [**automated and repeated**](/orchestrator/) data collection,
## Working with Data Source Connectors
Even though data source connectors are generally designed for [**automated and repeated**](/orchestrator/) data collection,
they can be triggered manually at any time.

- For manual import of ad-hoc data, see [Data Import in Storage](/storage/files/), or our [tutorial on manual data loading](/tutorial/load/).
- Configure a [sample extractor](/tutorial/load/googledrive/) (GoogleDrive).
- Configure a [Database extractor](/tutorial/load/database/);
other SQL database extractors are configured in the exact same way.
- Configure a [sample data source connector](/tutorial/load/googledrive/) (GoogleDrive).
- Configure a [database data source connector](/tutorial/load/database/);
other SQL database data source connectors are configured in the exact same way.

As bringing data into Keboola Connection is the main purpose of an extractor, go the path of least resistance:
As bringing data into Keboola is the main purpose of a data source connectors, go the path of least resistance:
**Get your data in first, and then convert it to what you want it to look like.**
To give an example, when connecting to existing information systems, do not modify the data in them.
Such data conversion can prove to be difficult and expensive. Extract what you need and shape it in Keboola Connection.
Such data conversion can prove to be difficult and expensive. Extract what you need and shape it in Keboola.

## Limitations
Some extractors have **limits inherent to their sources**. Unfortunately, there is not much we can do about it.
For example, the Twitter extractor will not let you access the history of a particular tweet account beyond a certain point
For example, the Twitter data source connectors will not let you access the history of a particular tweet account beyond a certain point
because of the [limitations](https://stackoverflow.com/questions/1662151/getting-historical-data-from-twitter) of Twitter API.
33 changes: 33 additions & 0 deletions components/extractors/other/telemetry-data/telemetry-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,8 @@ data in storage, and the number of users. This combines data from different data
* `Snowflake Sandbox` - `kbc_snowflake_stats` (records with **sandbox** *snowflake_job_type*)
* `Transfromations` - `kbc_job` (jobs with **SQL** *transformation_type*)
* `Writers` - `kbc_job` (jobs with **writer** *component_type*)
* `BAPI Messages` - Buffer API (data streams) usage; only aggregated values available
* `BAPI Receiver` - Buffer API endpoints used; only aggregated values available

*Note: `organization_value` and `company_value` are available in **Organization** mode only.
You need data for all projects.*
Expand Down Expand Up @@ -327,6 +329,11 @@ organization, or the Keboola Connection platform itself.
| `kbc_token_id` | Unique identifier of the token containing stack identification | `47949_kbc-us-east-1` |
| `context_admin_email` | Email of the user in the context with the event (e.g., invitation or admin removal) | `john.doe@keboola.com` |
| `context_admin_name` | Name of the user in the context with the event (e.g., invitation or admin removal) | `Martin Matejka` |
| `context_merge_request_id` | ID of the merge request (related to branch merge request events in SOX projects) | `42` |
| `context_merge_request_name` | Name of the merge request (related to branch merge request events in SOX projects) | `Update of my configuration` |
| `context_operation` | Type of the merge request operation (`request_review`, `finish_review`, `approve`, `merge`, `publish`) | `request_review` |
| `context_state_from` | Original state of the merge request operation (`development`, `in_review`, `approved`, `in_merge`) | `in_review` |
| `context_state_to` | End state of the merge request operation (`in_review`, `approved`, `in_merge`, `published`) | `approved` |

#### Security event operations

Expand Down Expand Up @@ -436,6 +443,8 @@ organization, or the Keboola Connection platform itself.
|`auditLog.storageBackendConnection.deleted`
|`auditLog.storageBackendConnection.listed`
|`auditLog.storageBackendConnection.updated`
|`auditLog.mergeRequest.created`
|`auditLog.mergeRequest.stateChanged`

#### Operation parameters

Expand Down Expand Up @@ -570,6 +579,30 @@ This table shows data about flow tasks.
| `task_kbc_component_id` | Unique KBC component identifier | `keboola.wr-db-snowflake_kbc-us-east-1` |
| `task_kbc_component_configuration_id` | Foreign key to the flow component configuration | `7880_kbc-us-east-1_keboola.wr-db-snowflake_952663182` |

### kbc_job_input_table
This table shows data about all input tables of the job.

| **Column** | **Description** | **Example** |
|---|---|---|
| `kbc_job_id` (PK) | KBC flow task identifier | `963416992_kbc-us-east-1` |
| `kbc_project_id` | Foreign key to the KBC project | `7880_kbc-us-east-1` |
| `table_id` | Identifier of the table | `in.c-in_sh_kbc_internal.kbc_project` |
| `kbc_project_table_id` (PK) | Foreign key to the KBC Table | `7880_kbc-us-east-1_in.c-in_sh_kbc_internal.kbc_project` |
| `table_name` | Name of the table | `kbc_project` |
| `mappings` | Number of times the table was used in the job input (i.e., one table can be used multiple times in the input mapping of the transformation) | `1` |

### kbc_job_output_table
This table shows data about all output tables of the job.

| **Column** | **Description** | **Example** |
|---|---|---|
| `kbc_job_id` (PK) | KBC flow task identifier | `909588277_kbc-us-east-1` |
| `kbc_project_id` | Foreign key to the KBC project | `7880_kbc-us-east-1` |
| `table_id` | Identifier of the table | `out.c-kbc_billing.kbc_event` |
| `kbc_project_table_id` (PK) | Foreign key to the KBC Table | `7880_kbc-us-east-1_out.c-kbc_billing.kbc_event` |
| `table_name` | Name of the table | `kbc_event` |
| `mappings` | Number of times the table was used in the job output (i.e., one table can be written multiple times to the storage in the output mapping of the transformation) | `1` |

### kbc_notification_subscription
This table shows data subscriptions to notifications sent by Keboola Connection (mostly flow notifications).

Expand Down
Loading

0 comments on commit 952f624

Please sign in to comment.