Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 1Password API #17

Merged
merged 6 commits into from
Aug 18, 2024
Merged

Add 1Password API #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
49a82de
add 1password
8naama Aug 18, 2024
8a4a67b
test >> example in 1password readme
8naama Aug 18, 2024
affeb31
clarify `days_back_fetch` in 1password explanation
8naama Aug 18, 2024
46bfa6f
Merge remote-tracking branch 'refs/remotes/origin/feature/1password' …
8naama Aug 18, 2024
799e43e
update function description too
8naama Aug 18, 2024
c529979
changes following review
8naama Aug 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 38 additions & 12 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,19 @@ Configure your API inputs under `apis`. For every API, mention the input type un
For structuring custom API calls use type `general` API with the parameters below.

## Configuration Options
| Parameter Name | Description | Required/Optional | Default |
|--------------------|-----------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in next requests based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| Parameter Name | Description | Required/Optional | Default |
|--------------------|---------------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| next_body | If needed to update the body in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |

## Pagination Configuration Options
If needed, you can configure pagination.
Expand Down Expand Up @@ -207,6 +208,27 @@ By default `cloudflare` API type:
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| pagination_off | True if builtin pagination should be off, False otherwise | Optional | `False` |

</details>
<details>
<summary>
<span><a href="./src/apis/onepassword/README.md">1Password</a></span>
</summary>

By default `1password` API type has built in pagination settings and sets the `response_data_path` to `items` field.

## Configuration Options
| Parameter Name | Description | Required/Optional | Default |
|--------------------------|--------------------------------------------------------------------------------------------------------------|-------------------|-------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| onepassword_bearer_token | The 1Password Bearer token | Required | - |
| url | The request URL | Required | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | - |
| days_back_fetch | The amount of days to fetch back in the first request. Applies a filter on 1password `start_time` parameter. | Optional | - |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| onepassword_limit | 1Password limit for number of events to return in a single request (allowed range: 100 to 1000) | Optional | 100 |
| pagination_off | True if builtin pagination should be off, False otherwise | Optional | `False` |

</details>


Expand Down Expand Up @@ -262,6 +284,10 @@ docker stop -t 30 logzio-api-fetcher
```

## Changelog:
- **0.2.1**:
- Add 1Password Support
- Add `next_body` support to allow more customization in general settings
- Support integers and boolean as values in pagination 'equals' stop condition
- **0.2.0**:
- **Breaking changes!!**
- Deprecate configuration fields:
Expand Down
24 changes: 21 additions & 3 deletions src/apis/general/Api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import logging
from pydantic import BaseModel, Field
import requests
from typing import Union
from typing import Union, Optional

from src.utils.processing_functions import extract_vars, substitute_vars
from src.apis.general.PaginationSettings import PaginationSettings, PaginationType
Expand All @@ -30,31 +30,37 @@ class ApiFetcher(BaseModel):
:param method: Optional, the method to use for the request (default: GET)
:param pagination_settings: Optional, PaginationSettings object that defines how to perform pagination
:param next_url: Optional, If needed update a param in the url according to the response as we go
:param next_body: Optional, If needed update a param in the body according to the response as we go
:param response_data_path: Optional, The path to find the data within the response.
:param additional_fields: Optional, 'key: value' pairs that should be added to the API logs.
:param scrape_interval_minutes: the interval between scraping jobs.
:param url_vars: Not passed to the class, array of params that is generated based on next_url.
:param body_vars: Not passed to the class, array of params that is generated based on next_body.
"""
name: str = Field(default="")
url: str
headers: dict = Field(default={})
body: Union[str, dict, list] = Field(default=None)
method: ReqMethod = Field(default=ReqMethod.GET, frozen=True)
pagination_settings: PaginationSettings = Field(default=None, frozen=True, alias="pagination")
pagination_settings: Optional[PaginationSettings] = Field(default=None, frozen=True, alias="pagination")
next_url: str = Field(default=None)
next_body: Union[str, dict, list] = Field(default=None)
response_data_path: str = Field(default=None, frozen=True)
additional_fields: dict = Field(default={})
scrape_interval_minutes: int = Field(default=1, alias="scrape_interval", ge=1)
url_vars: list = Field(default=[], init=False, init_var=True)
body_vars: list = Field(default=[], init=False, init_var=True)

def __init__(self, **data):
"""
Makes sure to format the body and generate the url_vars based on next_url.
Makes sure to format the body and generate the url_vars based on next_url and body_vars based on next_body.
:param data: the fields for creation of the class.
"""
super().__init__(**data)
self.body = self._format_body(self.body)
self.next_body = self._format_body(self.next_body)
self.url_vars = extract_vars(self.next_url)
self.body_vars = extract_vars(self.next_body)
if not self.name:
self.name = self.url
if not self.additional_fields.get("type"):
Expand Down Expand Up @@ -208,6 +214,14 @@ def update_next_url(self, new_next_url):
self.next_url = new_next_url
self.url_vars = extract_vars(self.next_url)

def update_next_body(self, new_next_body):
"""
Supports updating the next request body format to make sure the 'self.body_vars' is updated accordingly.
:param new_next_body: new format for the next body. (if in future some customized APIs will need it supported)
"""
self.next_body = new_next_body
self.body_vars = extract_vars(self.next_body)

def send_request(self):
"""
Manages the request:
Expand Down Expand Up @@ -236,4 +250,8 @@ def send_request(self):
# Update the url if needed
if self.next_url:
self.url = substitute_vars(self.next_url, self.url_vars, r)

# Update the body if needed
if self.next_body:
self.body = substitute_vars(self.next_body, self.body_vars, r)
return responses
25 changes: 13 additions & 12 deletions src/apis/general/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,19 @@ For structuring custom API calls use type `general` API with the parameters belo
- [Example](#example)

## Configuration
| Parameter Name | Description | Required/Optional | Default |
|--------------------|-----------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in next requests based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| Parameter Name | Description | Required/Optional | Default |
|--------------------|---------------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| next_body | If needed to update the body in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |

## Pagination Configuration Options
If needed, you can configure pagination.
Expand Down
5 changes: 3 additions & 2 deletions src/apis/general/StopPaginationSettings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from enum import Enum
import logging
from pydantic import BaseModel, Field, model_validator
from typing import Union

logger = logging.getLogger(__name__)

Expand All @@ -23,7 +24,7 @@ class StopPaginationSettings(BaseModel):
"""
field: str
condition: StopCondition
value: str = Field(default=None, frozen=True)
value: Union[str, int, bool] = Field(default=None, frozen=True)

@model_validator(mode='after')
def _check_conditional_fields(self):
Expand All @@ -32,7 +33,7 @@ def _check_conditional_fields(self):
if we got condition as 'contains' or 'equals' >> that we also got value
:return: self
"""
if self.condition in (StopCondition.EQUALS, StopCondition.CONTAINS) and not self.value:
if self.condition in (StopCondition.EQUALS, StopCondition.CONTAINS) and self.value is None:
raise ValueError(f"Used stop condition {self.condition} but missing required 'value' field.")
return self

Expand Down
Loading