Skip to content

Commit 8965862

Browse files
M-BuskasennesanneufeldcapAduriltsenkwong
authored
Develop -> main (#18)
* adjust README and configs * initial commit, asset creation working * add db scripts * added postgres service to piepline * echo env variables * added variables for daps * set postgres envs * removed echo check * add example of identity check policy (#14) * validation Endpoint is only one variable now * adapted entries in config file * added vault config * remove fh-extension * fixed edc_vault path default * provided default * added deb stateful set * add some dependencies * provided db access stuff * update configuration to match local setup and edc version 0.4.1 * update comment * change postres credentials * updated config * added portal db to pg admin * Update Dockerfile for EDC Deployment --------- Co-authored-by: asennes <alexandra.sennes@capgemini.com> Co-authored-by: Andreas Neufeld <andreas.neufeld@capgemini.com> Co-authored-by: Peter Melinat <peter.melinat@altow.de> Co-authored-by: tsenkwong <67427208+tsenkwong@users.noreply.github.com> Co-authored-by: anneufeldcap <144432245+anneufeldcap@users.noreply.github.com>
1 parent 770e340 commit 8965862

28 files changed

+799
-519
lines changed

‎.github/workflows/continious_integration.yml

+13-2
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,28 @@ env:
77
REGISTRY: ghcr.io
88
REGISTRY_NAMESPACE: possible-x
99
K8S_NAMESPACE: ${{ github.ref == 'refs/heads/main' && 'mvd-001-demo' || 'edc-dev' }}
10+
POSTGRES_USER: postgres
11+
POSTGRES_PASSWORD: postgres
12+
POSTGRES_DB: postgres
1013
jobs:
1114
build-and-push-image:
1215
runs-on: ubuntu-latest
1316
permissions:
1417
contents: read
1518
packages: write
16-
19+
services:
20+
postgres:
21+
image: postgres:16
22+
env:
23+
POSTGRES_USER: ${{ env.POSTGRES_USER }}
24+
POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}
25+
POSTGRES_DB: ${{ env.POSTGRES_DB }}
26+
ports:
27+
- 5432:5432
28+
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
1729
steps:
1830
- name: Checkout repository
1931
uses: actions/checkout@v3
20-
2132
- name: Set up JDK 17
2233
uses: actions/setup-java@v4
2334
with:

‎README.md

+31-17
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,7 @@ This document explains how to run an EDC with the `POSSIBLE-X` and the `IONOS S3
66

77
- java 17
88
- gradle
9-
10-
## Examples
11-
For experimenting with the running EDC, some Postman/Insomnia collections were added to this repo at `postman/`:
12-
13-
- `POSSIBLE-X-IONOS-S3.postman_collection.json` is a Postman collection with examples for performing a transfer from and to an IONOS S3 Bucket
14-
- `POSSIBLE-X.Insomnia_IONOS-S3.json` is an Insomnia collection with examples for performing a transfer from and to an IONOS S3 Bucket
15-
- `POSSIBLE-X.postman_collection.json` is a Postman collection with examples for performing a transfer from and to an HTTP API
16-
- `Test.postman_environment.json` is a Postman environment file which sets up the collections to interact with the POSSIBLE EDCs running in the IONOS cloud
9+
- Postman/Insomnia (_optional_)
1710

1811
## Steps
1912
### Checkout the repo
@@ -23,47 +16,68 @@ For experimenting with the running EDC, some Postman/Insomnia collections were a
2316
git clone https://github.com/POSSIBLE-X/possible-x-edc-extension.git
2417
```
2518

19+
### Create git token
20+
- open the github settings for your personal access tokens: https://github.com/settings/tokens
21+
- generate a new token. select only the scope `read:packages` for it.
22+
2623
### Compiling
2724

28-
- export your `GitHub` authentication data
25+
- export your `GitHub` authentication data. use the token created in the previous step.
2926
```
3027
export USERNAME_GITHUB=<YOUR USERNAME> or <YOUR TOKEN NAME>
3128
export TOKEN_GITHUB=<YOUR TOKEN>
3229
```
33-
- go to your the main folder and execute the following:
30+
- go to the main folder and execute the following:
3431
```
3532
./gradlew build
3633
```
3734

38-
### Edit config file
35+
### Edit config files
3936

40-
- Open the `connector/resources/config.properties` file and edit the following fields:
37+
- Open the `connector/resources/config.properties` file and edit the following fields. Take the values from the keepass DB in the `possible-x-infra` repo and insert them.
4138

4239
| Field name | Description |
4340
|---------------------------------|------------------------------------------------------------|
44-
| `possible.catalog.jwt.token` | Authorization token to access the Possible-X Catalog |
45-
| `possible.catalog.endpoint` | Endpoint of the Possible-X Catalog for the SD registration |
4641
| `possible.connector.edcVersion` | Version of the Connector |
4742
| `edc.ionos.access.key` | IONOS Access Key Id to access S3 |
4843
| `edc.ionos.secret.key` | IONOS Secret Access Key to access S3 |
4944
| `edc.ionos.endpoint` | IONOS S3 Endpoint |
5045
| `edc.ionos.token` | IONOS token to allow S3 provisioning |
5146

52-
To know more the `IONOS S3 Extension` please check this [site](https://github.com/ionos-cloud/edc-ionos-s3).
47+
- Add these fields to the `provider-configuration.properties` and `consumer-configuration.properties` as well, before starting one dedicated consumer and one dedicated provider instance.
5348

49+
To know more about the `IONOS S3 Extension` please check this [site](https://github.com/ionos-cloud/edc-ionos-s3).
5450

5551
### Running
5652

57-
- Execute the following command:
53+
Either execute the following command, for starting one instance:
5854
```
5955
java -Dedc.fs.config=connector/resources/config.properties -jar connector/build/libs/connector.jar
6056
```
57+
Or execute the following commands, for starting one dedicated consumer and one dedicated provider instance:
58+
```
59+
java -Dedc.fs.config=connector/resources/provider-configuration.properties -jar connector/build/libs/connector.jar
60+
```
61+
```
62+
java -Dedc.fs.config=connector/resources/consumer-configuration.properties -jar connector/build/libs/connector.jar
63+
```
6164

6265
### Interacting
6366

6467
- Please follow the documentation of the `documentation` repository (LINK TBD).
6568

66-
### Continuous Integration
69+
## Examples
70+
For experimenting with the running EDC, some Postman/Insomnia collections were added to this repo at `postman/`:
71+
72+
- `POSSIBLE-X-IONOS-S3.postman_collection.json` is a Postman collection with examples for performing a transfer from and to an IONOS S3 Bucket
73+
- `POSSIBLE-X.Insomnia_IONOS-S3.json` is an Insomnia collection with examples for performing a transfer from and to an IONOS S3 Bucket
74+
- `POSSIBLE-X.postman_collection.json` is a Postman collection with examples for performing a transfer from and to an HTTP API
75+
- `Test.postman_environment.json` is a Postman environment file which sets up the collections to interact with the POSSIBLE EDCs running in the IONOS cloud
76+
77+
Import the collection into Postman/Insomnia. In the collection's settings, fill the Environment Variables with the path and port of your running EDC.
78+
Check the corresponding `[...]-configuration.properties` for the correct values.
79+
80+
## Continuous Integration
6781
A Github Action Pipeline (Build and Deploy EDC) was implemented to build and deploy the Artifcats to the DEV environment
6882

6983
The Pipeline Builds a docker container and deploys it to the IONOS Cloud PossibleX Kubernetes Cluster

‎connector/build.gradle.kts

+26-26
Original file line numberDiff line numberDiff line change
@@ -43,49 +43,49 @@ val edcVersion: String by project
4343

4444

4545
dependencies {
46-
implementation("${edcGroup}:boot:${edcVersion}")
47-
implementation("${edcGroup}:connector-core:${edcVersion}")
4846

47+
implementation("${edcGroup}:boot:${edcVersion}")
48+
implementation("${edcGroup}:control-plane-core:${edcVersion}")
49+
implementation("${edcGroup}:control-plane-api:${edcVersion}")
50+
implementation("${edcGroup}:control-plane-api-client:${edcVersion}")
4951
implementation("${edcGroup}:api-observability:${edcVersion}")
5052
implementation("${edcGroup}:configuration-filesystem:${edcVersion}")
5153

52-
implementation("${edcGroup}:iam-mock:${edcVersion}")
53-
implementation("${edcGroup}:http:${edcVersion}")
54-
55-
implementation("${edcGroup}:control-plane-api-client:${edcVersion}")
56-
implementation("${edcGroup}:control-plane-api:${edcVersion}")
57-
implementation("${edcGroup}:control-plane-core:${edcVersion}")
58-
//implementation("${edcGroup}:control-plane-transfer:${edcVersion}")
59-
implementation("${edcGroup}:dsp:${edcVersion}")
6054
implementation("${edcGroup}:auth-tokenbased:${edcVersion}")
61-
implementation("${edcGroup}:management-api:${edcVersion}")
62-
implementation("${edcGroup}:data-plane-http:${edcVersion}")
63-
//implementation(project(":extension"))
55+
implementation("${edcGroup}:management-api:${edcVersion}")
56+
57+
//implementation("${edcGroup}:iam-mock:${edcVersion}")
6458

59+
implementation("${edcGroup}:oauth2-service:${edcVersion}")
60+
implementation("${edcGroup}:oauth2-daps:${edcVersion}")
61+
implementation("${edcGroup}:vault-filesystem:${edcVersion}")
6562

63+
implementation("${edcGroup}:dsp:${edcVersion}")
64+
//file-transfer
6665
implementation("${edcGroup}:data-plane-core:${edcVersion}")
67-
implementation("${edcGroup}:data-plane-api:${edcVersion}")
68-
implementation("${edcGroup}:data-plane-http:${edcVersion}")
6966
implementation("${edcGroup}:data-plane-client:${edcVersion}")
70-
implementation("${edcGroup}:data-plane-selector-core:${edcVersion}")
7167
implementation("${edcGroup}:data-plane-selector-client:${edcVersion}")
72-
implementation("${edcGroup}:data-plane-selector-api:${edcVersion}")
73-
74-
implementation("${edcGroup}:data-plane-client:${edcVersion}")
75-
68+
implementation("${edcGroup}:data-plane-selector-core:${edcVersion}")
7669
implementation("${edcGroup}:transfer-data-plane:${edcVersion}")
77-
implementation("${edcGroup}:transfer-pull-http-dynamic-receiver:${edcVersion}")
78-
implementation("${edcGroup}:validator-data-address-http-data:${edcVersion}")
79-
//implementation("${edcGroup}:json-ld:${edcVersion}")
70+
implementation("${edcGroup}:data-plane-http:${edcVersion}")
71+
implementation("${edcGroup}:http:${edcVersion}")
8072

81-
//implementation("${edcGroup}:vault-hashicorp:${edcVersion}")
82-
//implementation("${edcGroup}:asset-index-sql:${edcVersion}")
73+
implementation(project(":policy-extension"))
8374

8475
//IONOS
8576
implementation ("com.ionoscloud.edc:provision-ionos-s3:v2.2.0")
8677
implementation ("com.ionoscloud.edc:data-plane-ionos-s3:v2.2.0")
8778

88-
//implementation("de.fraunhofer.iais.eis.ids.infomodel:java:${fraunhoferVersion}")
79+
implementation("${edcGroup}:asset-index-sql:${edcVersion}")
80+
implementation("${edcGroup}:contract-definition-store-sql:${edcVersion}")
81+
implementation("${edcGroup}:contract-negotiation-store-sql:${edcVersion}")
82+
implementation("${edcGroup}:policy-definition-store-sql:${edcVersion}")
83+
implementation("${edcGroup}:policy-monitor-store-sql:${edcVersion}")
84+
implementation("${edcGroup}:sql-lease:${edcVersion}")
85+
implementation("${edcGroup}:sql-pool-apache-commons:${edcVersion}")
86+
implementation("${edcGroup}:transaction-local:$edcVersion")
87+
implementation("${edcGroup}:transaction-datasource-spi:$edcVersion")
88+
implementation ("org.postgresql:postgresql:42.7.2")
8989
}
9090

9191
repositories {

‎connector/resources/config.properties

+8-3
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,18 @@ web.http.public.path=/public
1313
web.http.data.port=8186
1414
web.http.data.path=/data
1515
edc.api.auth.key=password
16-
possible.catalog.jwt.token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib3BlcmF0b3IiXX19.lMkYKTViVNVPFH49ntdkruLe5EaWRUYt1YL-1Y7b0gc
17-
possible.catalog.endpoint=https://possible.fokus.fraunhofer.de/api/hub/repo/catalogues/test-provider/datasets/origin
1816
possible.connector.edcVersion=0.4.1
1917

2018
edc.ionos.endpoint =https://s3-eu-central-2.ionoscloud.com
2119
ids.webhook.address=http://localhost:8282
2220
edc.vault.hashicorp.url=http://localhost:8200
2321
edc.vault.hashicorp.token=test-token
2422
edc.vault.hashicorp.timeout.seconds=30
25-
edc.dataplane.token.validation.endpoint=http://localhost:8184/control/token
23+
edc.dataplane.token.validation.endpoint=http://localhost:8184/control/token
24+
25+
edc.datasource.default.url=jdbc:postgresql://consumer-edc-possible-x-edc-postgres.edc-dev
26+
edc.datasource.default.name=admin
27+
edc.datasource.default.password=ftGbHW.Qd4rYaiHVzEsj
28+
29+
# this will be available in version >=v0.9.0 and allows to skip manual sql initialization
30+
#edc.sql.schema.autocreate=true

‎connector/resources/consumer-configuration.properties

+25-11
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
edc.participant.id=consumer
2-
edc.dsp.callback.address=http://localhost:29194/protocol
31
web.http.port=29191
42
web.http.path=/api
53
web.http.management.port=29193
@@ -11,16 +9,32 @@ web.http.public.path=/public
119
web.http.control.port=29192
1210
web.http.control.path=/control
1311

14-
web.http.data.port=9196
15-
web.http.data.path=/data
12+
edc.dsp.callback.address=http://localhost:29194/protocol
13+
edc.participant.id=20:1D:9C:04:0A:71:B9:E7:8C:28:9D:70:A6:84:43:59:2D:BA:E8:B3:keyid:20:1D:9C:04:0A:71:B9:E7:8C:28:9D:70:A6:84:43:59:2D:BA:E8:B3
1614
edc.api.auth.key=password
17-
possible.catalog.jwt.token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib3BlcmF0b3IiXX19.lMkYKTViVNVPFH49ntdkruLe5EaWRUYt1YL-1Y7b0gc
18-
possible.catalog.endpoint=https://possible.fokus.fraunhofer.de/api/hub/repo/catalogues/test-provider/datasets/origin
19-
possible.connector.edcVersion=0.6.0
15+
edc.vault=/app/vault.properties
16+
edc.dataplane.token.validation.endpoint=http://localhost:4567/token
2017

2118

2219
edc.ionos.endpoint =https://s3-eu-central-2.ionoscloud.com
23-
ids.webhook.address=http://localhost:8282
24-
edc.vault.hashicorp.url=http://localhost:8200
25-
edc.vault.hashicorp.token=test-token
26-
edc.vault.hashicorp.timeout.seconds=30
20+
21+
#edc.mock.client.id="1234"
22+
23+
#edc.mock.region=us
24+
25+
edc.oauth.token.url=http://localhost:4567/token
26+
edc.oauth.certificate.alias=1
27+
edc.oauth.private.key.alias=1
28+
edc.oauth.client.id=20:1D:9C:04:0A:71:B9:E7:8C:28:9D:70:A6:84:43:59:2D:BA:E8:B3:keyid:20:1D:9C:04:0A:71:B9:E7:8C:28:9D:70:A6:84:43:59:2D:BA:E8:B3
29+
edc.oauth.provider.jwks.url=http://localhost:4567/jwks.json
30+
edc.oauth.provider.audience=idsc:IDS_CONNECTORS_ALL
31+
edc.iam.token.scope=idsc:IDS_CONNECTOR_ATTRIBUTES_ALL
32+
edc.keystore=/home/possible/workspace/localdeployment/initial_data/edc/consumer.pfx
33+
edc.keystore.password=zw9Eie7ztAqpl0Rbd/GEatmvzEREXEzL
34+
35+
edc.datasource.default.url=jdbc:postgresql://localhost:5432/edcconsumer
36+
edc.datasource.default.user=postgres
37+
edc.datasource.default.password=postgres
38+
39+
# this will be available in version >=v0.9.0 and allows to skip manual sql initialization
40+
#edc.sql.schema.autocreate=true

‎connector/resources/provider-configuration.properties

+25-12
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
edc.participant.id=provider
2-
edc.dsp.callback.address=http://localhost:19194/protocol
31
web.http.port=19191
42
web.http.path=/api
53
web.http.management.port=19193
@@ -10,17 +8,32 @@ web.http.public.port=19291
108
web.http.public.path=/public
119
web.http.control.port=19192
1210
web.http.control.path=/control
13-
edc.dataplane.api.public.baseurl=http://localhost:19291/public
1411

15-
web.http.data.port=8186
16-
web.http.data.path=/data
12+
edc.dsp.callback.address=http://localhost:19194/protocol
13+
edc.participant.id=59:0B:DD:26:41:AC:57:D7:ED:76:D5:84:F8:BC:AC:8E:4C:C7:56:70:keyid:59:0B:DD:26:41:AC:57:D7:ED:76:D5:84:F8:BC:AC:8E:4C:C7:56:70
1714
edc.api.auth.key=password
18-
possible.catalog.jwt.token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib3BlcmF0b3IiXX19.lMkYKTViVNVPFH49ntdkruLe5EaWRUYt1YL-1Y7b0gc
19-
possible.catalog.endpoint=https://possible.fokus.fraunhofer.de/api/hub/repo/catalogues/test-provider/datasets/origin
20-
possible.connector.edcVersion=0.6.0
15+
edc.vault=/app/vault.properties
16+
edc.dataplane.token.validation.endpoint=http://localhost:4567/token
2117

2218
edc.ionos.endpoint =https://s3-eu-central-2.ionoscloud.com
23-
ids.webhook.address=http://localhost:8282
24-
edc.vault.hashicorp.url=http://localhost:8200
25-
edc.vault.hashicorp.token=test-token
26-
edc.vault.hashicorp.timeout.seconds=30
19+
20+
#edc.mock.client.id="5678"
21+
22+
#edc.mock.region=eu
23+
24+
edc.oauth.token.url=http://localhost:4567/token
25+
edc.oauth.certificate.alias=1
26+
edc.oauth.private.key.alias=1
27+
edc.oauth.client.id=59:0B:DD:26:41:AC:57:D7:ED:76:D5:84:F8:BC:AC:8E:4C:C7:56:70:keyid:59:0B:DD:26:41:AC:57:D7:ED:76:D5:84:F8:BC:AC:8E:4C:C7:56:70
28+
edc.oauth.provider.jwks.url=http://localhost:4567/jwks.json
29+
edc.oauth.provider.audience=idsc:IDS_CONNECTORS_ALL
30+
edc.iam.token.scope=idsc:IDS_CONNECTOR_ATTRIBUTES_ALL
31+
edc.keystore=/home/possible/workspace/localdeployment/initial_data/edc/provider.pfx
32+
edc.keystore.password=F4HKbkgAORZWwzWasY10RhsxHt99LaoE
33+
34+
edc.datasource.default.url=jdbc:postgresql://localhost:5432/edcprovider
35+
edc.datasource.default.user=postgres
36+
edc.datasource.default.password=postgres
37+
38+
# this will be available in version >=v0.9.0 and allows to skip manual sql initialization
39+
#edc.sql.schema.autocreate=true

‎deployment/helm/possible-x-edc/templates/configmap.yaml

+15-1
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,18 @@ data:
2929
edc.dsp.callback.address={{ .Values.edc.dsp.callback.address }}
3030
edc.receiver.http.endpoint={{ .Values.edc.receiver.http.endpoint }}/receiver/{{ .Values.edc.ids.id }}/callback
3131
edc.public.key.alias={{ .Values.edc.public.key.alias }}
32-
edc.dataplane.token.validation.endpoint={{ .Values.edc.dataplane.token.validation.endpoint }}{{ .Values.web.http.control.path }}/token
32+
edc.dataplane.token.validation.endpoint={{ .Values.edc.dataplane.token.validation.endpoint }}
33+
edc.oauth.token.url={{ .Values.edc.oauth.token.url }}
34+
edc.oauth.certificate.alias={{ .Values.edc.oauth.certificate.alias }}
35+
edc.oauth.private.key.alias={{ .Values.edc.oauth.private.key.alias }}
36+
edc.oauth.client.id={{ .Values.edc.oauth.client.id }}
37+
edc.oauth.provider.jwks.url={{ .Values.edc.oauth.provider.jwks.url }}
38+
edc.oauth.provider.audience={{ .Values.edc.oauth.provider.audience }}
39+
edc.iam.token.scope={{ .Values.edc.iam.token.scope }}
40+
edc.keystore={{ .Values.edc.keystore.path }}
41+
edc.keystore.password={{ .Values.edc.keystore.password }}
42+
edc.vault={{ .Values.edc.vault.path }}
43+
edc.datasource.default.url=jdbc:postgresql://{{ .Values.edc.datasource.url }}
44+
edc.datasource.default.name={{ .Values.edc.datasource.name }}
45+
edc.datasource.default.user={{ .Values.edc.datasource.name }}
46+
edc.datasource.default.password={{ .Values.edc.datasource.password }}

‎deployment/helm/possible-x-edc/templates/db_service.yaml

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2+
{{- if .Values.persistence.db.enabled -}}
3+
apiVersion: v1
4+
kind: Service
5+
metadata:
6+
name: {{ include "possible-x-edc.fullname" . }}-postgres
7+
spec:
8+
ports:
9+
- port: 5432
10+
name: postgres
11+
selector:
12+
app: {{ include "possible-x-edc.fullname" . }}-postgres
13+
{{- end }}

0 commit comments

Comments
 (0)