Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Project] Bump version #446

Merged
merged 1 commit into from
Feb 23, 2024
Merged

[Project] Bump version #446

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lakesoul-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ SPDX-License-Identifier: Apache-2.0
<parent>
<artifactId>lakesoul-parent</artifactId>
<groupId>com.dmetasoul</groupId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>lakesoul-common</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<build>
<plugins>
<plugin>
Expand Down
8 changes: 4 additions & 4 deletions lakesoul-flink/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ SPDX-License-Identifier: Apache-2.0
<parent>
<artifactId>lakesoul-parent</artifactId>
<groupId>com.dmetasoul</groupId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>lakesoul-flink</artifactId>
<version>2.5.0-flink-1.17-SNAPSHOT</version>
<version>flink-1.17-${revision}</version>
<properties>
<flink.version>1.17.1</flink.version>
<scala.binary.version>2.12</scala.binary.version>
Expand All @@ -31,7 +31,7 @@ SPDX-License-Identifier: Apache-2.0
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-common</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand All @@ -42,7 +42,7 @@ SPDX-License-Identifier: Apache-2.0
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-io-java</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand Down
8 changes: 4 additions & 4 deletions lakesoul-presto/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@
<parent>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-parent</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
</parent>

<artifactId>lakesoul-presto</artifactId>
<version>2.5.0-presto-0.28-SNAPSHOT</version>
<version>presto-0.28-${revision}</version>

<properties>
<maven.compiler.source>8</maven.compiler.source>
Expand All @@ -33,12 +33,12 @@
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-common</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
</dependency>
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-io-java</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand Down
8 changes: 4 additions & 4 deletions lakesoul-spark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ SPDX-License-Identifier: Apache-2.0
<parent>
<artifactId>lakesoul-parent</artifactId>
<groupId>com.dmetasoul</groupId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>lakesoul-spark</artifactId>
<version>2.5.0-spark-3.3-SNAPSHOT</version>
<version>spark-3.3-${revision}</version>

<properties>
<scalatest.version>3.2.14</scalatest.version>
Expand All @@ -36,12 +36,12 @@ SPDX-License-Identifier: Apache-2.0
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-common</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
</dependency>
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-io-java</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand Down
11 changes: 2 additions & 9 deletions native-io/lakesoul-io-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ SPDX-License-Identifier: Apache-2.0
<parent>
<artifactId>lakesoul-parent</artifactId>
<groupId>com.dmetasoul</groupId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<modelVersion>4.0.0</modelVersion>
<artifactId>lakesoul-io-java</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<packaging>jar</packaging>

<properties>
Expand Down Expand Up @@ -79,13 +79,6 @@ SPDX-License-Identifier: Apache-2.0
<scope>test</scope>
</dependency>

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
</dependency>

<!-- jnr-ffi deps-->
<dependency>
<groupId>com.github.jnr</groupId>
Expand Down
3 changes: 2 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ SPDX-License-Identifier: Apache-2.0
<modelVersion>4.0.0</modelVersion>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul-parent</artifactId>
<version>2.5.0-SNAPSHOT</version>
<version>${revision}</version>
<inceptionYear>2022</inceptionYear>
<modules>
<module>lakesoul-common</module>
Expand Down Expand Up @@ -51,6 +51,7 @@ SPDX-License-Identifier: Apache-2.0
</developers>

<properties>
<revision>2.6.0-SNAPSHOT</revision>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
Expand Down
11 changes: 6 additions & 5 deletions website/docs/01-Getting Started/01-setup-local-env.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ After unpacking spark package, you could find LakeSoul distribution jar from htt
wget https://dmetasoul-bucket.obs.cn-southwest-2.myhuaweicloud.com/releases/spark/spark-3.3.2-bin-hadoop-3.tgz
tar xf spark-3.3.2-bin-hadoop-3.tgz
export SPARK_HOME=${PWD}/spark-3.3.2-bin-hadoop3
wget https://github.com/lakesoul-io/LakeSoul/releases/download/v2.5.1/lakesoul-spark-2.5.1-spark-3.3.jar -P $SPARK_HOME/jars
wget https://github.com/lakesoul-io/LakeSoul/releases/download/vVAR::VERSION/lakesoul-spark-VAR::VERSION-spark-3.3.jar -P $SPARK_HOME/jars
```

:::tip
Expand Down Expand Up @@ -91,16 +91,17 @@ spark.sql.catalog.lakesoul | org.apache.spark.sql.lakesoul.catalog.LakeSoulCatal
spark.sql.defaultCatalog | lakesoul | set default catalog for spark

### 1.5 Setup Flink environment
Download LakeSoul Flink jars：https://github.com/lakesoul-io/LakeSoul/releases/download/v2.5.1/lakesoul-flink-2.5.1-flink-1.17.jar
Download Flink jars：https://dlcdn.apache.org/flink/flink-1.17.2/flink-1.17.2-bin-scala_2.12.tgz
Download LakeSoul Flink jar: https://github.com/lakesoul-io/LakeSoul/releases/download/vVAR::VERSION/lakesoul-flink-flink-1.17-VAR::VERSION.jar

Download Flink: https://dlcdn.apache.org/flink/flink-1.17.2/flink-1.17.2-bin-scala_2.12.tgz

#### 1.5.1 Start Flink SQL shell
After creating the pg database and `lakesoul_home` configuration file, place the LakeSoul Flink jars in the FLink directory.
Enter the Flink installation directory and execute the following command:
```shell
export lakesoul_home=/opt/soft/pg.property && ./bin/start-cluster.sh

export lakesoul_home=/opt/soft/pg.property && ./bin/sql-client.sh embedded -j lakesoul-flink-2.5.1-flink-1.17.jar
export lakesoul_home=/opt/soft/pg.property && ./bin/sql-client.sh embedded -j lakesoul-flink-flink-1.17-VAR::VERSION.jar
```

#### 1.5.2 Write data to object storage service
Expand Down Expand Up @@ -216,7 +217,7 @@ docker run --net lakesoul-docker-compose-env_default --rm -ti \
-v $(pwd)/lakesoul.properties:/opt/spark/work-dir/lakesoul.properties \
--env lakesoul_home=/opt/spark/work-dir/lakesoul.properties bitnami/spark:3.3.1 \
spark-shell \
--packages com.dmetasoul:lakesoul-spark:2.5.1-spark-3.3 \
--packages com.dmetasoul:lakesoul-spark:spark-3.3-VAR::VERSION \
--conf spark.sql.extensions=com.dmetasoul.lakesoul.sql.LakeSoulSparkSessionExtension \
--conf spark.sql.catalog.lakesoul=org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog \
--conf spark.sql.defaultCatalog=lakesoul \
Expand Down
8 changes: 4 additions & 4 deletions website/docs/01-Getting Started/02-spark-guide.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,22 +35,22 @@ Run spark-shell/spark-sql/pyspark with the `LakeSoulSparkSessionExtension` sql e
<TabItem value="Spark SQL" label="Spark SQL" default>

```bash
spark-sql --conf spark.sql.extensions=com.dmetasoul.lakesoul.sql.LakeSoulSparkSessionExtension --conf spark.sql.catalog.lakesoul=org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog --conf spark.sql.defaultCatalog=lakesoul --jars lakesoul-spark-2.5.1-spark-3.3.jar
spark-sql --conf spark.sql.extensions=com.dmetasoul.lakesoul.sql.LakeSoulSparkSessionExtension --conf spark.sql.catalog.lakesoul=org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog --conf spark.sql.defaultCatalog=lakesoul --jars lakesoul-spark-spark-3.3-VAR::VERSION.jar
```

</TabItem>

<TabItem value="Scala" label="Scala">

```bash
spark-shell --conf spark.sql.extensions=com.dmetasoul.lakesoul.sql.LakeSoulSparkSessionExtension --conf spark.sql.catalog.lakesoul=org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog --conf spark.sql.defaultCatalog=lakesoul --jars lakesoul-spark-2.5.1-spark-3.3.jar
spark-shell --conf spark.sql.extensions=com.dmetasoul.lakesoul.sql.LakeSoulSparkSessionExtension --conf spark.sql.catalog.lakesoul=org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog --conf spark.sql.defaultCatalog=lakesoul --jars lakesoul-spark-spark-3.3-VAR::VERSION.jar
```
</TabItem>
<TabItem value="PySpark" label="PySpark">

```bash
wget https://github.com/lakesoul-io/LakeSoul/tree/main/python/lakesoul/spark/tables.py
pyspark --conf spark.sql.extensions=com.dmetasoul.lakesoul.sql.LakeSoulSparkSessionExtension --conf spark.sql.catalog.lakesoul=org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog --conf spark.sql.defaultCatalog=lakesoul --jars lakesoul-spark-2.5.1-spark-3.3.jar --py-files tables.py
pyspark --conf spark.sql.extensions=com.dmetasoul.lakesoul.sql.LakeSoulSparkSessionExtension --conf spark.sql.catalog.lakesoul=org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog --conf spark.sql.defaultCatalog=lakesoul --jars lakesoul-spark-spark-3.3-VAR::VERSION.jar --py-files tables.py
```
</TabItem>

Expand All @@ -63,7 +63,7 @@ Include maven dependencies in your project:
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul</artifactId>
<version>2.5.1-spark-3.3</version>
<version>spark-3.3-VAR::VERSION</version>
</dependency>
```

Expand Down
4 changes: 2 additions & 2 deletions website/docs/01-Getting Started/03-Flink-Guide.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ export LAKESOUL_PG_PASSWORD=root

## SQL
### Download LakeSoul Flink Jar
It can be downloaded from the LakeSoul Release page: https://github.com/lakesoul-io/LakeSoul/releases/download/v2.5.1/lakesoul-flink-2.5.1-flink-1.17.jar.
It can be downloaded from the LakeSoul Release page: https://github.com/lakesoul-io/LakeSoul/releases/download/vVAR::VERSION/lakesoul-flink-flink-1.17-VAR::VERSION.jar.
### Start SQL Client
```bash
# Start Flink SQL Client
bin/sql-client.sh embedded -j lakesoul-flink-2.5.1-flink-1.17.jar
bin/sql-client.sh embedded -j lakesoul-flink-flink-1.17-VAR::VERSION.jar
```
## Create Table

Expand Down
2 changes: 1 addition & 1 deletion website/docs/02-Tutorials/02-flink-cdc-sink/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Submit a LakeSoul Flink CDC Sink job to the Flink cluster started above:
```bash
./bin/flink run -ys 1 -yjm 1G -ytm 2G \
-c org.apache.flink.lakesoul.entry.MysqlCdc\
lakesoul-flink-2.5.1-flink-1.17.jar \
lakesoul-flink-flink-1.17-VAR::VERSION.jar \
--source_db.host localhost \
--source_db.port 3306 \
--source_db.db_name test_cdc \
Expand Down
4 changes: 2 additions & 2 deletions website/docs/02-Tutorials/07-kafka-topics-data-to-lakesoul.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ export lakesoul_home=./pg.properties && ./bin/spark-submit \
--driver-memory 4g \
--executor-memory 4g \
--master local[4] \
./jars/lakesoul-spark-2.5.1-spark-3.3.jar \
./jars/lakesoul-spark-spark-3.3-VAR::VERSION.jar \
localhost:9092 test.* /tmp/kafka/data /tmp/kafka/checkpoint/ kafka earliest false
```

Expand Down Expand Up @@ -157,6 +157,6 @@ export lakesoul_home=./pg.properties && ./bin/spark-submit \
--driver-memory 4g \
--executor-memory 4g \
--master local[4] \
./jars/lakesoul-spark-2.5.1-spark-3.3.jar \
./jars/lakesoul-spark-spark-3.3-VAR::VERSION.jar \
localhost:9092 test.* /tmp/kafka/data /tmp/kafka/checkpoint/ kafka earliest false http://localhost:8081
```
10 changes: 5 additions & 5 deletions website/docs/03-Usage Docs/02-setup-spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@ To use `spark-shell`, `pyspark` or `spark-sql` shells, you should include LakeSo

#### Use Maven Coordinates via --packages
```bash
spark-shell --packages com.dmetasoul:lakesoul-spark:2.5.1-spark-3.3
spark-shell --packages com.dmetasoul:lakesoul-spark:spark-3.3-VAR::VERSION
```

#### Use Local Packages
You can find the LakeSoul packages from our release page: [Releases](https://github.com/lakesoul-io/LakeSoul/releases).
Download the jar file and pass it to `spark-submit`.
```bash
spark-submit --jars "lakesoul-spark-2.5.1-spark-3.3.jar"
spark-submit --jars "lakesoul-spark-spark-3.3-VAR::VERSION.jar"
```

Or you could directly put the jar into `$SPARK_HOME/jars`
Expand All @@ -34,7 +34,7 @@ Include maven dependencies in your project:
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul</artifactId>
<version>2.5.1-spark-3.3</version>
<version>spark-3.3-VAR::VERSION</version>
</dependency>
```

Expand Down Expand Up @@ -144,7 +144,7 @@ taskmanager.memory.task.off-heap.size: 3000m
:::

### Add LakeSoul Jar to Flink's directory
Download LakeSoul Flink Jar from: https://github.com/lakesoul-io/LakeSoul/releases/download/v2.5.1/lakesoul-flink-2.5.1-flink-1.17.jar
Download LakeSoul Flink Jar from: https://github.com/lakesoul-io/LakeSoul/releases/download/vVAR::VERSION/lakesoul-flink-flink-1.17-VAR::VERSION.jar

And put the jar file under `$FLINK_HOME/lib`. After this, you could start flink session cluster or application as usual.

Expand All @@ -155,6 +155,6 @@ Add the following to your project's pom.xml
<dependency>
<groupId>com.dmetasoul</groupId>
<artifactId>lakesoul</artifactId>
<version>2.5.1-flink-1.17</version>
<version>flink-1.17-VAR::VERSION</version>
</dependency>
```
Loading
Loading