Skip to content

Releases: zetaris/lightning-catalog

v0.2

08 Jan 23:38
@jason-zetaris jason-zetaris
v0.2
a7893e9
Compare
Choose a tag to compare
v0.2 Latest
Latest

ver 0.2, December 18, 2024

Introduction:

Welcome to the latest release of our software! Version 0.2 brings more features for data engineers, scientist and developers to run data catalog on top of Apache Spark.

Key Highlights:

Supporting new UI, more data sources, unified semantic layer, and data quality.

Defects fix :

  • can't ingest data sources in azure blob

Improvements :

  • Run metastore over hdfs, blob storage(s3, azure blob)

New Features:

  • New UI: UI allows to navigate data sources, running SQL, render ERD and run Data Quality check.

  • Unstructured Data source(pdf, image, video, text)

  • Unified Semantic Layer

  • Database constraints over any data sources

  • Business Data Quality

  • new data sources:

pdf
image
video
txt
Assets 4
Loading

v0.1

22 Apr 06:23
@jason-zetaris jason-zetaris
v0.1
35b4b3e
Compare
Choose a tag to compare
v0.1

Release Notes: Version 0.1

Date: April 22, 2024

Introduction:

Welcome to the latest release of our software! Version 0.1 brings initial featuers for data engineers, scientist and developers to run data catalog on top of Apache Spark.

Key Highlights:

Introducing data catalog on top of Apache Spark for the preparing data at any scale in ad-hoc analytics, data warehouse, lake house and ML project

New Features:

  • Multi level namespaces: User can create anonymous name space

  • Management of source system endpoint : Lightning catalog manages an endpoint of source system providing unified access to them through SQL and Apache SPARK API, which makes data discovery easy.

  • Query(SQL) capabilities : Lightning catalog allows to run ad-hoc query(SQL) over underlying heterogenous source systems in federate way, E,g Join between multiple source system without moving data.

  • Create virtual db schema : Lightning catalog allows to create virtual DB schema that keeps tables from different data sources

  • Supported data sources:
    DeltaLake
    Iceberg
    H2
    Snowflake
    Posstgres
    Oracle
    Mssql
    Redshift
    Terradata
    MySQL
    DB2
    SQLLite
    MariaDB
    Derby
    HANA
    Greenplum
    Vertica
    Netezza
    Csv
    Parquet
    Orc
    Json
    Avro

Additional Information:

For more information about this release, including detailed documentation and FAQs, please visit distribution web site,
https://www.zetaris.com/lightning-opensource
Document in git hub : https://github.com/zetaris/lightning-catalog/tree/master/doc

Assets 5
Loading