Merge pull request #44 from howardwu-db/feature/issue_41

Added gifs and changed some readme text

README.md

@@ -2,12 +2,15 @@
 
 <!-- Top bar will be removed from PyPi packaged versions -->
 <!-- Dont remove: exclude package -->
+
 [Documentation](https://databrickslabs.github.io/dlt-meta/) |
 [Release Notes](CHANGELOG.md) |
-[Examples](https://github.com/databrickslabs/dlt-meta/tree/main/examples) 
+[Examples](https://github.com/databrickslabs/dlt-meta/tree/main/examples)
+
 <!-- Dont remove: end exclude package -->
 
 ---
+
 <p align="left">
     <a href="https://databrickslabs.github.io/dlt-meta/">
         <img src="https://img.shields.io/badge/DOCS-PASSING-green?style=for-the-badge" alt="Documentation Status"/>
@@ -39,38 +42,46 @@
     </a>
 </p>
 
-[![lines of code](https://tokei.rs/b1/github/databrickslabs/dlt-meta)]([https://codecov.io/github/databrickslabs/dlt-meta](https://github.com/databrickslabs/dlt-meta))
+[![lines of code](https://tokei.rs/b1/github/databrickslabs/dlt-meta)](<[https://codecov.io/github/databrickslabs/dlt-meta](https://github.com/databrickslabs/dlt-meta)>)
 
 ---
 
 # Project Overview
-```DLT-META``` is a metadata-driven framework based on Databricks [Delta Live Tables](https://www.databricks.com/product/delta-live-tables) (aka DLT) which lets you automate your bronze and silver data pipelines.
 
-With this framework you need to record the source and target metadata in an onboarding json file which acts as the data flow specification aka Dataflowspec. A single generic ```DLT``` pipeline takes the ```Dataflowspec``` and runs your workloads.
+`DLT-META` is a metadata-driven framework based on Databricks [Delta Live Tables](https://www.databricks.com/product/delta-live-tables) (aka DLT) which lets you automate your bronze and silver data pipelines.
+
+With this framework you need to record the source and target metadata in an onboarding json file which acts as the data flow specification aka Dataflowspec. A single generic `DLT` pipeline takes the `Dataflowspec` and runs your workloads.
 
 ### Components:
 
-#### Metadata Interface 
+#### Metadata Interface
+
 - Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.json)
 - Capture [Data Quality Rules](https://github.com/databrickslabs/dlt-meta/tree/main/examples/dqe/customers/bronze_data_quality_expectations.json)
-- Capture  processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json)
+- Capture processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json)
 
 #### Generic DLT pipeline
+
 - Apply appropriate readers based on input metadata
-- Apply data quality rules with DLT expectations 
+- Apply data quality rules with DLT expectations
 - Apply CDC apply changes if specified in metadata
 - Builds DLT graph based on input/output metadata
 - Launch DLT pipeline
 
 ## High-Level Process Flow:
+
 ![DLT-META High-Level Process Flow](./docs/static/images/solutions_overview.png)
 
 ## Steps
+
 ![DLT-META Stages](./docs/static/images/dlt-meta_stages.png)
 
 ## Getting Started
+
 Refer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting_started)
+
 ### Databricks Labs DLT-META CLI lets you run onboard and deploy in interactive python terminal
+
 #### pre-requisites:
 
 - Python 3.8.0 +
@@ -80,7 +91,6 @@ Refer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting
 - Install Databricks CLI on macOS:
 - ![macos_install_databricks](docs/static/images/macos_1_databrickslabsmac_installdatabricks.gif)
 
-
 - Install Databricks CLI on Windows:
 - ![windows_install_databricks.png](docs/static/images/windows_install_databricks.png)
 
@@ -93,28 +103,33 @@ databricks auth login --host WORKSPACE_HOST
     To enable debug logs, simply add `--debug` flag to any command.
 
 ### Installing dlt-meta:
+
 - Install dlt-meta via Databricks CLI:
 
 ```commandline
     databricks labs install dlt-meta
 ```
-### Onboard using dlt-meta CLI:
 
+### Onboard using dlt-meta CLI:
 
 If you want to run existing demo files please follow these steps before running onboard command:
 
 ```commandline
-    git clone dlt-meta
+    git clone https://github.com/databrickslabs/dlt-meta.git
 ```
+
 ```commandline
     cd dlt-meta
 ```
+
 ```commandline
     python -m venv .venv
 ```
+
 ```commandline
-    source .venv/bin/activate 
+    source .venv/bin/activate
 ```
+
 ```commandline
     pip install databricks-sdk
 ```
@@ -123,105 +138,45 @@ If you want to run existing demo files please follow these steps before running
     databricks labs dlt-meta onboard
 ```
 
-Above command will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder.
-
-```     Provide onboarding file path (default: demo/conf/onboarding.template): 
-        Provide onboarding files local directory (default: demo/): 
-        Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): 
-        Provide databricks runtime version (default: 14.2.x-scala2.12): 
-        Run onboarding with unity catalog enabled?
-        [0] False
-        [1] True
-        Enter a number between 0 and 1: 1
-        Provide unity catalog name: ravi_dlt_meta_uc
-        Provide dlt meta schema name (default: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead): 
-        Provide dlt meta bronze layer schema name (default: dltmeta_bronze_cf5956873137432294892fbb2dc34fdb): 
-        Provide dlt meta silver layer schema name (default: dltmeta_silver_5afa2184543342f98f87b30d92b8c76f): 
-        Provide dlt meta layer
-        [0] bronze
-        [1] bronze_silver
-        [2] silver
-        Enter a number between 0 and 2: 1
-        Provide bronze dataflow spec table name (default: bronze_dataflowspec): 
-        Provide silver dataflow spec table name (default: silver_dataflowspec): 
-        Overwrite dataflow spec?
-        [0] False
-        [1] True
-        Enter a number between 0 and 1: 1
-        Provide dataflow spec version (default: v1): 
-        Provide environment name (default: prod): prod
-        Provide import author name (default: ravi.gawai): 
-        Provide cloud provider name
-        [0] aws
-        [1] azure
-        [2] gcp
-        Enter a number between 0 and 2: 0
-        Do you want to update ws paths, catalog, schema details to your onboarding file?
-        [0] False
-        [1] True
-```
+![onboardingDLTMeta.gif](docs/content/getting_started/onboardingDLTMeta.gif)
+
+Above commands will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder.
+![onboardingDLTMeta_2.gif](docs/content/getting_started/onboardingDLTMeta_2.gif)
+
+
 - Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs
+
 ### depoly using dlt-meta CLI:
+
 - Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command
-- ```commandline 
+- ```commandline
      databricks labs dlt-meta deploy
-   ```
+  ```
 - - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps
 - - Bronze DLT
-```
-    Deploy DLT-META with unity catalog enabled?
-    [0] False
-    [1] True
-    Enter a number between 0 and 1: 1
-    Provide unity catalog name: ravi_dlt_meta_uc
-    Deploy DLT-META with serverless?
-    [0] False
-    [1] True
-    Enter a number between 0 and 1: 1
-    Provide dlt meta layer
-    [0] bronze
-    [1] silver
-    Enter a number between 0 and 1: 0
-    Provide dlt meta onboard group: A1  
-    Provide dlt_meta dataflowspec schema name: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead
-    Provide bronze dataflowspec table name (default: bronze_dataflowspec): 
-    Provide dlt meta pipeline name (default: dlt_meta_bronze_pipeline_2aee3eb837f3439899eef61b76b80d53): 
-    Provide dlt target schema name: dltmeta_bronze_cf5956873137432294892fbb2dc34fdb
-```
+
+![deployingDLTMeta_bronze.gif](docs/content/getting_started/deployingDLTMeta_bronze.gif)
+
 
 - Silver DLT
 - - ```commandline
        databricks labs dlt-meta deploy
     ```
 - - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps
-```
-    Deploy DLT-META with unity catalog enabled?
-    [0] False
-    [1] True
-    Enter a number between 0 and 1: 1
-    Provide unity catalog name: ravi_dlt_meta_uc
-    Deploy DLT-META with serverless?
-    [0] False
-    [1] True
-    Enter a number between 0 and 1: 1
-    Provide dlt meta layer
-    [0] bronze
-    [1] silver
-    Enter a number between 0 and 1: 1
-    Provide dlt meta onboard group: A1
-    Provide dlt_meta dataflowspec schema name: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead
-    Provide silver dataflowspec table name (default: silver_dataflowspec): 
-    Provide dlt meta pipeline name (default: dlt_meta_silver_pipeline_2147545f9b6b4a8a834f62e873fa1364): 
-    Provide dlt target schema name: dltmeta_silver_5afa2184543342f98f87b30d92b8c76f
-```
+
+![deployingDLTMeta_silver.gif](docs/content/getting_started/deployingDLTMeta_silver.gif)
+
+
 ## More questions
+
 Refer to the [FAQ](https://databrickslabs.github.io/dlt-meta/faq)
 and DLT-META [documentation](https://databrickslabs.github.io/dlt-meta/)
 
 # Project Support
+
 Please note that all projects released under [`Databricks Labs`](https://www.databricks.com/learn/labs)
- are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements 
-(SLAs).  They are provided AS-IS and we do not make any guarantees of any kind.  Please do not submit a support ticket 
+are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements
+(SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket
 relating to any issues arising from the use of these projects.
 
 Any issues discovered through the use of this project should be filed as issues on the Github Repo.