From 18bd174bf5d37ab31eb69551d36f5adffaaf61e5 Mon Sep 17 00:00:00 2001 From: Sascha Dobschal Date: Thu, 18 Aug 2022 07:25:23 +0200 Subject: [PATCH 1/4] Add IPFS Utilities application --- applications/ipfs_utilities.md | 176 +++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 applications/ipfs_utilities.md diff --git a/applications/ipfs_utilities.md b/applications/ipfs_utilities.md new file mode 100644 index 00000000000..a261e18c31c --- /dev/null +++ b/applications/ipfs_utilities.md @@ -0,0 +1,176 @@ +# W3F Grant Proposal + +- **Project Name:** Substrate IPFS Utilities +- **Team Name:** [TDSoftware](https://www.tdsoftware.de/) +- **Payment Address:** 0x8Db0972d9F40357526B879A002d60CCf6B4a8882 +- **Level:** 3 + + + +## Project Overview + +This application is about the utilities of Substrate within IPFS. + + +### Overview + +IPFS is a web3 solution to store and provide big-sized and not structured data. The datas address (index) is based on it's content, which guarantees that the content or file isn't changed or manipulated in any way. Currently there is no satisfying way of an integration for IPFS on Substrate blockchains. + +Currently, the easiest way to record IPFS CIDs on a Substrate-based blockchain, is to call an extrinsic and pass the IPFS CID as an input parameter to store it on chain. The CID is often related to some non-fungible token (NFT), and becomes a metadata of the token. + +In the solution, it will need much more affairs to the validation process on the chain side, if the developer wants to make sure and trust the input CID will point to the corresponding file on the chain. Besides, the CID is listed via gRPC as a string field, it can not be presented event the browser already supports IPFS. + + +### Project Details + +#### Upload utilities +In the first part, detailed research is needed to upload data to IPFS, including +- Accepting bytes as an input parameter of extrinsic +- Uploading `&[u8]` or `[u8]` to IPFS, may or may not in off-chain worker +- Storing CID information on chain, also the CID version 0 and CID version 1 could be handled +- Benchmarking on the extrinsic call to know the maximum size of bytes can work + +#### Read/access utilities +In the second part, the redirect feature will be provided based on the gRPC mechanism, the details are listed +- Accept a function with `Result` return, and take the content of String as the location to redirect +- Developer can define the redirect location as CIDv0 or CIDv1. +- Base on Post/Redirect/Get ([PRG](https://en.wikipedia.org/wiki/Post/Redirect/Get)) pattern +- The return code is HTTP 303 See Other +- E.g. a user can `#[prg(name = "nft_getFileById")]` + + +### Ecosystem Fit + +The implementation helps people using IPFS with substrate, this is a common scenario. +People fork substrate and try to solve this in [offchain_ipfs](https://github.com/rs-ipfs/substrate/tree/offchain_ipfs), +but it is based on a two years old substrate version, and not actively updated. + + +#### Technology Stack + +Blockchain +- Substrate +- Rust +- IPFS + +## Team :busts_in_silhouette: + +### Team members + +- Sascha Dobschal +- Antonio Yang +- Lasse Linnamäki +- Markus Fitzner + +The team setup might change depending on the availability at TDSoftware. With 40+ employees, TDSoftware has various developers that might contribute to the project. In all probability the members listed above will attend. + +### Contact + +- **Contact Name:** Sascha Dobschal +- **Contact Email:** s.dobschal@tdsoftware.de + + +### Legal Structure + +- **Registered Address:** Schlossgasse 20, 07743 Jena, Germany +- **Registered Legal Entity:** TDSoftware GmbH + + +### Team's experience +We have years of experience bringing ideas to life with a user-centered focus using blockchain and mobile technology. We have worked closely with many customers to implement their solutions and have already gained experience with blockchain technologies. +Our most relevant projects are, among others: +- SubIdentity (W3F Grant), Identity Directory for Substrate Chains +- Exchange for trading digital assets (Customer is a Top 200 Token of CMC) +- Token Swap WebApp (Customer is a Top 200 Token of CMC) +- [Blockchain, NFT Pallets & App](https://artists.niftymarket.com/) (In Development) + +We look forward to contributing to the web3 ecosystem with an open-source project next. + + +### Team Code Repos + +Source code will be in: +- https://github.com/TDSoftware + +Team profiles: +- https://github.com/dobschal +- https://github.com/yanganto +- https://github.com/markusdent + + +### Team LinkedIn Profiles + +- https://www.linkedin.com/company/tdsoftware-gmbh/mycompany/ +- https://www.linkedin.com/in/dobschal/ + + + +## Development Status :open_book: + +A concept and solution draft was created as part of this application and can be found in the Project Details chapter. + + + +## Development Roadmap :nut_and_bolt: + +### Overview + +- **Total Estimated Duration:** 4 Months +- **Full-Time Equivalent (FTE):** 2 +- **Total Costs:** 178,880 DAI + +### Milestone 1 - Research and PoC implementation on upload part + +- **Estimated duration:** 1 Month +- **FTE:** 2 +- **Costs:** 44,720 DAI + +After the first research, the PoC pallet implementation will be developed. The goal of milestone 1, is a fully working round trip (except the redirect feature). A PoC client, reading a file, calling an extrinsic and uploading the content of a file,, will be developed. The CID information from IPFS will be stored in the pallet, then the uploaded file is verified by other IPFS tools. + + Number | Deliverable | Specification | +| -----: | ----------- | ------------- | +| 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | +| 0b. | Documentation | We will provide both **inline documentation** of the code and a basic **tutorial** that explains how a developer can re-use the implementation. | +| 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | +| 1. | Implement IPFS upload extrinsic | The pallet implementation will be added, that allow bytes to uploads. | +| 2. | Implement POC client | The client can read data from file, call extrinsic and upload the content in bytes. | +| 3. | Verification | Verify the file uploaded | + +### Milestone 2 - PoC implementation on redirect part + +- **Estimated Duration:** 1 Month +- **FTE:** 2 +- **Costs:** 44,720 + +The goal of the second milestone is to redirect utilities of Substrate. A gRPC request can be based on a PRG mechanism to redirect to any file server, which is not limited by IPFS. + + Number | Deliverable | Specification | +| -----: | ----------- | ------------- | +| 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | +| 0b. | Documentation | We will provide both **inline documentation** of the code and a basic **tutorial** that explains how a developer can re-use the implementation. | +| 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | +| 1. | Implement | `#[prg(name = "nft_getFileById")]` will be designed and developed that allows users to redirect a request, which returns `Result`, to a file server. | + +### Milestone 3 - Integration and Benchmark + +- **Estimated Duration:** 2 Months +- **FTE:** 2 +- **Costs:** 89,440 DAI + +The goal of the third milestone is to have several features and APIs that cover potential use cases. + + Number | Deliverable | Specification | +| -----: | ----------- | ------------- | +| 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | +| 0b. | Documentation | We will provide both **inline documentation** of the code and a basic **tutorial** that explains how a developer can re-use the implementation. | +| 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | +| 1. | Configure | The IPFS pallet implementation will be added with IPFS gateway in configure. | +| 2. | Query | The IPFS pallet can be queried and return a different version of CID as String. | +| 3. | Example | The example pallet can use IPFS pallet to store some data in HTML format in IPFS and provide gRPC to redirect requests to get the HTML content. | +| 4. | Benchmark | Benchmark on the size of the file the IPFS pallet can be accepted. | + + + +## Additional Information :heavy_plus_sign: + +This is our second application for an open-source project to innovate the web3 Ecosystem. From 6dd064448ee4544aea09417c9388bcd3636ba00d Mon Sep 17 00:00:00 2001 From: Sascha Dobschal Date: Sun, 28 Aug 2022 18:26:57 +0200 Subject: [PATCH 2/4] remove Milestone 3 --- applications/ipfs_utilities.md | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/applications/ipfs_utilities.md b/applications/ipfs_utilities.md index a261e18c31c..1c75a019b3f 100644 --- a/applications/ipfs_utilities.md +++ b/applications/ipfs_utilities.md @@ -115,9 +115,9 @@ A concept and solution draft was created as part of this application and can be ### Overview -- **Total Estimated Duration:** 4 Months +- **Total Estimated Duration:** 2 Months - **Full-Time Equivalent (FTE):** 2 -- **Total Costs:** 178,880 DAI +- **Total Costs:** 89,440 DAI ### Milestone 1 - Research and PoC implementation on upload part @@ -151,24 +151,6 @@ The goal of the second milestone is to redirect utilities of Substrate. A gRPC r | 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | | 1. | Implement | `#[prg(name = "nft_getFileById")]` will be designed and developed that allows users to redirect a request, which returns `Result`, to a file server. | -### Milestone 3 - Integration and Benchmark - -- **Estimated Duration:** 2 Months -- **FTE:** 2 -- **Costs:** 89,440 DAI - -The goal of the third milestone is to have several features and APIs that cover potential use cases. - - Number | Deliverable | Specification | -| -----: | ----------- | ------------- | -| 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | -| 0b. | Documentation | We will provide both **inline documentation** of the code and a basic **tutorial** that explains how a developer can re-use the implementation. | -| 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | -| 1. | Configure | The IPFS pallet implementation will be added with IPFS gateway in configure. | -| 2. | Query | The IPFS pallet can be queried and return a different version of CID as String. | -| 3. | Example | The example pallet can use IPFS pallet to store some data in HTML format in IPFS and provide gRPC to redirect requests to get the HTML content. | -| 4. | Benchmark | Benchmark on the size of the file the IPFS pallet can be accepted. | - ## Additional Information :heavy_plus_sign: From e50b81f9db47ebd5bdc343d10085e4becd3661da Mon Sep 17 00:00:00 2001 From: Sascha Dobschal Date: Tue, 6 Sep 2022 11:15:09 +0200 Subject: [PATCH 3/4] Update IPFS util application --- applications/ipfs_utilities.md | 41 +++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/applications/ipfs_utilities.md b/applications/ipfs_utilities.md index 1c75a019b3f..d75954f03d5 100644 --- a/applications/ipfs_utilities.md +++ b/applications/ipfs_utilities.md @@ -17,22 +17,31 @@ This application is about the utilities of Substrate within IPFS. IPFS is a web3 solution to store and provide big-sized and not structured data. The datas address (index) is based on it's content, which guarantees that the content or file isn't changed or manipulated in any way. Currently there is no satisfying way of an integration for IPFS on Substrate blockchains. Currently, the easiest way to record IPFS CIDs on a Substrate-based blockchain, is to call an extrinsic and pass the IPFS CID as an input parameter to store it on chain. The CID is often related to some non-fungible token (NFT), and becomes a metadata of the token. +In NFT scenario, it would need much more affairs to the validation process on the chain side, if the developer wants to make sure and trust the input CID will point to the corresponding file on the chain. Besides that, the CID is listed via gRPC as a string field, it can not be a presented event if the browser already supports IPFS. -In the solution, it will need much more affairs to the validation process on the chain side, if the developer wants to make sure and trust the input CID will point to the corresponding file on the chain. Besides, the CID is listed via gRPC as a string field, it can not be presented event the browser already supports IPFS. +This project will follow a different approach. The upload and linking of a file to an NFT e.g. can be done by clients thru one extrinsic. On upload of a file with the IPFS pallet, the pallet will return the ticket id immediately. Once the file is uploaded to IPFS, the CID of the file on IPFS will be stored on chain. The ticket id and CID both can be used to fetch the file via RPC call. While the upload to IPFS is processing, a cache logic can already provide the file based on the ticket id. Once the CID is retrieved and the file is available on IPFS, the cache can be cleared and all fetch requests are redirected to IPFS. ### Project Details +#### Example: NFT with IPFS hosted file + +This is a draft on how to integrate the IPFS util with an NFT. The upload process is async as it takes some time. During upload a ticket id links to the processed file. Once uploaded the CID is used. + +![](https://tdsoftware-dev.de/w3f-grant-program/ipfs-util-components.png) + #### Upload utilities -In the first part, detailed research is needed to upload data to IPFS, including +In the first part, detailed research is needed to upload data to IPFS, including: - Accepting bytes as an input parameter of extrinsic -- Uploading `&[u8]` or `[u8]` to IPFS, may or may not in off-chain worker -- Storing CID information on chain, also the CID version 0 and CID version 1 could be handled +- Uploading `&[u8]` or `[u8]` to IPFS thru an off-chain worker +- Storing CID information on chain +- Handling the CID version 0 and CID version 1 - Benchmarking on the extrinsic call to know the maximum size of bytes can work + #### Read/access utilities -In the second part, the redirect feature will be provided based on the gRPC mechanism, the details are listed -- Accept a function with `Result` return, and take the content of String as the location to redirect +In the second part, the redirect and fetch feature will be provided based on the gRPC mechanism. This will allow users to fetch the linked files from IPGS. The details are listed: +- Accept a function with `Result` as return value, and take the content of String as the location to redirect - Developer can define the redirect location as CIDv0 or CIDv1. - Base on Post/Redirect/Get ([PRG](https://en.wikipedia.org/wiki/Post/Redirect/Get)) pattern - The return code is HTTP 303 See Other @@ -53,6 +62,7 @@ Blockchain - Rust - IPFS + ## Team :busts_in_silhouette: ### Team members @@ -109,6 +119,8 @@ Team profiles: A concept and solution draft was created as part of this application and can be found in the Project Details chapter. +We will reuse as much as possible the existing developed approach from: [offchain::ipfs](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/offchain_ipfs.md). +This solution is not handling the CID in a proper way. E.g. instead of returning the CID, it's just logging it out. It also provides the reading of the file thru an extrinsic. As the file is public on IPFS, that is not necessary. ## Development Roadmap :nut_and_bolt: @@ -119,22 +131,22 @@ A concept and solution draft was created as part of this application and can be - **Full-Time Equivalent (FTE):** 2 - **Total Costs:** 89,440 DAI -### Milestone 1 - Research and PoC implementation on upload part +### Milestone 1 - Research and final implementation of file upload feature - **Estimated duration:** 1 Month - **FTE:** 2 - **Costs:** 44,720 DAI -After the first research, the PoC pallet implementation will be developed. The goal of milestone 1, is a fully working round trip (except the redirect feature). A PoC client, reading a file, calling an extrinsic and uploading the content of a file,, will be developed. The CID information from IPFS will be stored in the pallet, then the uploaded file is verified by other IPFS tools. +After the first research, the pallet implementation will be developed. The goal of milestone 1, is a fully working round trip (except the redirect feature). A web3 client, reading a file, calling an extrinsic and uploading the content of a file, will be developed. The CID information from IPFS will be stored in the pallet, then the uploaded file is verified by other IPFS tools. Number | Deliverable | Specification | | -----: | ----------- | ------------- | | 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | | 0b. | Documentation | We will provide both **inline documentation** of the code and a basic **tutorial** that explains how a developer can re-use the implementation. | | 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | -| 1. | Implement IPFS upload extrinsic | The pallet implementation will be added, that allow bytes to uploads. | -| 2. | Implement POC client | The client can read data from file, call extrinsic and upload the content in bytes. | -| 3. | Verification | Verify the file uploaded | +| 1. | Implement IPFS upload extrinsic | The pallet implementation will be added, that allows to upload bytes of files to IPFS. The data will be cached to provide it during processing. This will return a ticket id for the file. Once uploaded the CID is available too. | +| 2. | Implement client | The web3 client app can read data from file, call extrinsic and upload the content in bytes. | +| 3. | Verification | Verify the file uploaded and store the CID on chain. | ### Milestone 2 - PoC implementation on redirect part @@ -142,14 +154,17 @@ After the first research, the PoC pallet implementation will be developed. The g - **FTE:** 2 - **Costs:** 44,720 -The goal of the second milestone is to redirect utilities of Substrate. A gRPC request can be based on a PRG mechanism to redirect to any file server, which is not limited by IPFS. +The goal of the second milestone is to redirect utilities of Substrate. A gRPC request can be based on a PRG mechanism to redirect to any file server, which is not limited by IPFS. A cache will be added to allow fetching the file, when the upload to IPFS is still in progress. Number | Deliverable | Specification | | -----: | ----------- | ------------- | | 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | | 0b. | Documentation | We will provide both **inline documentation** of the code and a basic **tutorial** that explains how a developer can re-use the implementation. | | 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | -| 1. | Implement | `#[prg(name = "nft_getFileById")]` will be designed and developed that allows users to redirect a request, which returns `Result`, to a file server. | +| 0d. | Code | We will provide the whole code in a public GitHub repository. | +| 0e. | Article | We will write and publish a Medium article to spread the word and give developers an introduction to the project and how to use it. | +| 1. | Implement Fetch File | E.g. `#[prg(name = "nft_getFileById")]` will be designed and developed that allows users to redirect a request, which returns `Result`, to a file server. | +| 2. | Implement Cache | Implement a file cache, that returns the file if upload to IPFS is still in progress. | From 6b5db3fbce7dd0ee79859f1712b56bcebff697ee Mon Sep 17 00:00:00 2001 From: Sascha Dobschal Date: Tue, 27 Sep 2022 10:14:30 +0200 Subject: [PATCH 4/4] Update IPFS utilities grant application --- applications/ipfs_utilities.md | 101 ++++++++++++++++++++------------- 1 file changed, 60 insertions(+), 41 deletions(-) diff --git a/applications/ipfs_utilities.md b/applications/ipfs_utilities.md index d75954f03d5..a467b7b87ec 100644 --- a/applications/ipfs_utilities.md +++ b/applications/ipfs_utilities.md @@ -6,53 +6,69 @@ - **Level:** 3 - ## Project Overview -This application is about the utilities of Substrate within IPFS. +IPFS is a web3 solution to store and provide big-sized and not structured data. The datas address is based on it's content (CID: Content Identifier), which guarantees that the content or file isn't changed or manipulated in any way. The same CID will always point to the same content spread in the IPFS network. Currently there is no satisfying way of an integration for IPFS on Substrate blockchains. An example use case are NFTs (Non-Fungible Tokens): An NFT for instance might be linked to an external artwork (image, audio file, video file, ...) that is stored within IPFS. +### Problem -### Overview +Currently, the easiest way to record IPFS CIDs on a Substrate-based blockchains, is to call an extrinsic and pass the IPFS CID as an input parameter to store it on chain. So two separate steps are necessary: +1. Add (artwork) file to IPFS to retrieve CID +2. Store CID for NFT on chain -IPFS is a web3 solution to store and provide big-sized and not structured data. The datas address (index) is based on it's content, which guarantees that the content or file isn't changed or manipulated in any way. Currently there is no satisfying way of an integration for IPFS on Substrate blockchains. +Because this are two separate actions to be performed by a client, we cannot consider this to be secure. +In the second step we cannot verify that the previously uploaded content in the IPFS network is the correct wanted one. -Currently, the easiest way to record IPFS CIDs on a Substrate-based blockchain, is to call an extrinsic and pass the IPFS CID as an input parameter to store it on chain. The CID is often related to some non-fungible token (NFT), and becomes a metadata of the token. -In NFT scenario, it would need much more affairs to the validation process on the chain side, if the developer wants to make sure and trust the input CID will point to the corresponding file on the chain. Besides that, the CID is listed via gRPC as a string field, it can not be a presented event if the browser already supports IPFS. +In addition, there is no redirect feature on Substrate blockchains to request a file from IPFS via Substrate RPC. -This project will follow a different approach. The upload and linking of a file to an NFT e.g. can be done by clients thru one extrinsic. On upload of a file with the IPFS pallet, the pallet will return the ticket id immediately. Once the file is uploaded to IPFS, the CID of the file on IPFS will be stored on chain. The ticket id and CID both can be used to fetch the file via RPC call. While the upload to IPFS is processing, a cache logic can already provide the file based on the ticket id. Once the CID is retrieved and the file is available on IPFS, the cache can be cleared and all fetch requests are redirected to IPFS. +### Solution +Provide a pallet implementation that allows clients to add files to IPFS and store the related CID on chain with one RPC. For instance when minting an NFT with a given artwork file. + +In addition to that, the Substrate blockchain can offer an RPC API to read the file from IPFS too. Therefore a redirect feature needs to be implemented based on the PRG mechanism. +As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can used to retrieve the file. -### Project Details -#### Example: NFT with IPFS hosted file +### Project Details -This is a draft on how to integrate the IPFS util with an NFT. The upload process is async as it takes some time. During upload a ticket id links to the processed file. Once uploaded the CID is used. +#### Previous Work -![](https://tdsoftware-dev.de/w3f-grant-program/ipfs-util-components.png) +The implementation from [offchain_ipfs](https://github.com/rs-ipfs/substrate/tree/offchain_ipfs) provides a working local IPFS node and utilities to interact with that node. This upcoming project will use the existing IPFS node implementation and will add adjustments and more functionality as shown below. The current implementation especially misses a proper handling of the CID and has no redirect feature to fetch files from IPFS with Substrate. It also uses a signed extrinsic method to fetch files from IPFS: that is not necessary, because the file is public available. -#### Upload utilities -In the first part, detailed research is needed to upload data to IPFS, including: -- Accepting bytes as an input parameter of extrinsic -- Uploading `&[u8]` or `[u8]` to IPFS thru an off-chain worker -- Storing CID information on chain +#### Upload Utilities +In the first part, detailed research on the existing implementation is needed. The following things will be implemented or adjusted: +- Accepting bytes as an input parameter of an extrinsic to end up in IPFS +- Adding `&[u8]` or `[u8]` to the local IPFS node thru an off-chain worker +- Wait for the added file to be available thru a public IPFS gateway like ipfs.io +- Store CID information on chain +- After file is available thru public IPFS gateway, remove it from the on-chain storage - Handling the CID version 0 and CID version 1 -- Benchmarking on the extrinsic call to know the maximum size of bytes can work +- Benchmarking on the extrinsic call to know the maximum size of bytes accepted + +

+ +

-#### Read/access utilities -In the second part, the redirect and fetch feature will be provided based on the gRPC mechanism. This will allow users to fetch the linked files from IPGS. The details are listed: -- Accept a function with `Result` as return value, and take the content of String as the location to redirect -- Developer can define the redirect location as CIDv0 or CIDv1. -- Base on Post/Redirect/Get ([PRG](https://en.wikipedia.org/wiki/Post/Redirect/Get)) pattern -- The return code is HTTP 303 See Other -- E.g. a user can `#[prg(name = "nft_getFileById")]` +#### Read/Access Utilities +In the second part, the redirect and fetch feature will be provided based on a PRG mechanism. This will allow users to fetch the linked files from IPFS by requesting the Substrate blockchain. The second part includes: +- Implement RPC API for file redirect to public IPFS gateway +- Based on Post/Redirect/Get ([PRG](https://en.wikipedia.org/wiki/Post/Redirect/Get)) pattern for gRPC +- API will return HTTP status code 303, so that the redirect will change the HTTP method from POST to GET +- API will return the actual file location. String value with URL. +- Will redirect to public IPFS gateway. E.g. ipfs.io +- This feature is not limited to IPFS and can be used with any file server +- Developers can define the redirect location as CIDv0 or CIDv1. + +

+ +

### Ecosystem Fit -The implementation helps people using IPFS with substrate, this is a common scenario. -People fork substrate and try to solve this in [offchain_ipfs](https://github.com/rs-ipfs/substrate/tree/offchain_ipfs), -but it is based on a two years old substrate version, and not actively updated. +The implementation helps people using IPFS with Substrate, this is a common scenario. +Especially for NFT products this solution is helpful to avoid external middleware implemetations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure. #### Technology Stack @@ -65,7 +81,7 @@ Blockchain ## Team :busts_in_silhouette: -### Team members +### Team Members - Sascha Dobschal - Antonio Yang @@ -119,7 +135,7 @@ Team profiles: A concept and solution draft was created as part of this application and can be found in the Project Details chapter. -We will reuse as much as possible the existing developed approach from: [offchain::ipfs](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/offchain_ipfs.md). +As described above, we will reuse the existing IPFS node implementation developed by: [offchain::ipfs](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/offchain_ipfs.md). This solution is not handling the CID in a proper way. E.g. instead of returning the CID, it's just logging it out. It also provides the reading of the file thru an extrinsic. As the file is public on IPFS, that is not necessary. @@ -129,32 +145,34 @@ This solution is not handling the CID in a proper way. E.g. instead of returning - **Total Estimated Duration:** 2 Months - **Full-Time Equivalent (FTE):** 2 -- **Total Costs:** 89,440 DAI +- **Total Costs:** 66,000 DAI -### Milestone 1 - Research and final implementation of file upload feature +### Milestone 1 - Research and implementation of file upload feature - **Estimated duration:** 1 Month - **FTE:** 2 -- **Costs:** 44,720 DAI +- **Costs:** 33,000 DAI -After the first research, the pallet implementation will be developed. The goal of milestone 1, is a fully working round trip (except the redirect feature). A web3 client, reading a file, calling an extrinsic and uploading the content of a file, will be developed. The CID information from IPFS will be stored in the pallet, then the uploaded file is verified by other IPFS tools. +After the first research, the pallet implementation will be developed. The goal of milestone 1, is a fully working round trip (except the redirect feature). A web3 client, reading a file, calling an extrinsic and uploading the content of a file, will be developed. The CID information from IPFS will be stored in the pallet. Please see the chapter above ("Project Details > Upload utilities") for details. Number | Deliverable | Specification | | -----: | ----------- | ------------- | | 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | | 0b. | Documentation | We will provide both **inline documentation** of the code and a basic **tutorial** that explains how a developer can re-use the implementation. | | 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | -| 1. | Implement IPFS upload extrinsic | The pallet implementation will be added, that allows to upload bytes of files to IPFS. The data will be cached to provide it during processing. This will return a ticket id for the file. Once uploaded the CID is available too. | -| 2. | Implement client | The web3 client app can read data from file, call extrinsic and upload the content in bytes. | -| 3. | Verification | Verify the file uploaded and store the CID on chain. | +| 0d. | Code | We will provide the whole code in a public GitHub repository. | +| 1. | Implementation of the IPFS upload extrinsic | The pallet implementation will be added, that takes bytes of a file as input parameters. Adjust the extrinsic method `ipfs_add_bytes` to trigger an event `IpfsAdd(CID)`. | +| 2. | Implementation to retrieve CID and store it on chain | The off-chain worker needs to be extended. Data (bytes) received will be added to the local IPFS node via the IPFS offchain worker. Once added, it stores the CID on chain. | +| 3. | Implementation of a web3 client | For testing and the use case of a tutorial, a web3 client with dedicated UI will be implemented. The client app can read the data from a file, call the implemented extrinsic and uploads the content in bytes to the Substrate node. | +| 4. | Benchmark and adjustments | To check the maximum accepted file size, a benchmarking is performed. The result should give some concrete limits that will be added to the implemetation. | -### Milestone 2 - PoC implementation on redirect part +### Milestone 2 - Implementation of the redirect/fetch file feature - **Estimated Duration:** 1 Month - **FTE:** 2 -- **Costs:** 44,720 +- **Costs:** 33,000 DAI -The goal of the second milestone is to redirect utilities of Substrate. A gRPC request can be based on a PRG mechanism to redirect to any file server, which is not limited by IPFS. A cache will be added to allow fetching the file, when the upload to IPFS is still in progress. +The goal of the second milestone is to implement the redirect/fetch utilities. A RPC request, based on a PRG mechanism, can be used to redirect to any file server, which is not limited to IPFS. If the file is not available thru a public IPFS gateway, the local node can be used to retrieve the file. Number | Deliverable | Specification | | -----: | ----------- | ------------- | @@ -163,8 +181,9 @@ The goal of the second milestone is to redirect utilities of Substrate. A gRPC r | 0c. | Testing | Core functions will be covered by unit tests as far as reasonably applicable to ensure functionality and robustness. In the documentation, we will describe how to run these tests. | | 0d. | Code | We will provide the whole code in a public GitHub repository. | | 0e. | Article | We will write and publish a Medium article to spread the word and give developers an introduction to the project and how to use it. | -| 1. | Implement Fetch File | E.g. `#[prg(name = "nft_getFileById")]` will be designed and developed that allows users to redirect a request, which returns `Result`, to a file server. | -| 2. | Implement Cache | Implement a file cache, that returns the file if upload to IPFS is still in progress. | +| 1. | Fetch File Implementation | Implement the logic that allows clients to fetch a file from IPFS with the Substrate node. E.g. `#[prg(name = "nft_getFileById")]` will be designed and developed that allows users to redirect a request, which returns `Result`, to a file server. The PRG mechanism is used to provide a RPC API that can redirect to a file server. The existing IPFS "get file" extrinsic will be obsolet as we provide this new RPC API. | +| 2. | Web3 Client Extension | The web3 client will be extend to show the (PRG) redirect feature and retrieve corresponding files via a public IPFS gateway. | +| 3. | Local Node Fetch Implementation | Implement a file fetch RPC API, that returns the file from the local embedded IPFS node instead of the public IPFS gateway. This is helpful when waiting for the file to be available on the public IPFS gateway. |