diff --git a/docs/2024/index.md b/docs/2024/index.md index 70b31ca55..e9db2a779 100644 --- a/docs/2024/index.md +++ b/docs/2024/index.md @@ -26,7 +26,7 @@ More info to come here. | :--------------------------------------------------- | :----------------------------------------------------------- | | [Aaditya Singh](https://github.com/aadsingh) | [Overhaul Scheduler Design](/docs/2024/scheduler) | | [Abdelrahman Jamal](https://github.com/Hero2323) | [AI Powered License Detection](/docs/2024/license-detection) | -| [Abhishek Kumar](https://github.com/abhi-kumar17871) | [SPDX 3.0 Support](/docs/2024/spdx30) | +| [Abhishek Kumar](https://github.com/abhi-kumar17871) | [Support SPDX 3.0 Reports](/docs/2024/spdx30) | | [Akash Sah](https://github.com/AkashSah2003) | [SPDX License Expression](/docs/2024/spdx-expression) | | [Divij Sharma](https://github.com/dvjsharma) | [REST API Improvements](/docs/2024/rest) | | [Rajul Jha](https://github.com/rajuljha) | [Improving CI Scanner](/docs/2024/ci-scanner) | diff --git a/docs/2024/scheduler/asset/set_priority.png b/docs/2024/scheduler/asset/set_priority.png new file mode 100644 index 000000000..26e77dd95 Binary files /dev/null and b/docs/2024/scheduler/asset/set_priority.png differ diff --git a/docs/2024/scheduler/updates/Discussion/2024-06-14.md b/docs/2024/scheduler/updates/Discussion/2024-06-14.md new file mode 100644 index 000000000..511901898 --- /dev/null +++ b/docs/2024/scheduler/updates/Discussion/2024-06-14.md @@ -0,0 +1,29 @@ +title: week-06 +author: Aaditya Singh +--- + +# Meeting 10 +*(June 14, 2024)* + +## Participants: + - [Gaurav Mishra](https://github.com/GMishx) + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + +## Discussion Topics: + +**Contributor:** + - [Aaditya Singh](https://github.com/Aaditya-Singh78): Could you provide more details on the ER diagram? I need to know the specific parameters to integrate into the code. + +**Mentor:** + - [Gaurav Mishra](https://github.com/GMishx): Let's discuss the data within the database and the essential criteria to consider when operating the scheduler: + + 1. Certain agents like delagent and software heritage must operate singularly due to their time-intensive nature, affecting users who need to run multiple agents. + + 2. Some agents depend on external APIs with rate limits. Surpassing these limits could lead to access restrictions or complete blocks on agent processing. + + 3. A maintenance agent exists to support our website, focusing on dependency resolutions and performance enhancements. + + 4. Ensure responsiveness is a priority—analysis for users who request first should be completed first. diff --git a/docs/2024/scheduler/updates/Discussion/2024-06-21.md b/docs/2024/scheduler/updates/Discussion/2024-06-21.md new file mode 100644 index 000000000..6242a52dd --- /dev/null +++ b/docs/2024/scheduler/updates/Discussion/2024-06-21.md @@ -0,0 +1,59 @@ +--- +title: week-07 +author: Aaditya Singh +--- + +--- + +# Meeting 14 +*(June 21,2024)* + +## Participants + +### Mentors + + - [Katharina Ettinger](https://github.com/EttingerK) + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) + - [Kaushlendra Pratap](https://github.com/Kaushl2208) + - [Gaurav Mishra](https://github.com/GMishx) + +### Contributors + + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + +## Discussion: + +**Contributor:** + + - [Aaditya Singh](https://github.com/Aaditya-Singh78): Presented a thorough analysis of a use case, identifying several important aspects: + 1. Clear separation of dependent and independent job queues. + 2. Implementation of a rate limiter using a sliding window log. + 3. Application of semaphores for queue synchronization. + 4. Integration of feedback through *[Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)*. + + ![golang_architecture](../../asset/golang_arch.png) + + > ⚠️ In the diagram above, mutexes and channels for synchronization haven't been considered yet. Introducing single semaphore into the diagram might undermine the goal of achieving parallelism in extreme situations. + + +- [Aaditya Singh](https://github.com/Aaditya-Singh78): Concluded that the architecture faces several challenges: + 1. Potential deadlocks from using multiple semaphores. + 2. Limitations of using a single semaphore under certain conditions affecting multiprogramming. + 3. The need for a load balancer to prioritize users based on their analysis needs or project size. + 4. Establishment of a heartbeat mechanism to monitor system performance. + +**Mentors**: + + - [Kaushlendra Pratap](https://github.com/Kaushl2208): Discussed the potential deadlock issues with the use of multiple semaphores. + + - [Gaurav Mishra](https://github.com/GMishx): Suggested the implementation of a heartbeat system, possibly through Grafana, to oversee system health. + + - [Katharina Ettinger](https://github.com/EttingerK): Asked about the scalability of the current approach given the project size and complexity. + + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd): Proposed integrating a priority mechanism within the dependent queue to optimize processing time for smaller or less complex projects. + + >ℹ️: Additional studies and enhancements are considered for these areas. diff --git a/docs/2024/scheduler/updates/Discussion/2024-06-28.md b/docs/2024/scheduler/updates/Discussion/2024-06-28.md new file mode 100644 index 000000000..7f64e8d32 --- /dev/null +++ b/docs/2024/scheduler/updates/Discussion/2024-06-28.md @@ -0,0 +1,80 @@ +--- +title: week-08 +author: Aaditya Singh +--- + + +# Meeting 16 +*(June 28,2024)* + +## Attendees: + + - [Katharina Ettinger](https://github.com/EttingerK) + + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) + + - [Avinal Kumar](https://github.com/avinal) + + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + + +## Discussion: + +- [Aaditya Singh](https://github.com/Aaditya-Singh78): + + Shared the updated architecture of the Scheduler via this link for team review: + + [Scheduler Architecture](../../asset/golang_arch.png) + + Detailed enhancements to overcome specific challenges: + 1. Deadlock handling through channels facilitating communication between two semaphores, intended to transmit data structures, ideally integers representing user IDs or group IDs. + 2. Addressing race conditions among go-routines. + 3. Implementing a heartbeat mechanism between go-routines and a load balancer, utilizing channels to check the state of go-routines, returning a boolean value to minimize overhead in frequent signaling. + +- [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd): Emphasized the importance of package size as a critical parameter in Fossology, suggesting communication to users about partitioning large packages or providing a UI page for package size-related settings. + + [Admin Scheduler](../../asset/set_priority.png) + + +- [Katharina Ettinger](https://github.com/EttingerK): Questioned the criteria for determining whether packages are of larger or smaller size before processing begins. + +- [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd): Clarified that the `Unpackage` agent determines the package size, stating that the current setup looks promising and encouraged starting coding for further enhancements. + +- [Katharina Ettinger](https://github.com/EttingerK): Inquired about the possibility of pausing smaller packages as a form of context switching. + +- [Aaditya Singh](https://github.com/Aaditya-Singh78): Noted that context switching is currently handled under go-routines, which are synchronized per user, making pausing not feasible at the moment. + +- [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd): Suggested the development of a web page to make informed decisions based on package size. + +- [Avinal Kumar](https://github.com/avinal): Complimented the improvements from the previous version and recommended proceeding with coding. + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/2024/scheduler/updates/Discussion/2024-07-05.md b/docs/2024/scheduler/updates/Discussion/2024-07-05.md new file mode 100644 index 000000000..edd05ca96 --- /dev/null +++ b/docs/2024/scheduler/updates/Discussion/2024-07-05.md @@ -0,0 +1,42 @@ +--- +title: week-09 +author: Aaditya Singh +--- + +--- + +# Meeting 16 +*(July 05, 2024)* + +## Attendees + +**Mentors:** + + - [Katharina Ettinger](https://github.com/EttingerK) + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) + - [Kaushlendra Pratap](https://github.com/Kaushl2208) + - [Avinal Kumar](https://github.com/avinal) + +**Contributors:** + + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + +## Discussions: + +**Contributors:** + + - [Aaditya Singh](https://github.com/Aaditya-Singh78): Focused on developing code through database integration and organizing the structure based on the hexagonal architecture to prevent circular dependencies and separate logic. Additionally, raised a question regarding the `sqlstatements.h` file. + +**Mentors:** + + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd): Clarified queries about the operations of the `sqlstatements.h` file, discussing the ER diagram of the table and how it interacts with user actions. + + - [Avinal Kumar](https://github.com/avinal): Recommended combining commands in `cmake` as follows: + ``` + custom_Target: + custom_command: () + ``` + Recommended reviewing the [go-cmake](https://github.com/mutse/go-cmake) repository for additional insights. diff --git a/docs/2024/scheduler/updates/Discussion/_category_.json b/docs/2024/scheduler/updates/Discussion/_category_.json new file mode 100644 index 000000000..335883f2c --- /dev/null +++ b/docs/2024/scheduler/updates/Discussion/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Discussion", + "position": 4 +} diff --git a/docs/2024/scheduler/updates/General/2024-06-13.md b/docs/2024/scheduler/updates/General/2024-06-13.md new file mode 100644 index 000000000..01b9f4a8f --- /dev/null +++ b/docs/2024/scheduler/updates/General/2024-06-13.md @@ -0,0 +1,101 @@ +--- +title: week-06 +author: Aaditya Singh +--- + + +# Meeting 09 +*(June 13, 2024)* + +## Attendees: + + - [Gaurav Mishra](https://github.com/GMishx) + + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) + + - [Kaushlendra Pratap](https://github.com/Kaushl2208) + + - [Ayush Bhardwaj](https://github.com/hastagAB) + + - [Abdelrahman Jamal](https://github.com/Hero2323) + + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + + - [Akash Sah](https://github.com/Akashsah2003) + + - [Divij Sharma](https://github.com/dvjsharma) + + - [Rajul Jha](https://github.com/rajuljha) + + - [Avinal Kumar](https://github.com/avinal) + +## Absentees: + + - [Katharina Ettinger](https://github.com/EttingerK) + + - [Samuel Dushimimana](https://github.com/dushimsam) + + - [Soham Banerjee](https://github.com/soham4abc) + + - [Shreya Singh](https://github.com/SinghShreya05) + + - [Abhishek Kumar](https://github.com/abhi-kumar17871) + + - [Valens Niyonsenga](https://github.com/valens200) + + - [Anupam Ghosh](https://github.com/ag4ums) + + - [Sahil Jha](mailto:sjha200000@gmail.com) + + - [Vasudev Maduri](https://github.com/vasudevmaduri) + + - [Shreya Gautam](https://github.com/ShreyaGautamm) + +## General Discussion + +**Mentor:** +- [Gaurav Mishra](https://github.com/GMishx): Provided project updates. + +**Contributors:** + +- [Abdelrahman Jamal](https://github.com/Hero2323) + + - Conducted tests with different models using Semantic Search for similarity matches. + + - Tested semantic transformers and worked on a basic license identifier. + +- [Shreya Gautam](https://github.com/ShreyaGautamm) + + - Was absent. + +- [Aaditya Singh](https://github.com/Aaditya-Singh78) + + - Following my last conversation with mentors, began rewriting scheduler code from C to Go. Submitted a pull request for the same. + + - Focused on implementing a queue and working on CLI code. + +- [Abhishek Kumar](https://github.com/abhi-kumar17871) + + - Developed SPDX 3.0 JSON format, exploring validation of generated reports. Plans to start on RDF report soon. + +- [Akash Sah](https://github.com/Akashsah2003) + + - Per discussions with mentors, utilized JSON format to convert license expressions and store them in a database (successful). + + - Developed a user interface for the same and is now adding comments, acknowledgments, and text for license expressions. + +- [Divij Sharma](https://github.com/dvjsharma) + + - Completed OAUTH implementation via API. Currently transitioning API from V1 to V2. + +- [Rajul Jha](https://github.com/rajuljha) + + - Focused on adding line number support for repo scans. Next, will work on nomos and ojo components. + +- [Valens Niyonsenga](https://github.com/valens200) + + - Implemented unit test cases for APIs such as License, Group, and Upload. Submitted a pull request for the same. diff --git a/docs/2024/scheduler/updates/General/2024-06-20.md b/docs/2024/scheduler/updates/General/2024-06-20.md new file mode 100644 index 000000000..df4da71e9 --- /dev/null +++ b/docs/2024/scheduler/updates/General/2024-06-20.md @@ -0,0 +1,71 @@ +--- +title: week-07 +author: Aaditya Singh +--- + + +# Meeting 13 +*(June 20, 2024)* + +## Participants: + + - [Katharina Ettinger](https://github.com/EttingerK) + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) + - [Kaushlendra Pratap](https://github.com/Kaushl2208) + - [Vasudev Maduri](https://github.com/vasudevmaduri) + - [Samuel Dushimimana](https://github.com/dushimsam) + - [Abdelrahman Jamal](https://github.com/Hero2323) + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + - [Akash Sah](https://github.com/Akashsah2003) + - [Divij Sharma](https://github.com/dvjsharma) + - [Rajul Jha](https://github.com/rajuljha) + - [Valens Niyonsenga](https://github.com/valens200) + - [Abhishek Kumar](https://github.com/abhi-kumar17871) + +## Not Present: + + - [Gaurav Mishra](https://github.com/GMishx) + - [Soham Banerjee](https://github.com/soham4abc) + - [Shreya Singh](https://github.com/SinghShreya05) + - [Anupam Ghosh](https://github.com/ag4ums) + - [Sahil Jha](mailto:sjha200000@gmail.com) + - [Shreya Gautam](https://github.com/ShreyaGautamm) + - [Ayush Bhardwaj](https://github.com/hastagAB) + - [Avinal Kumar](https://github.com/avinal) + +## Overview + +**Mentor:** +- [Kaushlendra Pratap](https://github.com/Kaushl2208): Shared general updates about the project. + +**Contributors:** + +- [Abdelrahman Jamal](https://github.com/Hero2323) + - Focused on Semantic Search to find closely related results. + - Evaluated a semantic transformer and developed a basic license identifier (Demo of current work). + +- [Shreya Gautam](https://github.com/ShreyaGautamm) + - Was absent. + +- [Aaditya Singh](https://github.com/Aaditya-Singh78) + - Following discussions with mentors, conducted a gap analysis and worked on the system architecture (Demo of current structure). + +- [Abhishek Kumar](https://github.com/abhi-kumar17871) + - Developed SPDX 3.0 JSON format, noting that the validation tool does not support this format (Discussion with Linux community needed by Gaurav). + +- [Akash Sah](https://github.com/Akashsah2003) + - Completed a UI for license expression including comment acknowledgements and related texts (Demo of current implementation). + +- [Divij Sharma](https://github.com/dvjsharma) + - Implemented OAUTH through an API and is now working on copyright endpoints for user copyrights (Demo of OAUTH). + +- [Rajul Jha](https://github.com/rajuljha) + - Modified the nomos scanner to include byte information, initiated work on custom keyword functionality, and revised GH Action workflow (Significant time reduction). + +- [Valens Niyonsenga](https://github.com/valens200) + - Added unit tests for APIs related to License, Group, and Upload, and is now focusing on completing license API tests. + - Exploring FOSSology through YouTube tutorials. diff --git a/docs/2024/scheduler/updates/General/2024-06-27.md b/docs/2024/scheduler/updates/General/2024-06-27.md new file mode 100644 index 000000000..016a8b9d9 --- /dev/null +++ b/docs/2024/scheduler/updates/General/2024-06-27.md @@ -0,0 +1,106 @@ +--- +title: Week-08 +author: Aaditya Singh +--- + + +# Meeting 15 +*(June 27,2024)* + +## Attendees: + + - [Katharina Ettinger](https://github.com/EttingerK) + + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) + + - [Gaurav Mishra](https://github.com/GMishx) + + - [Kaushlendra Pratap](https://github.com/Kaushl2208) + + - [Abdelrahman jamal](https://github.com/Hero2323) + + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + + - [Akash Sah](https://github.com/Akashsah2003) + + - [Divij Sharma](https://github.com/dvjsharma) + + - [Rajul Jha](https://github.com/rajuljha) + + - [Abhishek Kumar](https://github.com/abhi-kumar17871) + +## Missed: + + - [Valens Niyonsenga](https://github.com/valens200) + + - [Samuel Dushimimana](https://github.com/dushimsam) + + - [Vasudev Maduri](https://github.com/vasudevmaduri) + + - [Soham Banerjee](https://github.com/soham4abc) + + - [Shreya Singh](https://github.com/SinghShreya05) + + - [Anupam Ghosh](https://github.com/ag4ums) + + - [Sahil Jha](mailto:sjha200000@gmail.com) + + - [Shreya Gautam](https://github.com/ShreyaGautamm) + + - [Ayush Bhardwaj](https://github.com/hastagAB) + + - [Avinal Kumar](https://github.com/avinal) + +## General + +- [**Shaheem Azmal M MD**](https://github.com/shaheemazmalmmd): Addressed general updates, reviewed pull requests, and resolved issues. Focused on developing the License Database UI and backend REST API. + +- [**Akash Sah**](https://github.com/Akashsah2003): Completed the report generation for SPDX, handling tag values and License expressions. Currently enhancing the main license codebase, delving into the codebase, and clarifying aspects in the application report. + +- [**Rajul Jha**](https://github.com/rajuljha): Concluded work on custom keywords functionality, which aids in directing scanner operations to specified locations and keys as requested by the user. + Continued development on the differential scan design, utilizing the `GitHub API` and `GitLab API` to implement distinct tagging strategies discussed in the previous meeting, which assist in distinguishing scan types. + +- [**Gaurav Mishra**](https://github.com/GMishx): Discussed the architecture of differential scanning, which requires two inputs per your suggestions. Several methods were considered, inspired by Git's differential scanning mechanisms in pull requests and merging capabilities. Identified four modes of execution: + 1. Based on Pull Request + 2. Scanning of a repo + 3. User-defined Tags + 4. Current release vs. Pre-release + +- [**Abhishek Kumar**](https://github.com/abhi-kumar17871): Spent the past week working on the validation of JSON Schema. Successfully generated a tag value report. + +- [**Shaheem Azmal M MD**](https://github.com/shaheemazmalmmd): Inquired about the validation of the JSON report. + +- [**Aaditya Singh**](https://github.com/Aaditya-Singh78): Based on discussions from the previous week, proposed solutions to encountered problems, including: + 1. Deadlock Handling + 2. Heartbeat Mechanism + 3. Load based on User + 4. Synchronization between Semaphores. + Explained how the Golang Scheduler functions. + +- [**Gaurav Mishra**](https://github.com/GMishx): Advised on organizing PRs based on commits to track weekly progress rather than batching all commits into one PR. + +- [**Divij Sharma**](https://github.com/dvjsharma): + +- [**Abdelrahman Jamal**](https://github.com/Hero2323): Worked on integrating semantic search with LLM models. + + + + + + + + + + + + + + + + + + diff --git a/docs/2024/scheduler/updates/General/2024-07-04.md b/docs/2024/scheduler/updates/General/2024-07-04.md new file mode 100644 index 000000000..996c2a934 --- /dev/null +++ b/docs/2024/scheduler/updates/General/2024-07-04.md @@ -0,0 +1,69 @@ +--- +title: week-09 +author: Aaditya Singh +--- + + +# Meeting 15 + +*(July 04, 2024)* + +## Attendees: + + - [Gaurav Mishra](https://github.com/GMishx) + - [Kaushlendra Pratap](https://github.com/Kaushl2208) + - [Avinal Kumar](https://github.com/avinal) + - [Abdelrahman Jamal](https://github.com/Hero2323) + - [Aaditya Singh](https://github.com/Aaditya-Singh78) + - [Abhishek Kumar](https://github.com/abhi-kumar17871) + - [Akash Sah](https://github.com/Akashsah2003) + - [Divij Sharma](https://github.com/dvjsharma) + - [Rajul Jha](https://github.com/rajuljha) + - [Valens Niyonsenga](https://github.com/valens200) + +## Absent: + - [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) + - [Ayush Bhardwaj](https://github.com/hastagAB) + - [Samuel Dushimimana](https://github.com/dushimsam) + - [Soham Banerjee](https://github.com/soham4abc) + - [Shreya Singh](https://github.com/SinghShreya05) + +## Discussions: + +**Mentors:** + + - [Gaurav Mishra](https://github.com/GMishx): Provided general project updates. + + - [Avinal Kumar](https://github.com/avinal): Advised [Aaditya Singh](https://github.com/Aaditya-Singh78) to continue using the existing `cmake` build system, explaining that `bazel`, while efficient, does not meet the current needs of Fossology. Changing the build system now is not advisable. + + - [Gaurav Mishra](https://github.com/GMishx): Suggested [Akash Sah](https://github.com/Akashsah2003) to open an issue on the SPDX tool. + +**Contributors:** + +- [Divij Sharma](https://github.com/dvjsharma) + - Developed endpoints supporting standard copyright agents for URLs, emails, and authors. + - Updated details in a previously mentioned PR and completed unit tests for Job endpoints. + +- [Abhishek Kumar](https://github.com/abhi-kumar17871) + - Conducted validations of JSON RDF & LDA formats and completed work on GitHub workflows & tag value report generation. + +- [Rajul Jha](https://github.com/rajuljha) + - Completed differential scans and revised PR accordingly. Plans to test the allow-list functionality next week and explore GitHub actions and Docker Compose. + +- [Abdelrahman Jamal](https://github.com/Hero2323) + - Achieved approximately 70% accuracy in semantic searches, effectively identifying correct licenses. Aims to increase accuracy to 90% next week and will require integration with LLM and some prompt engineering. + +- [Aaditya Singh](https://github.com/Aaditya-Singh78) + - Began coding the scheduler through database connectivity. Facing decisions about the build system `cmake`, which does not support Go. Created documentation on the general coding structure in Go, following the hexagonal architectural pattern. + +- [Akash Sah](https://github.com/Akashsah2003) + - Addressed bugs in the tag value report. Focused on improving the RDF report and integrating it into the database's license table to eliminate duplicates. + +- [Shreya Gautam](https://github.com/ShreyaGautamm) + - Written a Python script to fetch copyright content from the database and modified the copyright. Will tackle a GitHub action issue next. + +- [Valens Niyonsenga](https://github.com/valens200) + - Finalized the addition of controllers and models concerning permission licenses on agents. Plans to work on test cases. \ No newline at end of file diff --git a/docs/2024/scheduler/updates/General/_category_ .json b/docs/2024/scheduler/updates/General/_category_ .json new file mode 100644 index 000000000..b8507683c --- /dev/null +++ b/docs/2024/scheduler/updates/General/_category_ .json @@ -0,0 +1,4 @@ +{ + "label": "General", + "position": 3 +} diff --git a/docs/2024/scheduler/updates/General/_category_.json b/docs/2024/scheduler/updates/General/_category_.json new file mode 100644 index 000000000..b8507683c --- /dev/null +++ b/docs/2024/scheduler/updates/General/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "General", + "position": 3 +} diff --git a/docs/2024/spdx30/_category_.json b/docs/2024/spdx30/_category_.json index e9771b420..d9927caba 100644 --- a/docs/2024/spdx30/_category_.json +++ b/docs/2024/spdx30/_category_.json @@ -1,4 +1,4 @@ { - "label": "SPDX 3.0 Support", + "label": "Support SPDX 3.0 Reports", "position": 5 } diff --git a/docs/2024/spdx30/index.md b/docs/2024/spdx30/index.md index e2a26f886..256d483be 100644 --- a/docs/2024/spdx30/index.md +++ b/docs/2024/spdx30/index.md @@ -6,7 +6,7 @@ slug: /2024/spdx30/ ## Author @@ -15,17 +15,30 @@ SPDX-FileCopyrightText: 2024 Abhishek Kumar ## Contact info -- [Email](mailto:email.here) -- [LinkedIn](https://linkedin.com/in/my-user) +- [Email](mailto:akumar17871@gmail.com) +- [LinkedIn](https://www.linkedin.com/in/akumar17871/) ## Project title -SPDX 3.0 Support +Support SPDX 3.0 Reports ## What's the project about? -Insert Text Here +This project aims to support the generation, parsing, and interpreting of SPDX 3.0 reports. Currently, FOSSology uses SPDX 2.3 for report generation and ingestion, but with the introduction of SPDX 3.0, it has become the industry standard. Upgrading to SPDX 3.0 will address the limitations of the older version, such as compatibility issues and difficulties in integrating with other tools. + +In this project, I will be working on the generation of SPDX 3.0 reports in multiple formats, implementing SPDX 3.0 profiles, and then the ingestion of these reports. ## What should be done? -What are the plans for the project? +1. Support generation of SPDX 3.0 reports in multiple formats + - Include support for JSON-LD, plain JSON, tag:value, RDF, and CSV formats to meet diverse needs and use cases. + +2. Support of SPDX 3.0 profiles + - Begin with Core, Software, and Licensing profiles. + - Expand to additional profiles such as Lite if possible. + +3. Support ingestion of SPDX 3.0 reports + - Ensure FOSSology can read and process SPDX 3.0 reports for seamless integration and compliance. + +4. Improve compatibility and standardization + - Ensure smoother integration and compliance with industry standards. diff --git a/docs/2024/spdx30/updates/2023-05-30.md b/docs/2024/spdx30/updates/2023-05-30.md deleted file mode 100644 index 4ab24b322..000000000 --- a/docs/2024/spdx30/updates/2023-05-30.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: Community bonding -author: Abhishek Kumar ---- - - -# Meeting 1 - -*(May 30,2024)* - -## Attendees: - -## Discussion: - -# Meeting 2 - -*(May 18,2023)* - -## Attendees: - -## Discussion: diff --git a/docs/2024/spdx30/updates/2024-05-07.md b/docs/2024/spdx30/updates/2024-05-07.md new file mode 100644 index 000000000..824e17ea4 --- /dev/null +++ b/docs/2024/spdx30/updates/2024-05-07.md @@ -0,0 +1,62 @@ +--- +title: Community bonding +author: Abhishek Kumar +tags: [gsoc24, spdx30] +--- + + +## Meeting 1 + +*(May 7, 2024)* + +### Discussion: +This was the first meeting with mentors and other fellow contributors. In this meeting, mentors and contributors introduced themselves. We agreed on the time and platform for general weekly meetings. + +## Meeting 2 + +*(May 9, 2024)* + +### Discussion: +* In this meeting, mentors talked about what we can do in community bonding period and emphasized the importance of communication in open source community. +* Q&A session was held to address the general issues. + + +## Week 1 Activities + +*(May 9, 2024 - May 15, 2024)* + +* Revised the basics of PHP and Template Engine- TWIG and other skills required for the project. +* Gone through the documentation and codebase to get clear idea how SPDX v2.3 was implemented and what are the changes that are needed to be related to the project. + + +## Meeting 3 + +*(May 16, 2024)* + +### Discussion: +* Contributors provided updates on their progress. +* Mentors checked if everyone was able to set up their local development environment. + + +## Week 2 Activities + +*(May 16, 2024 - May 22, 2024)* + +* I had some issues while setting up the local environment as I was working on ARM architecture machine having Ubuntu 22.04 installed on my virtual machine. +* So, I rectified the issue by commenting out the installation of python dependencies in [fo-postinstall.in](https://github.com/fossology/fossology/blob/6e6b00c2ded6a1db7647d0da9e97c78ed9ffddf8/install/fo-postinstall.in#L261-L263) file, as it was creating conflicts with other installed dependencies. + + +## Meeting 4 + +*(May 23, 2024)* + +### Discussion: +* Contributors shared their updates. +* Mentors urged contributors to address some of the issues to enhance their understanding of the project. +* Timing for the specific-project weekly meetings were finalized. + + diff --git a/docs/2024/spdx30/updates/2024-05-28.md b/docs/2024/spdx30/updates/2024-05-28.md new file mode 100644 index 000000000..f5f72b1d4 --- /dev/null +++ b/docs/2024/spdx30/updates/2024-05-28.md @@ -0,0 +1,54 @@ +--- +title: Week 1 +author: Abhishek Kumar +tags: [gsoc24, spdx30] +--- + + +# Week 1 meeting and activities + +_(May 28,2024)_ + +## Attendees: + +* [Abhishek Kumar](https://github.com/abhi-kumar17871) +* [Gaurav Mishra](https://github.com/GMishx) +* [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) +* [Dearsh Oberoi](https://github.com/deo002) +* [Akash Sah](https://github.com/Akashsah2003) + +## Discussions + +* I presented the basic SPDX v3.0 JSON-ld format report to the mentors. +* Got a feedback for the implementation of the required fields and profiles in the reports . +* Also was asked to validate the reports generated using the [SPDX python tool](https://github.com/spdx/tools-python). +* Next Step: Work on Licensing profile in JSON-ld reports. + +## Core and software profile use cases + +* ### Core profile + * The Core profile describes the foundational classes and properties that are used by all profiles of the SPDX model. + +* ### Software profile + * The Software profile contains information about files, packages, SBOMs, snippets, and artifacts of the software application. + +The [model image](/img/spdx30/model_Core+Software.png) for the Core+Software profile. +I have implemented the following use cases as required by the organization for this profile: +* Person +* Agent +* Tool +* File +* Package +* Person with full CreationInfo +* Package with ExternalIdentifier +* Relationship with Package containing Files +* SpdxDocument with Files + +## Updates + +* Started working on SPDX v3.0 JSON-ld format report generation. +* In this week, I targeted to implement Core and Software profiles in this format. diff --git a/docs/2024/spdx30/updates/2024-06-04.md b/docs/2024/spdx30/updates/2024-06-04.md new file mode 100644 index 000000000..3a0e558ce --- /dev/null +++ b/docs/2024/spdx30/updates/2024-06-04.md @@ -0,0 +1,40 @@ +--- +title: Week 2 +author: Abhishek Kumar +tags: [gsoc24, spdx30] +--- + + +# Week 2 Meeting and Activities + +_(June 4, 2024)_ + +## Attendees: + +* [Abhishek Kumar](https://github.com/abhi-kumar17871) +* [Gaurav Mishra](https://github.com/GMishx) +* [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) +* [Dearsh Oberoi](https://github.com/deo002) +* [Akash Sah](https://github.com/Akashsah2003) + +## Discussions + +* I presented the changes after implementing the Core and Software Profiles made in the SPDX v3.0 report in JSON-ld format to the mentor. +* I discussed with the mentors about the SPDX python tool for validation of v3.0 reports as it was not optimised for v3.0 reports. So, it was decided to look for it later or will perform the validation manually. +* Next Step: Work on JSON report generation. + +## Licensing profile use cases +The Licensing profile describes the aspects of licensing for the software application under three categories (sub-directories) - Licensing, SimpleLicensing, and ExpandedLicensing. + +The Licensing category describes information about declared licenses and concluded (detected) licenses. The SimpleLicensing category describes information about text-formatted licenses. The ExpandedLicensing category describes information about parseable and machine-readable licenses. + +The [model image](/img/spdx30/model_Licensing.png) for the Licensing profile. + +## Activities + +* In this week, I continued to work on the SPDX v3.0 JSON-ld format report generation for Licensing profile. +* Made a [PR#2750](https://github.com/fossology/fossology/pull/2750) for Generation of SPDX v3.0 report in JSON format. \ No newline at end of file diff --git a/docs/2024/spdx30/updates/2024-06-11.md b/docs/2024/spdx30/updates/2024-06-11.md new file mode 100644 index 000000000..8f5a9f160 --- /dev/null +++ b/docs/2024/spdx30/updates/2024-06-11.md @@ -0,0 +1,33 @@ +--- +title: Week 3 +author: Abhishek Kumar +tags: [gsoc24, spdx30] +--- + + +# Week 3 Meeting and Activities + +_(June 11, 2024)_ + +## Attendees: + +* [Abhishek Kumar](https://github.com/abhi-kumar17871) +* [Gaurav Mishra](https://github.com/GMishx) +* [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) +* [Dearsh Oberoi](https://github.com/deo002) +* [Akash Sah](https://github.com/Akashsah2003) + +## Discussions + +* Presented the JSON-ld report. +* There was a build issue in my latest PR. I started to look into it as well. +* Next Step: Work on RDF report generation. + +## Activities + +* In this week, I worked upon SPDX v3.0 report in JSON format. +* Followed the [JSON Serialization](https://github.com/spdx/spdx-3-model/blob/main/serialization/json.md) for report generation in JSON format. diff --git a/docs/2024/spdx30/updates/2024-06-18.md b/docs/2024/spdx30/updates/2024-06-18.md new file mode 100644 index 000000000..433c72552 --- /dev/null +++ b/docs/2024/spdx30/updates/2024-06-18.md @@ -0,0 +1,36 @@ +--- +title: Week 4 +author: Abhishek Kumar +tags: [gsoc24, spdx30] +--- + + +# Week 4 Meeting and Activities + +_(June 18, 2024)_ + +## Attendees: + +* [Abhishek Kumar](https://github.com/abhi-kumar17871) +* [Gaurav Mishra](https://github.com/GMishx) +* [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) +* [Dearsh Oberoi](https://github.com/deo002) +* [Akash Sah](https://github.com/Akashsah2003) + +## Discussions + +* Have completed generation of SPDX v3.0 report in JSON format. +* Also discussed to the mentor about [validation of JSON-ld using JSON Schema, and SHACL Model](https://github.com/spdx/spdx-3-model/blob/main/serialization/json_ld/validation.md). +* We decided to follow the JSON Schema and SHACL Model for initial phase of validation. +* Next Step: Work on the validation of JSON-ld documents. + +## Activities + +* In this week, I worked upon the RDF report generation. +* I followed the [ontology](https://spdx.org/rdf/3.0.0/spdx-model.ttl) present in the SPDX-3 model github repository. +* I also started looking into the fields which were absent in the previous formats and have rectified them using the JSON schema. + diff --git a/docs/2024/spdx30/updates/2024-06-25.md b/docs/2024/spdx30/updates/2024-06-25.md new file mode 100644 index 000000000..67323d582 --- /dev/null +++ b/docs/2024/spdx30/updates/2024-06-25.md @@ -0,0 +1,36 @@ +--- +title: Week 5 +author: Abhishek Kumar +tags: [gsoc24, spdx30] +--- + + +# Week 5 Meeting and Activities + +_(June 25, 2024)_ + +## Attendees: + +* [Abhishek Kumar](https://github.com/abhi-kumar17871) +* [Gaurav Mishra](https://github.com/GMishx) +* [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd) +* [Dearsh Oberoi](https://github.com/deo002) +* [Akash Sah](https://github.com/Akashsah2003) + +## Discussions + +* Presented my work SPDX v3.0 report in JSON and RDF format and also validated the reports using. +* Started resolving the issue while building the package in GitHub workflow. +* Next Step: Resolve the build issue and work on tag:value and CSV report generation. + +## Activities + +* In this week, I have validated the SPDX v3.0 report in JSON-ld, JSON and RDF format. +* Also add the various fields which were required in Core, Software and Licensing profiles by the organization. +* I have started working for the SPDX v3.0 report generation in tag:value format. + + diff --git a/static/img/spdx30/model_Core+Software.png b/static/img/spdx30/model_Core+Software.png new file mode 100644 index 000000000..30163a5f1 Binary files /dev/null and b/static/img/spdx30/model_Core+Software.png differ diff --git a/static/img/spdx30/model_Licensing.png b/static/img/spdx30/model_Licensing.png new file mode 100644 index 000000000..4e0590f48 Binary files /dev/null and b/static/img/spdx30/model_Licensing.png differ