Sample Data for posts, posts_votes, and posts_attachments #3278
Conversation
WalkthroughThis pull request introduces several new JSON sample data files, including those for posts, comments, votes (for both posts and comments), post attachments, and a binary file detection function. The data import utility has been enhanced to process these new data types, including handling file uploads for post attachments via a MinIO bucket. Additionally, the TypeScript ignore checking script has been updated to skip processing non-text files. Changes
Sequence Diagram(s)sequenceDiagram
participant Loader as loadSampleData.ts
participant DB as Database
participant MinIO as MinIO Service
Loader->>DB: Insert new record (e.g., post attachment)
alt Data type is "post_attachments"
loop For each attachment
Loader->>Loader: Construct file path & read file content
Loader->>MinIO: Upload file to "talawa" bucket
alt Upload fails
MinIO-->>Loader: Error response
Loader->>Loader: Log error
else Upload succeeds
MinIO-->>Loader: Success confirmation
end
end
end
sequenceDiagram
participant Checker as detect_ts_ignore.py
participant FS as FileSystem
Checker->>FS: Read file (obtain extension)
Checker->>Checker: is_binary_file(filepath) check
alt File is binary
Checker-->>Checker: Skip file
else File is not binary
Checker->>FS: Read file content
Checker->>Checker: Scan for "@ts-ignore" comments
end
Possibly related PRs
Suggested labels
Suggested reviewers
📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (1)
🧰 Additional context used🪛 GitHub Actions: Pull request workflow.github/workflows/scripts/detect_ts_ignore.py[error] 1-1: Unauthorized changes were made in the following files: .github/workflows/scripts/detect_ts_ignore.py. To override this, apply the 'ignore-sensitive-files-pr' label. 🔇 Additional comments (3)
✨ Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
Our Pull Request Approval ProcessThanks for contributing! Testing Your CodeRemember, your PRs won't be reviewed until these criteria are met:
Our policies make our code better. ReviewersDo not assign reviewers. Our Queue Monitors will review your PR and assign them.
Reviewing Your CodeYour reviewer(s) will have the following roles:
CONTRIBUTING.mdRead our CONTRIBUTING.md file. Most importantly:
Other
|
…ation/talawa-api into sample_data_posts
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (5)
sample_data/images/01JMQ3EABXAD1F3KTW1NQGF99C.jpegis excluded by!**/*.jpegsample_data/images/01JMQ3F1B55K272B25V19781D0.jpegis excluded by!**/*.jpegsample_data/images/01JMQ3F9E5C806CN6M8EV31E01.jpegis excluded by!**/*.jpegsample_data/images/01JMQ3FGBEDRB9AM6XP9C97RF8.jpegis excluded by!**/*.jpegsample_data/images/01JMQ3FPS789Z1J6BBQ01YDHVQ.jpegis excluded by!**/*.jpeg
📒 Files selected for processing (6)
sample_data/comment_votes.json(1 hunks)sample_data/comments.json(1 hunks)sample_data/post_attachements.json(1 hunks)sample_data/post_votes.json(1 hunks)sample_data/posts.json(1 hunks)src/utilities/loadSampleData.ts(4 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: Analyse Code With CodeQL (typescript)
🔇 Additional comments (4)
src/utilities/loadSampleData.ts (1)
16-22:❓ Verification inconclusive
Verify MinioClient configuration.
The MinioClient configuration relies on environment variables. Ensure these variables are documented in the project's README or .env.example file.
🏁 Script executed:
#!/bin/bash # Check if MinIO environment variables are documented rg -A 1 "API_MINIO_(ACCESS_KEY|END_POINT|PORT|SECRET_KEY|USE_SSL)" README.md .env.exampleLength of output: 144
Manual Verification Needed for MinioClient Environment Documentation
It appears that the environment variables (API_MINIO_ACCESS_KEY, API_MINIO_END_POINT, API_MINIO_PORT, API_MINIO_SECRET_KEY, API_MINIO_USE_SSL) used to configure the MinioClient are not documented in the README.md or in a .env.example file. Please manually verify if these variables are documented elsewhere or update the documentation (preferably in the README.md and/or by adding a .env.example file) to ensure proper configuration.
sample_data/post_votes.json (1)
1-58:✅ Verification successful
Verify data integrity.
Need to verify:
- All referenced posts exist
- Vote types are valid ("up_vote" or "down_vote" only)
- No duplicate votes from same user on same post
🏁 Script executed:
#!/bin/bash # Check referenced posts and validate vote data echo "Checking post references and vote data..." jq -r '.[] | [.postId, .creatorId, .type] | @tsv' sample_data/post_votes.json | while IFS=$'\t' read -r postId creatorId type; do # Check if post exists rg -q "$postId" sample_data/posts.json || echo "Missing post: $postId" # Validate vote type if [[ "$type" != "up_vote" && "$type" != "down_vote" ]]; then echo "Invalid vote type: $type" fi done # Check for duplicate votes echo "Checking for duplicate votes..." jq -r 'group_by([.postId, .creatorId]) | map(select(length > 1)) | .[]' sample_data/post_votes.jsonLength of output: 1709
Data Integrity Verified – No Issues Found
- All vote records reference existing posts in
sample_data/posts.json.- Each vote’s type is valid, containing only
"up_vote"or"down_vote".- No duplicate votes were detected for any user on the same post.
sample_data/comments.json (1)
1-59: Well-structured JSON sample data for comments.
The JSON array correctly defines eight comment objects, and each object includes all the expected keys (id,body,createdAt,creatorId,postId) with properly formatted values (e.g., ISO 8601 timestamps). This sample data aligns well with the intended use for testing and development.sample_data/posts.json (1)
1-66: Correct and complete JSON sample data for posts.
The JSON array is properly formed and includes all the expected fields (id,caption,createdAt,creatorId,organizationId) for each post record. The timestamps use the ISO 8601 format and the sample entries are consistent with the schema used in the application.
|
the test if failing as the ts-ignore script tries to read .jpeg file as a text file, will make change to ignore these files in the script |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop-postgres #3278 +/- ##
====================================================
- Coverage 48.26% 48.12% -0.15%
====================================================
Files 455 455
Lines 33937 34036 +99
Branches 908 908
====================================================
Hits 16381 16381
- Misses 17556 17655 +99 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Actionable comments posted: 5
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (3)
.github/workflows/scripts/detect_ts_ignore.py(3 hunks)sample_data/post_attachments.json(1 hunks)src/utilities/loadSampleData.ts(4 hunks)
🧰 Additional context used
🪛 Ruff (0.8.2)
.github/workflows/scripts/detect_ts_ignore.py
42-70: Missing explicit return at the end of function able to return non-None value
Add explicit return statement
(RET503)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: Run tests for talawa api
- GitHub Check: Analyse Code With CodeQL (typescript)
🔇 Additional comments (9)
.github/workflows/scripts/detect_ts_ignore.py (2)
8-8: LGTM!The
osmodule import is correctly placed with other standard library imports and is necessary for file path operations.
20-29: LGTM!The
is_binary_filefunction is well-documented and efficiently implemented. The use oflower()ensures case-insensitive extension matching.src/utilities/loadSampleData.ts (7)
8-8: Looks good importing Minio client.
No issues found with importing Minio client.
204-213: No issues inserting posts.
The insertion logic for "posts" is straightforward and aligns with the existing pattern.
214-223: Post votes insertion looks fine.
Data mapping and insertion for "post_votes" follows the same pattern and appears correct.
259-268: Comments insertion follows the pattern correctly.
The function for inserting "comments" data is consistent and looks good.
269-277: Comment votes insertion is consistent.
The "comment_votes" data insertion mirrors the approach used for the other inserts. No issues found.
313-327: Checking record counts for new tables is correct.
The additions tocheckCountAfterImportfor the new tables ensure coverage for "posts", "post_votes", "post_attachments", "comments", and "comment_votes".
359-368: Collections array expanded successfully.
All relevant tables (posts, post_votes, post_attachments, comments, comment_votes) have been properly added to thecollectionsarray.
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (2)
.github/workflows/scripts/detect_ts_ignore.py(3 hunks)src/utilities/loadSampleData.ts(4 hunks)
🧰 Additional context used
🪛 Ruff (0.8.2)
.github/workflows/scripts/detect_ts_ignore.py
81-81: Trailing comma missing
Add trailing comma
(COM812)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: Run tests for talawa api
- GitHub Check: Analyse Code With CodeQL (typescript)
🔇 Additional comments (9)
src/utilities/loadSampleData.ts (5)
8-8: Good import of Minio client.Importing the Minio client is properly done here. No issues found.
204-213: Looks good for handling post data.The insertion of posts data with date parsing is clean and consistent with the existing approach for other entities.
214-223: Posts votes import logic is straightforward.No issues found. The pattern for parsing and inserting post votes matches the rest of the file.
261-279: Consistent approach for comments and comment votes.Mapping and inserting comments and comment votes follow the same pattern as other tables, which is good for maintainability.
324-328: Record count checks for new tables are in place.Adding posts, post_votes, post_attachments, comments, and comment_votes to the check is an excellent integration to verify data presence after import.
.github/workflows/scripts/detect_ts_ignore.py (4)
8-8: Import of os to support file checks is appropriate.No concerns with this import usage since it's needed for file extension checks.
46-55: Clean helper function for binary file detection.This function is straightforward and effectively relies on file extension checks. Overall, it's well-structured.
70-88: Properly skipping binary files before scanning.The condition to bypass files in
IGNORED_EXTENSIONSis correct and prevents reading binary files as text. The logic for printing errors upon encountering@ts-ignorereferences is consistent and clear.🧰 Tools
🪛 Ruff (0.8.2)
81-81: Trailing comma missing
Add trailing comma
(COM812)
18-43: 🧹 Nitpick (assertive)Enhance style by adding a trailing comma.
The defined ignored extensions set is comprehensive and well-documented. To satisfy some linters or style conventions, consider adding a trailing comma at the end for consistency.
IGNRORED_EXTENSIONS = { ... ".docx", -} +},Likely an incorrect or invalid review comment.
|
there are no tests written for |
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (1)
src/utilities/loadSampleData.ts (1)
16-22: 🧹 Nitpick (assertive)Validate environment variables or log warnings when values are empty.
Using fallback to empty strings is acceptable, but logging a warning or throwing an error if credentials are undefined can help diagnose misconfiguration at runtime more rapidly.
const minioClient = new MinioClient({ - accessKey: process.env.API_MINIO_ACCESS_KEY || "", - endPoint: process.env.API_MINIO_END_POINT || "minio", - port: Number(process.env.API_MINIO_PORT), - secretKey: process.env.API_MINIO_SECRET_KEY || "", - useSSL: process.env.API_MINIO_USE_SSL === "true", + accessKey: process.env.API_MINIO_ACCESS_KEY || (() => { + console.warn("API_MINIO_ACCESS_KEY is undefined"); + return ""; + })(), + endPoint: process.env.API_MINIO_END_POINT || (() => { + console.warn("API_MINIO_END_POINT is undefined. Using default: 'minio'"); + return "minio"; + })(), + ... });🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 16-22: src/utilities/loadSampleData.ts#L16-L22
Added lines #L16 - L22 were not covered by tests
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (2)
.github/workflows/scripts/detect_ts_ignore.py(3 hunks)src/utilities/loadSampleData.ts(4 hunks)
🧰 Additional context used
🪛 Ruff (0.8.2)
.github/workflows/scripts/detect_ts_ignore.py
82-82: Trailing comma missing
Add trailing comma
(COM812)
🪛 GitHub Actions: Pull request workflow
.github/workflows/scripts/detect_ts_ignore.py
[error] 1-1: Unauthorized changes were made in the following files: .github/workflows/scripts/detect_ts_ignore.py. To override this, apply the 'ignore-sensitive-files-pr' label.
🪛 GitHub Check: codecov/patch
src/utilities/loadSampleData.ts
[warning] 8-8: src/utilities/loadSampleData.ts#L8
Added line #L8 was not covered by tests
[warning] 16-22: src/utilities/loadSampleData.ts#L16-L22
Added lines #L16 - L22 were not covered by tests
[warning] 204-280: src/utilities/loadSampleData.ts#L204-L280
Added lines #L204 - L280 were not covered by tests
[warning] 324-328: src/utilities/loadSampleData.ts#L324-L328
Added lines #L324 - L328 were not covered by tests
[warning] 361-370: src/utilities/loadSampleData.ts#L361-L370
Added lines #L361 - L370 were not covered by tests
🔇 Additional comments (5)
src/utilities/loadSampleData.ts (2)
8-8: Prefer adding test coverage for Minio import usage.Although the import for the Minio client is straightforward, static analysis indicates these lines lack test coverage. Consider adding tests (e.g., mocking Minio) to ensure future changes do not break this import or its usage.
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 8-8: src/utilities/loadSampleData.ts#L8
Added line #L8 was not covered by tests
324-328: Good addition for verifying record counts.Thank you for adding the new tables to the record count check. This is helpful for validating data insertion. Consider adding integration tests for these expansions to confirm correctness under various scenarios (e.g. empty/partial data).
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 324-328: src/utilities/loadSampleData.ts#L324-L328
Added lines #L324 - L328 were not covered by tests.github/workflows/scripts/detect_ts_ignore.py (3)
8-8: No issues with importing os.This import enables path/extension checks and is appropriate to support your binary file detection.
18-44: Thanks for including a comprehensive list of common binary extensions.Your set of ignored file extensions now covers a wide range of binary formats, aligning with typical usage patterns.
46-56: Extension-based binary detection logic looks solid.Using the file extension for quick checks is a practical approach to bypass reading large binary files.
|
@prayanshchh You don't need to write vitest for this file, This has been taken care of in new upgraded script, your changes will be merged with that which is already tested. |
palisadoes
merged commit 381c644
into
PalisadoesFoundation:develop-postgres
|
@palisadoes I was updating the branch, was it not needed? |
|
@prayanshchh The script has been upgraded to new version and place, with proper testing, it is under development so your logics are integrated to it, don't worry. |
|
alright then |
What kind of change does this PR introduce?
sample data for posts, post_votes, post_attachments, comments and comment_votes
Issue Number:
#3193
Fixes #
#3193
Snapshots/Videos:
Screencast.from.2025-02-22.21-10-21.webm
If relevant, did you update the documentation?
Summary
Does this PR introduce a breaking change?
Checklist
CodeRabbit AI Review
Test Coverage
Other information
Have you read the contributing guide?
Summary by CodeRabbit