fixed dashboard backend unittest #187
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer's Guide by SourceryThis PR implements several major changes to improve the codebase, including:
Class diagram for AnalyzerReport and related classesclassDiagram
class AnalyzerReport {
+ForeignKey config
+ForeignKey data_model_content_type
+IntegerField data_model_object_id
+GenericForeignKey data_model
+clean()
+get_data_model_class(job)
+data_model_class
+_validation_before_data_model()
+_create_data_model_dictionary()
+create_data_model()
}
class BaseDataModel {
<<interface>>
}
class FileDataModel {
+get_fields()
}
class IPDataModel {
+get_fields()
}
class DomainDataModel {
+get_fields()
}
class ObservableTypes {
+HASH
+IP
+DOMAIN
+URL
}
AnalyzerReport --> BaseDataModel : uses
AnalyzerReport --> ObservableTypes
BaseDataModel <|-- FileDataModel
BaseDataModel <|-- IPDataModel
BaseDataModel <|-- DomainDataModel
Class diagram for Job and AbstractReportclassDiagram
class Job {
+ForeignKey investigation
+CharField status
+retry()
+set_final_status()
+kill_if_ongoing()
+execute()
+user_month_submissions(user)
}
class AbstractReport {
+CharField status
+JSONField report
+ArrayField errors
+user
+process_time()
+get_value(search_from, fields)
}
class Investigation {
+CharField status
+set_correct_status(save)
}
Job --> AbstractReport : contains
Job --> Investigation : references
AbstractReport --> User : references
Investigation --> Job : contains
Class diagram for VisualizableDownload and related classesclassDiagram
class VisualizableObject {
+size
+alignment
+disable
}
class VisualizableDownload {
+String value
+String payload
+String copy_text
+String description
+Boolean add_metadata_in_description
+String link
+String mimetype
+type
+attributes
}
VisualizableObject <|-- VisualizableDownload
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
Here's the code health analysis summary for commits Analysis Summary
|
CI Failure Feedback 🧐(Checks updated until commit 40e1146)
✨ CI feedback usage guide:The CI feedback tool (
In addition to being automatically triggered, the tool can also be invoked manually by commenting on a PR: where Configuration options
See more information about the |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey @gitworkflows - I've reviewed your changes and they look great!
Here's what I looked at during the review
- 🟡 General issues: 1 issue found
- 🟡 Security: 1 issue found
- 🟡 Testing: 1 issue found
- 🟡 Complexity: 1 issue found
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| class IETFReportSerializer(FlexFieldsModelSerializer): | ||
| class Meta: | ||
| model = IETFReport | ||
| fields = "__all__" |
There was a problem hiding this comment.
🚨 issue (security): Explicitly specify serializer fields instead of using all to prevent exposing sensitive data
Using all could accidentally expose sensitive fields. List specific fields that should be exposed in the API.
| def perform_request_to_form(self, form) -> Response: | ||
| params, dest_url = self.compile_form_field(form) | ||
| logger.info(f"Job #{self.job_id}: Sending {params=} to submit url {dest_url}") | ||
| return requests.post( |
There was a problem hiding this comment.
issue: Add error handling for form submission timeouts and connection errors
The form submission should handle timeouts and connection errors gracefully with appropriate error messages.
Comment on lines
+37
to
+38
| result = ser.data | ||
| print(result) |
There was a problem hiding this comment.
issue (testing): Test case prints result without assertions
The test prints the serialized result but doesn't make any assertions about its content. Add assertions to verify the expected structure and values in the serialized output.
| return f"{self.provider}: {json.dumps(self.signature)}" | ||
|
|
||
|
|
||
| class BaseDataModel(models.Model): |
There was a problem hiding this comment.
issue (complexity): Consider extracting analyzer-specific data from BaseDataModel into a separate AnalyzerResult model
The BaseDataModel class is handling too many responsibilities by storing analyzer-specific data directly. This makes the model harder to maintain and understand. Consider extracting analyzer-specific data into dedicated models:
class AnalyzerResult(models.Model):
data_model = models.ForeignKey(BaseDataModel, on_delete=models.CASCADE)
analyzer_name = models.CharField(max_length=100)
evaluation = models.CharField(max_length=100, null=True)
external_references = SetField(models.URLField(), blank=True)
specific_data = models.JSONField()
class Meta:
unique_together = ['data_model', 'analyzer_name']
class BaseDataModel(models.Model):
# Core fields that apply to all types
tags = SetField(LowercaseCharField(max_length=100), blank=True)
malware_family = LowercaseCharField(max_length=100, null=True)
date = models.DateTimeField(default=now)
# Move analyzer-specific data to AnalyzerResult
@property
def get_analyzer_result(self, analyzer_name):
return self.analyzerresult_set.get(analyzer_name=analyzer_name)This approach:
- Separates generic and analyzer-specific concerns
- Makes it easier to add new analyzers without modifying BaseDataModel
- Provides better type safety than generic JSON fields
- Maintains backwards compatibility through properties
| jobFileSubTypesPerDay?.aggregation, | ||
| componentsFn: (respData) => { | ||
| const { values: mimetypeList } = respData; | ||
| if (!mimetypeList || !mimetypeList?.length) return null; |
There was a problem hiding this comment.
suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)
Suggested change
| if (!mimetypeList || !mimetypeList?.length) return null; | |
| if (!mimetypeList || !mimetypeList?.length) { |
Explanation
It is recommended to always use braces and create explicit statement blocks.Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).
|
|
||
| def run(self) -> dict: | ||
| result: {} = {} | ||
| if not ( |
There was a problem hiding this comment.
issue (code-quality): We've found these issues:
- Convert for loop into list comprehension (
list-comprehension) - Use f-string instead of string concatenation (
use-fstring-for-concatenation)
Comment on lines
+464
to
+466
| signatures = data_model.signatures.count() | ||
|
|
||
| if signatures: |
There was a problem hiding this comment.
suggestion (code-quality): Use named expression to simplify assignment and conditional (use-named-expression)
Suggested change
| signatures = data_model.signatures.count() | |
| if signatures: | |
| if signatures := data_model.signatures.count(): |
Comment on lines
+69
to
+72
| if ( | ||
| job.observable_classification == ObservableTypes.DOMAIN.value | ||
| or job.observable_classification == ObservableTypes.URL.value | ||
| ): |
There was a problem hiding this comment.
suggestion (code-quality): Replace multiple comparisons of same variable with in operator (merge-comparisons)
Suggested change
| if ( | |
| job.observable_classification == ObservableTypes.DOMAIN.value | |
| or job.observable_classification == ObservableTypes.URL.value | |
| ): | |
| if job.observable_classification in [ | |
| ObservableTypes.DOMAIN.value, | |
| ObservableTypes.URL.value, | |
| ]: |
| return self.get_data_model_class(self.job) | ||
|
|
||
| def _validation_before_data_model(self) -> bool: | ||
| if not self.status == self.STATUSES.SUCCESS.value: |
There was a problem hiding this comment.
suggestion (code-quality): Simplify logical expression using De Morgan identities (de-morgan)
Suggested change
| if not self.status == self.STATUSES.SUCCESS.value: | |
| if self.status != self.STATUSES.SUCCESS.value: |
Comment on lines
+236
to
+237
| org = self.report.report.get("autonomous_system_organization", None) | ||
| if org: |
There was a problem hiding this comment.
suggestion (code-quality): Use named expression to simplify assignment and conditional (use-named-expression)
Suggested change
| org = self.report.report.get("autonomous_system_organization", None) | |
| if org: | |
| if org := self.report.report.get("autonomous_system_organization", None): |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
🎉 Snyk checks have passed. No issues have been found so far.✅ security/snyk check is complete. No issues have been found. (View Details) |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| upper_case=True, | ||
| lower_case=True, | ||
| ) | ||
| logger.info(f"Generated fake password input {self.FAKE_PASSWORD_INPUT}") |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI 2 months ago
To fix the problem, we should avoid logging sensitive information such as passwords, email addresses, and phone numbers. Instead, we can log a generic message indicating that the fake data has been generated without including the actual values. This way, we maintain the functionality of the code without exposing sensitive information.
- Replace the logging statements that include sensitive information with generic messages.
- Specifically, update lines 84-97 to remove the sensitive data from the log messages.
Suggested changeset
1
api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
| @@ -82,7 +82,5 @@ | ||
| } | ||
| logger.info( | ||
| f"Generated name text input mapping {self._name_text_input_mapping}" | ||
| ) | ||
| logger.info("Generated name text input mapping.") | ||
| self.FAKE_EMAIL_INPUT: str = fake.email() | ||
| logger.info(f"Generated fake email input {self.FAKE_EMAIL_INPUT}") | ||
| logger.info("Generated fake email input.") | ||
| self.FAKE_PASSWORD_INPUT: str = fake.password( | ||
| @@ -94,5 +92,5 @@ | ||
| ) | ||
| logger.info(f"Generated fake password input {self.FAKE_PASSWORD_INPUT}") | ||
| logger.info("Generated fake password input.") | ||
| self.FAKE_TEL_INPUT: str = fake.phone_number() | ||
| logger.info(f"Generated fake tel input {self.FAKE_TEL_INPUT}") | ||
| logger.info("Generated fake tel input.") | ||
|
|
Copilot is powered by AI and may make mistakes. Always verify output.
| ) | ||
|
|
||
| logger.info( | ||
| f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}" |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI 2 months ago
To fix the problem, we need to ensure that sensitive information is not logged. This can be achieved by either removing the logging statements that include sensitive data or by sanitizing the data before logging it. In this case, we will sanitize the data by replacing sensitive values with a placeholder text like "[REDACTED]".
We will modify the logging statements on lines 150, 164, 168, and 176 to ensure that sensitive information is not logged. Specifically, we will replace the actual values of sensitive data with "[REDACTED]".
Suggested changeset
1
api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
| @@ -150,3 +150,3 @@ | ||
| logger.info( | ||
| f"Job #{self.job_id}: Found hidden input tag with {input_name=} and {input_value=}" | ||
| f"Job #{self.job_id}: Found hidden input tag with {input_name=} and input_value=[REDACTED]" | ||
| ) | ||
| @@ -168,3 +168,3 @@ | ||
| logger.info( | ||
| f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}" | ||
| f"Job #{self.job_id}: Sending value [REDACTED] for {input_name=}" | ||
| ) | ||
| @@ -175,3 +175,3 @@ | ||
| params, dest_url = self.compile_form_field(form) | ||
| logger.info(f"Job #{self.job_id}: Sending {params=} to submit url {dest_url}") | ||
| logger.info(f"Job #{self.job_id}: Sending params=[REDACTED] to submit url {dest_url}") | ||
| return requests.post( |
Copilot is powered by AI and may make mistakes. Always verify output.
Signed-off-by: gitworkflows <118260833+gitworkflows@users.noreply.github.com>
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 13180230 | Triggered | Username Password | 983802d | tests/auth/test_auth.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User description
(Please add to the PR name the issue/s that this PR would close if merged by using a Github keyword. Example:
<feature name>. Closes #999. If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)Description
Please include a summary of the change and link to the related issue.
Type of change
Please delete options that are not relevant.
Checklist
developdumpplugincommand and added it in the project as a data migration. ("How to share a plugin with the community")test_files.zipand you added the default tests for that mimetype in test_classes.py.FREE_TO_USE_ANALYZERSplaybook by following this guide.urlthat contains this information. This is required for Health Checks._monkeypatch()was used in its class to apply the necessary decorators.MockUpResponseof the_monkeypatch()method. This serves us to provide a valid sample for testing.Black,Flake,Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.testsfolder). All the tests (new and old ones) gave 0 errors.DeepSource,Django Doctorsor other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.Important Rules
PR Type
enhancement, tests, configuration changes, documentation
Description
PhishingFormCompiler,UrlDNA,Crowdsec,GreyNoiseIntel, andVirusTotalv3SampleDownload.PhishingExtractor,UrlDNA_New_Scan, andAnalyzerReport.Changes walkthrough 📝
27 files
test_views.py
Enhance and update test cases for API viewstests/api_app/test_views.py
MagicMock,patch, andZoneInfo.TLPs.
patchto mock datetime and function calls in tests.test_tasks.py
Update task tests with user and organization contexttests/threat_matrix/test_tasks.py
MembershipandOrganization.assertions.
__init__.py
Updated test cases for user authentication and status handling.tests/init.py
STATUSESfor status checks.test_models.py
Added test cases for `AnalyzerReport` data model methods.tests/api_app/analyzers_manager/test_models.py
AnalyzerReportdata model methods.test_nvd_cve.py
Enhanced NVD CVE tests with mock data and validation.tests/api_app/analyzers_manager/observable_analyzers/test_nvd_cve.py
test_websocket.py
Refactored websocket tests for job status handling.tests/api_app/test_websocket.py
STATUSESfor job status.test_views.py
Added test cases for data model view access and retrieval.tests/api_app/data_model_manager/test_views.py
DomainDataModel,IPDataModel, andFileDataModelviews.
test_analyzer_extractor.py
Refactored passive DNS tests for job status handling.tests/api_app/visualizers_manager/passive_dns/test_analyzer_extractor.py
STATUSESfor job status.test_classes.py
Added test cases for `VisualizableDownload` serialization.tests/api_app/visualizers_manager/test_classes.py
VisualizableDownloadclass.test_classes.py
Updated test data and cases for observable types.tests/api_app/analyzers_manager/test_classes.py
test_views.py
Updated pivot view tests for access control and restrictions.tests/api_app/pivots_manager/test_views.py
test_serializers.py
Update serializer tests with new domain and status checkstests/api_app/test_serializers.py
khulnasoft.comdomain.analyzers_data_model.STATUSES.test_serializers.py
Add test for DomainDataModelSerializer representationtests/api_app/data_model_manager/test_serializers.py
DomainDataModelSerializer.test_crons.py
Update cron tests with new job status referencestests/test_crons.py
STATUSES.test_classes.py
Update test classes with new job status referencestests/api_app/test_classes.py
STATUSES.test_classes.py
Update connector test classes with new domain and statustests/api_app/connectors_manager/test_classes.py
khulnasoft.comdomain.STATUSES.test_mixins.py
Clean up test mixins and update setuptests/api_app/test_mixins.py
IngestorConfig.VirusTotalMixinTestCase.test_models.py
Update investigation model tests with new status referencestests/api_app/investigations_manager/test_models.py
STATUSES.test_auth.py
Update auth tests with new password referencestests/auth/test_auth.py
threatmatrix.test_views.py
Update connector view tests with new status referencestests/api_app/connectors_manager/test_views.py
STATUSES.test_views.py
Update analyzer view tests with new status referencestests/api_app/analyzers_manager/test_views.py
STATUSES.test_models.py
Update model tests with new status referencestests/api_app/test_models.py
STATUSES.test_models.py
Add test for IPDataModel serializationtests/api_app/data_model_manager/test_models.py
IPDataModel.test_views.py
Add superuser access test for investigation viewstests/api_app/investigations_manager/test_views.py
validators.test.js
Add tests for visualizer data validation componentsfrontend/tests/components/jobs/result/visualizer/validators.test.js
files.test.js
Add tests for file utility functionsfrontend/tests/utils/files.test.js
humanReadbleSizefunction.charts.test.jsx
Add tests for dashboard chart componentsfrontend/tests/components/dashboard/charts.test.jsx
6 files
0128_analyzer_config_phishing_form_compiler.py
Add migration for Phishing Form Compiler analyzerapi_app/analyzers_manager/migrations/0128_analyzer_config_phishing_form_compiler.py
Phishing_Form_Compileranalyzer configuration.0132_analyzer_config_urldna_new_scan.py
Add migration for UrlDNA New Scan analyzerapi_app/analyzers_manager/migrations/0132_analyzer_config_urldna_new_scan.py
UrlDNA_New_Scananalyzer configuration.0001_initial.py
Initial migration for data model manager modelsapi_app/data_model_manager/migrations/0001_initial.py
IETFReport,Signature,IPDataModel,FileDataModel,and
DomainDataModel.0133_analyzer_config_urldna_search.py
Add migration for UrlDNA Search analyzerapi_app/analyzers_manager/migrations/0133_analyzer_config_urldna_search.py
UrlDNA_Searchanalyzer configuration.0129_analyzer_config_phishing_extractor.py
Add migration for Phishing Extractor analyzerapi_app/analyzers_manager/migrations/0129_analyzer_config_phishing_extractor.py
Phishing_Extractoranalyzer configuration.0136_alter_analyzerconfig_mapping_data_model_and_more.py
Alter AnalyzerConfig for data model mapping and file typesapi_app/analyzers_manager/migrations/0136_alter_analyzerconfig_mapping_data_model_and_more.py
AnalyzerConfigfields for mapping data model and file types.65 files
phishing_form_compiler.py
Implement Phishing Form Compiler analyzerapi_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
PhishingFormCompilerclass for phishing form analysis.Fakerfor generating fake data for form inputs.models.py
Add data models for IETF, Signature, and base dataapi_app/data_model_manager/models.py
IETFReport,Signature, and base data models.views.py
Update views for job status and aggregation enhancementsapi_app/views.py
STATUSESenum.aggregations.
plugin_report_querieswith Elasticsearch query filters.tasks.py
Update task status handling and loggingthreat_matrix/tasks.py
STATUSESenum.conversion.
models.py
Enhanced `AnalyzerReport` with data model integration and validation.api_app/analyzers_manager/models.py
GenericForeignKey,ContentType,ArrayField, andForeignKey.data_model_content_type,data_model_object_id, anddata_modelfields toAnalyzerReport.in
AnalyzerReport.MimeTypes.calculateto handlebufferasUnion[bytes, str].mapping_data_modelfield toAnalyzerConfig.models.py
Refactored job status handling to use `STATUSES`.api_app/models.py
StatustoSTATUSESfor consistency.STATUSESfor job status management.driver_wrapper.py
Added `DriverWrapper` class for Selenium WebDriver management.integrations/phishing_analyzers/analyzers/driver_wrapper.py
DriverWrapperclass for managing Selenium WebDriver.job.py
Enhanced job serializer with data model and status updates.api_app/serializers/job.py
STATUSES.analyzers_data_modelfield toJobSerializer.MimeTypes.calculateusage for file mimetype calculation.urldna.py
Added `UrlDNA` analyzer for URL analysis and scanning.api_app/analyzers_manager/observable_analyzers/urldna.py
UrlDNAclass for URL analysis.seleniumwire_request_serializer.py
Added serialization for Selenium Wire requests.integrations/phishing_analyzers/analyzers/seleniumwire_request_serializer.py
requests.
classes.py
Enhanced `BaseAnalyzerMixin` with data model creation and evaluation.api_app/analyzers_manager/classes.py
BaseAnalyzerMixin.after_run_successto handle data model creation.crowdsec.py
Enhanced `Crowdsec` analyzer with data model creation logic.api_app/analyzers_manager/observable_analyzers/crowdsec.py
Crowdsecanalyzer.extract_phishing_site.py
Added script for extracting and analyzing phishing sites.integrations/phishing_analyzers/analyzers/extract_phishing_site.py
extract_phishing_sitescript for analyzing phishing sites.sample_download.py
Added `SampleDownload` visualizer for handling file downloads.api_app/visualizers_manager/visualizers/sample_download.py
SampleDownloadvisualizer for file downloads.queryset.py
Refactored querysets to use `STATUSES` for status filtering.api_app/queryset.py
STATUSESfor filtering job and reportstatuses.
_alias_for_test.mixins.py
Refactored VirusTotal mixins for analyzer integration.api_app/mixins.py
BaseAnalyzerMixininheritance fromVirusTotalv3BaseMixin.VirusTotalv3AnalyzerMixinto includeBaseAnalyzerMixin.classes.py
Refactored report status handling and error logging.api_app/classes.py
STATUSES.after_run_failed.greynoiseintel.py
Enhanced `GreyNoiseIntel` analyzer with data model logic.api_app/analyzers_manager/observable_analyzers/greynoiseintel.py
GreyNoiseIntelanalyzer.admin.py
Added admin views for data models with custom display.api_app/data_model_manager/admin.py
DomainDataModel,FileDataModel, andIPDataModel.phishing_extractor.py
Added `PhishingExtractor` class for phishing analysis.api_app/analyzers_manager/observable_analyzers/phishing/phishing_extractor.py
PhishingExtractorclass for phishing analysis.classes.py
Added `VisualizableDownload` class for file download visualization.api_app/visualizers_manager/classes.py
VisualizableDownloadclass for handling file downloads.elastic.py
Refactored Elastic serializers for request and response handling.api_app/serializers/elastic.py
ElasticJobSerializerandElasticConfigSerializer.serializers.py
Added serializers for data models with related fields.api_app/data_model_manager/serializers.py
DomainDataModel,IPDataModel, andFileDataModel.yara_scan.py
Enhanced Yara scan with data model creation and signature handling.api_app/analyzers_manager/file_analyzers/yara_scan.py
0056_download_sample_vt.py
Add migration for VirusTotal sample download playbookapi_app/playbooks_manager/migrations/0056_download_sample_vt.py
PlaybookConfigandAnalyzerConfig.app.py
Set up Flask app for phishing analyzers with loggingintegrations/phishing_analyzers/app.py
virus_total.py
Improve VirusTotal ingestor configuration and loggingapi_app/ingestors_manager/ingestors/virus_total.py
configmethod to setforce_active_scanto False.runmethod.opencti.py
Simplify observable type mapping in OpenCTI connectorapi_app/connectors_manager/connectors/opencti.py
0131_analyzer_config_vt_sample_download.py
Add migration for VirusTotal sample download analyzerapi_app/analyzers_manager/migrations/0131_analyzer_config_vt_sample_download.py
AnalyzerConfigto support VirusTotal sampledownload.
models.py
Refactor Investigation model status handlingapi_app/investigations_manager/models.py
StatustoSTATUSESfor consistency.set_correct_status.maxmind.py
Enhance MaxMind analyzer with data model updatesapi_app/analyzers_manager/observable_analyzers/maxmind.py
_update_data_modelmethod to update evaluation based onorganization.
abuseipdb.py
Add data model updates to AbuseIPDB analyzerapi_app/analyzers_manager/observable_analyzers/abuseipdb.py
_update_data_modelmethod to set evaluation based on reportdata.
0002_domaindatamodel_resolutions_and_more.py
Update DomainDataModel with resolutions and IETF reportapi_app/data_model_manager/migrations/0002_domaindatamodel_resolutions_and_more.py
resolutionsfield toDomainDataModel.ietf_reportfield to useManyToManyField.0137_analyzerreport_data_model_content_type_and_more.py
Add data model fields to AnalyzerReportapi_app/analyzers_manager/migrations/0137_analyzerreport_data_model_content_type_and_more.py
data_model_content_typeanddata_model_object_id.AnalyzerReport.vt3_sample_download.py
Implement VirusTotal v3 sample download analyzerapi_app/analyzers_manager/observable_analyzers/vt/vt3_sample_download.py
VirusTotalv3SampleDownloadanalyzer class.dns.py
Update DNS visualizer with new status referencesapi_app/visualizers_manager/visualizers/dns.py
STATUSES.0138_alter_analyzerreport_data_model_content_type.py
Alter data model content type field in AnalyzerReportapi_app/analyzers_manager/migrations/0138_alter_analyzerreport_data_model_content_type.py
data_model_content_typefield to limit choices todata_model_manager.AnalyzerReport.views.py
Add viewsets for data models with paginationapi_app/data_model_manager/views.py
DomainDataModel,IPDataModel, andFileDataModel.talos.py
Enhance Talos analyzer with data model updatesapi_app/analyzers_manager/observable_analyzers/talos.py
_do_create_data_modeland_update_data_modelmethods.urls.py
Add URL routing for data model viewsetsapi_app/data_model_manager/urls.py
nvd_cve.py
Simplify CVE format validation in NVD CVE analyzerapi_app/analyzers_manager/observable_analyzers/nvd_cve.py
settings.enums.py
Define enums for data model attributesapi_app/data_model_manager/enums.py
queryset.py
Add method to retrieve data models in querysetapi_app/analyzers_manager/queryset.py
get_data_modelsmethod toAnalyzerReportQuerySet.0003_remove_ipdatamodel_ietf_report_and_more.py
Update IPDataModel with IETF report relationapi_app/data_model_manager/migrations/0003_remove_ipdatamodel_ietf_report_and_more.py
ietf_reportfield fromIPDataModel.ietf_reportas aManyToManyFieldinIPDataModel.intelx.py
Simplify session header update in IntelX analyzerapi_app/analyzers_manager/observable_analyzers/intelx.py
0139_alter_analyzerconfig_mapping_data_model.py
Alter mapping data model field in AnalyzerConfigapi_app/analyzers_manager/migrations/0139_alter_analyzerconfig_mapping_data_model.py
mapping_data_modelfield inAnalyzerConfig.tor.py
Enhance Tor analyzer with conditional data model creationapi_app/analyzers_manager/observable_analyzers/tor.py
_do_create_data_modelmethod with condition.urls.py
Update API URLs with data model routingapi_app/urls.py
plugin_report_queriespath.data_modelURLs in main routing.0134_analyzerconfig_mapping_data_model.py
Add mapping data model field to AnalyzerConfigapi_app/analyzers_manager/migrations/0134_analyzerconfig_mapping_data_model.py
mapping_data_modelfield toAnalyzerConfig.urlscan.py
Simplify header initialization in URLScan analyzerapi_app/analyzers_manager/observable_analyzers/urlscan.py
runmethod.load_file_same_playbook.py
Implement LoadFileSamePlaybook pivot classapi_app/pivots_manager/pivots/load_file_same_playbook.py
LoadFileSamePlaybookclass extendingLoadFile.compare.py
Enhance Compare pivot with error handlingapi_app/pivots_manager/pivots/compare.py
should_runmethod.fields.py
Add custom fields for data model managementapi_app/data_model_manager/fields.py
SetFieldandLowercaseCharFieldcustom fields.any_compare.py
Improve AnyCompare pivot with enhanced value handlingapi_app/pivots_manager/pivots/any_compare.py
should_runmethod to improve value retrieval.urlhaus.py
Enhance URLHaus analyzer with conditional data model creationapi_app/analyzers_manager/observable_analyzers/urlhaus.py
_do_create_data_modelmethod with condition.pe_info.py
Update PE Info analyzer with new status referencesapi_app/analyzers_manager/file_analyzers/pe_info.py
STATUSES.elf_info.py
Update ELF Info analyzer with new status referencesapi_app/analyzers_manager/file_analyzers/elf_info.py
STATUSES.classes.py
Update pivot classes with new status referencesapi_app/pivots_manager/classes.py
STATUSES.yara.py
Update Yara visualizer with new status referencesapi_app/visualizers_manager/visualizers/yara.py
STATUSES.queryset.py
Add BaseDataModelQuerySet with serialization methodapi_app/data_model_manager/queryset.py
BaseDataModelQuerySetclass withserializemethod.validators.js
Enhance visualizer validators with mimetype parsingfrontend/src/components/jobs/result/visualizer/validators.js
parseMimetypefunction for mimetype validation.parseElementFieldsto handle download components.apiURLs.js
Refactor and add new aggregate API URLsfrontend/src/constants/apiURLs.js
files.js
Add file utility functions for download and size formattingfrontend/src/utils/files.js
fileDownloadandhumanReadbleSizeutility functions.const.js
Extend visualizer component types with downloadfrontend/src/components/jobs/result/visualizer/elements/const.js
DOWNLOADtoVisualizerComponentType.charts.jsx
Implement dashboard chart components with rechartsfrontend/src/components/dashboard/charts.jsx
recharts.15 files
0035_pivot_config_phishingextractortoanalysis.py
Added migration forPhishingExtractorToAnalysispivot configuration.api_app/pivots_manager/migrations/0035_pivot_config_phishingextractortoanalysis.py
PhishingExtractorToAnalysispivot configuration.0055_playbook_config_phishingextractor.py
Added migration for `PhishingExtractor` playbook configuration.api_app/playbooks_manager/migrations/0055_playbook_config_phishingextractor.py
PhishingExtractorplaybook configuration.0054_playbook_config_phishinganalysis.py
Added migration for `PhishingAnalysis` playbook configuration.api_app/playbooks_manager/migrations/0054_playbook_config_phishinganalysis.py
PhishingAnalysisplaybook configuration.0005_alter_domaindatamodel_external_references_and_more.py
Altered data model fields for domain, file, and IP models.api_app/data_model_manager/migrations/0005_alter_domaindatamodel_external_references_and_more.py
DomainDataModel,FileDataModel, andIPDataModel.SetFieldwith specific base fields.0125_update_yara_repo.py
Updated Yara repository URLs in migration.api_app/analyzers_manager/migrations/0125_update_yara_repo.py
PluginConfigobjects.0135_data_mapping.py
Added migration for data mapping in `AnalyzerConfig`.api_app/analyzers_manager/migrations/0135_data_mapping.py
AnalyzerConfig.URLhaus,MaxMindGeoIP, andAbuseIPDB.0036_alter_extractedonenotefiles_resubmitdownloadedfile_loadfilesameplaybook.py
Updated pivot configurations for specific playbooks.api_app/pivots_manager/migrations/0036_alter_extractedonenotefiles_resubmitdownloadedfile_loadfilesameplaybook.py
ExtractedOneNoteFilesandResubmitDownloadedFile.PythonModuleandParameterfor pivots.0004_alter_domaindatamodel_evaluation_and_more.py
Altered evaluation fields in data models for consistency.api_app/data_model_manager/migrations/0004_alter_domaindatamodel_evaluation_and_more.py
evaluationfields in data models to useLowercaseCharField.0064_vt_sample_download.py
Added migration for VirusTotal sample download configuration.api_app/migrations/0064_vt_sample_download.py
0039_sample_download.py
Added migration for `Download_File` visualizer configuration.api_app/visualizers_manager/migrations/0039_sample_download.py
Download_Filevisualizer configuration.__init__.py
Add data model manager to installed appsthreat_matrix/settings/init.py
api_app.data_model_managerto installed apps.apps.py
Add app configuration for data model managerapi_app/data_model_manager/apps.py
DataModelConfigclass for app configuration.environment.js
Update documentation URL in environment constantsfrontend/src/constants/environment.js
THREATMATRIX_DOCS_URLto new documentation path.entrypoint.sh
Add entrypoint script for phishing analyzersintegrations/phishing_analyzers/entrypoint.sh
dependabot.yml
Update dependabot configuration for develop branch.github/dependabot.yml
develop.2 files
spyse.py
Reorder imports in Spyse analyzer for consistencyapi_app/analyzers_manager/observable_analyzers/spyse.py
email_sender.py
Reorder imports in EmailSender connector for consistencyapi_app/connectors_manager/connectors/email_sender.py
2 files
ldap_config.py
Update LDAP configuration documentation URLconfiguration/ldap_config.py
CHANGELOG.md
Update changelog with new documentation links and draft.github/CHANGELOG.md
41 files
README.md
...README.md
...
package-lock.json
...frontend/package-lock.json
...
pull_request_template.md
....github/pull_request_template.md
...
download.test.jsx
...frontend/tests/components/jobs/result/visualizer/elements/download.test.jsx
...
Dashboard.jsx
...frontend/src/components/dashboard/Dashboard.jsx
...
download.jsx
...frontend/src/components/jobs/result/visualizer/elements/download.jsx
...
Home.jsx
...frontend/src/components/home/Home.jsx
...
Dockerfile
...integrations/phishing_analyzers/Dockerfile
...
release_template.md
....github/release_template.md
...
compose.yml
...integrations/phishing_analyzers/compose.yml
...
visualizer.jsx
...frontend/src/components/jobs/result/visualizer/visualizer.jsx
...
AppHeader.test.jsx
...frontend/tests/layouts/AppHeader.test.jsx
...
Dashboard.test.jsx
...frontend/tests/components/dashboard/Dashboard.test.jsx
...
visualizer.test.jsx
...frontend/tests/components/jobs/result/visualizer/visualizer.test.jsx
...
start
...start
...
JobActionBar.jsx
...frontend/src/components/jobs/result/bar/JobActionBar.jsx
...
README.md
...frontend/README.md
...
pull_request_automation.yml
....github/workflows/pull_request_automation.yml
...
notifications.jsx
...frontend/src/components/jobs/notifications.jsx
...
Dockerfile
...integrations/malware_tools_analyzers/Dockerfile
...
plugin_report.json
...configuration/elastic_search_mappings/plugin_report.json
...
TLPSelectInput.jsx
...frontend/src/components/common/form/TLPSelectInput.jsx
...
GuideWrapper.jsx
...frontend/src/components/GuideWrapper.jsx
...
test.override.yml
...docker/test.override.yml
...
TokenPage.jsx
...frontend/src/components/user/token/TokenPage.jsx
...
TokenAccess.jsx
...frontend/src/components/user/token/TokenAccess.jsx
...
create_elastic_certs
...create_elastic_certs
...
PluginWrapper.jsx
...frontend/src/components/plugins/types/PluginWrapper.jsx
...
package.json
...frontend/package.json
...
requirements.txt
...integrations/phishing_analyzers/requirements.txt
...
compose-tests.yml
...integrations/phishing_analyzers/compose-tests.yml
...
project-requirements.txt
...requirements/project-requirements.txt
...
compose.yml
...integrations/malware_tools_analyzers/compose.yml
...
FUNDING.yml
....github/FUNDING.yml
...
compose.yml
...integrations/pcap_analyzers/compose.yml
...
compose.yml
...integrations/tor_analyzers/compose.yml
...
elasticsearch.override.yml
...docker/elasticsearch.override.yml
...
CONTRIBUTING.md
....github/CONTRIBUTING.md
...
compose.yml
...integrations/phoneinfoga/compose.yml
...
compose-tests.yml
...integrations/malware_tools_analyzers/compose-tests.yml
...
compose-tests.yml
...integrations/tor_analyzers/compose-tests.yml
...