Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/logs and alerts #60

Merged
merged 42 commits into from
Nov 30, 2023
Merged

Feat/logs and alerts #60

merged 42 commits into from
Nov 30, 2023

Conversation

JoaoM-py
Copy link
Member

Métricas e alertas

Issues relacionadas

#58

PR Type

Que tipo de mudança a PR introduz?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Other... Please describe:

Descreva a alteração

  • Organização das metricas e cadastro de logs e erros

Outras informações

  • Modelo json dos dados até onde o meu PC esta rodando
  {
    "_id": {
      "$oid": "65675811348dfb304c11a715"
    },
    "erros": [
      {
        "total_of_null_evaluations": {
          "value": 3275,
          "type": "null_avaluations"
        },
        "duplicate_records": {
          "value": 7384,
          "type": "duplicate_avaluations"
        }
      },
      [
        {
          "type": "stop_only_reviews",
          "day": {
            "$date": "2023-11-29T12:38:17.870Z"
          },
          "value": 53
        }
      ],
      {
        "memory_log_error": {
          "error_type": "<class 'numpy.core._exceptions._ArrayMemoryError'>",
          "error_message": "Unable to allocate 5.30 GiB for an array with shape (71200, 10000) and data type float64",
          "error_bugfix_message": "Considere reduzir a quandidade de dados"
        }
      }
    ],
    "metrics": [
      {
        "stage": "Acesso aos dados",
        "day": {
          "$date": "2023-11-29T12:26:10.336Z"
        },
        "time": "00:00:05"
      },
      {
        "total_of_data": 968384
      },
      {
        "stage": "Pré processamento",
        "day": {
          "$date": "2023-11-29T12:36:34.887Z"
        },
        "time": "00:10:24"
      },
      {
        "model_accuracy": 0.3333333333333333,
        "model_precision": 0.33638583638583636
      }
      {
        "stage": "Treinamento do modelo",
        "day": {
          "$date": "2023-11-29T12:37:28.298Z"
        },
        "time": "00:00:53"
      }
    ]
  }

MariaGabrielaReis and others added 29 commits October 16, 2023 00:44
* hotfix names

* Refactor: Format date (#27)

* refactor: capitalize sentiments (#24)

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* fix: translate sentiments (#28)

* Feat/separate training reviews (#35)

* feat: create function to get random data

* feat: create function to get training data

* feat: get training data

* chore: Manual sentiment classification

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>

* Feat/#0303 create classification model (#37)

* feat: update manual classification

* fix: translate topics and sentiments

* feat: add seaborn lib

* fix: translate topics, update reviews count

* feat: create classification model

* feat: training, test and apply classification model

* refactor: Update stars name

* refactor: Update training method

---------

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* Feat/#42 test coverage (#38)

* feat: create pipeline tests

* refactor: Update files names

* feat: Training data

* feat: Creating pipeline tests

* merge: Merge develop

* feat: Coverage and sonarcloud config

* refactor: Update processing and remove comments

* chore: update python version

* chore: update sonarcloud

* chore: update tests

* chore: update tests

* chore: update tests

---------

Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>
Co-authored-by: JoaoM-py <80930525+JoaoM-py@users.noreply.github.com>
Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>
…cks/mood-hound-nlp into refactor/optimize-pipeline
@JoaoM-py JoaoM-py added the feature request This issue is a function asked by the client label Nov 29, 2023
@JoaoM-py JoaoM-py self-assigned this Nov 29, 2023
Copy link
Member

@MariaGabrielaReis MariaGabrielaReis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vai na fé sz

Copy link

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

27.3% 27.3% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@JoaoM-py JoaoM-py merged commit 446c55b into develop Nov 30, 2023
2 of 3 checks passed
JoaoM-py added a commit that referenced this pull request Dec 1, 2023
* hotfix names

* Refactor: Format date (#27)

* refactor: capitalize sentiments (#24)

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* fix: translate sentiments (#28)

* Feat/separate training reviews (#35)

* feat: create function to get random data

* feat: create function to get training data

* feat: get training data

* chore: Manual sentiment classification

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>

* Feat/#0303 create classification model (#37)

* feat: update manual classification

* fix: translate topics and sentiments

* feat: add seaborn lib

* fix: translate topics, update reviews count

* feat: create classification model

* feat: training, test and apply classification model

* refactor: Update stars name

* refactor: Update training method

---------

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* Feat/#42 test coverage (#38)

* feat: create pipeline tests

* refactor: Update files names

* feat: Training data

* feat: Creating pipeline tests

* merge: Merge develop

* feat: Coverage and sonarcloud config

* refactor: Update processing and remove comments

* chore: update python version

* chore: update sonarcloud

* chore: update tests

* chore: update tests

* chore: update tests

* Feat/#0106 bring birth year and gender (#50)

* Tests and Sentment Analysis (#39)

* hotfix names

* Refactor: Format date (#27)

* refactor: capitalize sentiments (#24)

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* fix: translate sentiments (#28)

* Feat/separate training reviews (#35)

* feat: create function to get random data

* feat: create function to get training data

* feat: get training data

* chore: Manual sentiment classification

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>

* Feat/#0303 create classification model (#37)

* feat: update manual classification

* fix: translate topics and sentiments

* feat: add seaborn lib

* fix: translate topics, update reviews count

* feat: create classification model

* feat: training, test and apply classification model

* refactor: Update stars name

* refactor: Update training method

---------

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* Feat/#42 test coverage (#38)

* feat: create pipeline tests

* refactor: Update files names

* feat: Training data

* feat: Creating pipeline tests

* merge: Merge develop

* feat: Coverage and sonarcloud config

* refactor: Update processing and remove comments

* chore: update python version

* chore: update sonarcloud

* chore: update tests

* chore: update tests

* chore: update tests

---------

Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>
Co-authored-by: JoaoM-py <80930525+JoaoM-py@users.noreply.github.com>
Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* feat: Bring informations of the client

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>
Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>

* Feat/#11 new runtime tracker (#49)

* Tests and Sentment Analysis (#39)

* hotfix names

* Refactor: Format date (#27)

* refactor: capitalize sentiments (#24)

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* fix: translate sentiments (#28)

* Feat/separate training reviews (#35)

* feat: create function to get random data

* feat: create function to get training data

* feat: get training data

* chore: Manual sentiment classification

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>

* Feat/#0303 create classification model (#37)

* feat: update manual classification

* fix: translate topics and sentiments

* feat: add seaborn lib

* fix: translate topics, update reviews count

* feat: create classification model

* feat: training, test and apply classification model

* refactor: Update stars name

* refactor: Update training method

---------

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* Feat/#42 test coverage (#38)

* feat: create pipeline tests

* refactor: Update files names

* feat: Training data

* feat: Creating pipeline tests

* merge: Merge develop

* feat: Coverage and sonarcloud config

* refactor: Update processing and remove comments

* chore: update python version

* chore: update sonarcloud

* chore: update tests

* chore: update tests

* chore: update tests

---------

Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>
Co-authored-by: JoaoM-py <80930525+JoaoM-py@users.noreply.github.com>
Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* feat: New time metric, storage of metrics and

* feat: increase test coverage

* feat: Coverage

* feat: increase test coverage

* feature: Update sonar yml

* feat: Update test url

* feat: Update test url

* feat: Update test url

* feat: Update test url

* feat: Update test url

* feat: Config env

* refactor:Update utils imports

* refactor: Update env

* refactor: Update url and env

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>
Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>

* Feat/#48 unit tests (#51)

* Tests and Sentment Analysis (#39)

* hotfix names

* Refactor: Format date (#27)

* refactor: capitalize sentiments (#24)

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* fix: translate sentiments (#28)

* Feat/separate training reviews (#35)

* feat: create function to get random data

* feat: create function to get training data

* feat: get training data

* chore: Manual sentiment classification

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>

* Feat/#0303 create classification model (#37)

* feat: update manual classification

* fix: translate topics and sentiments

* feat: add seaborn lib

* fix: translate topics, update reviews count

* feat: create classification model

* feat: training, test and apply classification model

* refactor: Update stars name

* refactor: Update training method

---------

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* Feat/#42 test coverage (#38)

* feat: create pipeline tests

* refactor: Update files names

* feat: Training data

* feat: Creating pipeline tests

* merge: Merge develop

* feat: Coverage and sonarcloud config

* refactor: Update processing and remove comments

* chore: update python version

* chore: update sonarcloud

* chore: update tests

* chore: update tests

* chore: update tests

---------

Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>
Co-authored-by: JoaoM-py <80930525+JoaoM-py@users.noreply.github.com>
Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* feat: increase test coverage

* feat: Coverage

* feat: increase test coverage

* feature: Update sonar yml

* feat: Update test url

* feat: Update test url

* feat: Update test url

* feat: Update test url

* feat: Update test url

* feat: Config env

* refactor:Update utils imports

* refactor: Update env

* refactor: Update url and env

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>
Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>

* Feat: new classification model (#52)

* feat: add new model to sentiment analysis

* fix: use correct training and treat exceptions

* fix: remove console logs

* feat: increment reviews quantity

* Refactor/optimize pipeline (#56)

* feat: add new model to sentiment analysis

* fix: use correct training and treat exceptions

* fix: remove console logs

* feat: increment reviews quantity

* feat: select only necessary columns and apply your types

* feat: optimize clear data step

* Delete .env

* Refactor/optimize pipeline (#57)

* feat: add new model to sentiment analysis

* fix: use correct training and treat exceptions

* fix: remove console logs

* feat: increment reviews quantity

* feat: select only necessary columns and apply your types

* feat: optimize clear data step

* Delete .env

* style: format and remove unused files

* feat: create a file for pre processing step

* feat: create file to classification model

* feat: create file to topic model

* feat: get metrics from classification model

* feat: use new steps and adjust details

* fix: remove some code smells and run processing

* fix: update training and test data visualization

* Fix/unit tests (#59)

* Tests and Sentment Analysis (#39)

* hotfix names

* Refactor: Format date (#27)

* refactor: capitalize sentiments (#24)

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* fix: translate sentiments (#28)

* Feat/separate training reviews (#35)

* feat: create function to get random data

* feat: create function to get training data

* feat: get training data

* chore: Manual sentiment classification

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>

* Feat/#0303 create classification model (#37)

* feat: update manual classification

* fix: translate topics and sentiments

* feat: add seaborn lib

* fix: translate topics, update reviews count

* feat: create classification model

* feat: training, test and apply classification model

* refactor: Update stars name

* refactor: Update training method

---------

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* Feat/#42 test coverage (#38)

* feat: create pipeline tests

* refactor: Update files names

* feat: Training data

* feat: Creating pipeline tests

* merge: Merge develop

* feat: Coverage and sonarcloud config

* refactor: Update processing and remove comments

* chore: update python version

* chore: update sonarcloud

* chore: update tests

* chore: update tests

* chore: update tests

---------

Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>
Co-authored-by: JoaoM-py <80930525+JoaoM-py@users.noreply.github.com>
Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* feat: add new model to sentiment analysis

* fix: use correct training and treat exceptions

* fix: remove console logs

* feat: increment reviews quantity

* fix: tests of the updated pipeline

* feat: select only necessary columns and apply your types

* feat: optimize clear data step

* Delete .env

* style: format and remove unused files

* feat: create a file for pre processing step

* feat: create file to classification model

* feat: create file to topic model

* feat: get metrics from classification model

* feat: use new steps and adjust details

* fix: remove some code smells and run processing

* fix: update training and test data visualization

* feat: create and fix unit tests

* fix: remove .coverage

* reafacto: remove unused test

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>
Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>

* Feat/logs and alerts (#60)

* Tests and Sentment Analysis (#39)

* hotfix names

* Refactor: Format date (#27)

* refactor: capitalize sentiments (#24)

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* fix: translate sentiments (#28)

* Feat/separate training reviews (#35)

* feat: create function to get random data

* feat: create function to get training data

* feat: get training data

* chore: Manual sentiment classification

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>

* Feat/#0303 create classification model (#37)

* feat: update manual classification

* fix: translate topics and sentiments

* feat: add seaborn lib

* fix: translate topics, update reviews count

* feat: create classification model

* feat: training, test and apply classification model

* refactor: Update stars name

* refactor: Update training method

---------

Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* Feat/#42 test coverage (#38)

* feat: create pipeline tests

* refactor: Update files names

* feat: Training data

* feat: Creating pipeline tests

* merge: Merge develop

* feat: Coverage and sonarcloud config

* refactor: Update processing and remove comments

* chore: update python version

* chore: update sonarcloud

* chore: update tests

* chore: update tests

* chore: update tests

---------

Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>
Co-authored-by: JoaoM-py <80930525+JoaoM-py@users.noreply.github.com>
Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>

* feat: add new model to sentiment analysis

* fix: use correct training and treat exceptions

* fix: remove console logs

* feat: increment reviews quantity

* fix: tests of the updated pipeline

* feat: select only necessary columns and apply your types

* feat: optimize clear data step

* Delete .env

* style: format and remove unused files

* feat: create a file for pre processing step

* feat: create file to classification model

* feat: create file to topic model

* feat: get metrics from classification model

* feat: use new steps and adjust details

* fix: remove some code smells and run processing

* fix: update training and test data visualization

* feat: create and fix unit tests

* fix: remove .coverage

* reafacto: remove unused test

* feat: Organizing metrics, collect of logs and alerts

* fix: Pipeline exec time

* fix: code format and increase reviews quantity

* feat: increase reviews quantity

* fix: Pipeline exec time

* fix: Pipeline exec time

* fix: Pipeline exec time

* fix: Pipeline exec time format

---------

Co-authored-by: Maria Gabriela Reis <mariagabrielagreis@gmail.com>
Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>

---------

Co-authored-by: GabrielCamargoL <gabriel.cleite@outlook.com>
Co-authored-by: JoaoM-py <80930525+JoaoM-py@users.noreply.github.com>
Co-authored-by: JoaoM-py <joao.marcos2002@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request This issue is a function asked by the client
Projects
No open projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

2 participants