[Self-Hosted]: Upload failed: Invalid request parameters - unallowed value github_enterprise

[michaelfedell]

Describe the bug
Uploading the coverage report fails with an error: Upload failed: Invalid request parameters.

Environment (please complete the following information):

• CodeCov Self Hosted (running on kubernetes
• GitHub Enterprise Server (3.14)
• CodeCov GitHub Action @v5

To Reproduce
Steps to reproduce the behavior:

1. CodeCov Self-Hosted with GitHub App on GitHub Enterprise Server
2. Follow the "github-tutorial" repository from codecov docs
3. Error on upload from CI

Expected behavior
I expect the GitHub Action to properly upload a coverage report.

Additional context

I can successfully log in to our self-hosted codecov instance and see no other error or warning logs in any of the codecov containers.

GitHub Action Workflow

name: API workflow

on: [push, pull_request]

jobs:
  build:
    runs-on: arc-runner-default
    name: Test python API
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Install requirements
        run: pip install -r api/requirements.txt
      - name: Run tests and collect coverage
        run: pytest --cov=api.calculator --cov-report=xml
      - name: Upload coverage reports to Codecov with GitHub Action
        uses: codecov/codecov-action@v5
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
        with:
          url: https://codecov.mycompany.com
          git_service: github_enterprise

Logs pulled from codecov-api contain the following relevant logs:

{
  "message": "Received upload request github_enterprise",
  "asctime": "2025-01-28 21:31:31,486",
  "name": "upload.views.legacy",
  "levelname": "INFO",
  "lineno": 90,
  "pathname": "/app/upload/views/legacy.py",
  "funcName": "post",
  "threadName": "MainThread",
  "taskName": null,
  "version": "github_enterprise",
  "query_params": {},
  "commit": null,
  "logger.name": "upload.views.legacy",
  "logger.thread_name": "MainThread",
  "level": "INFO"
}
{
  "message": "Failed to parse upload request params",
  "asctime": "2025-01-28 21:31:31,499",
  "name": "upload.views.legacy",
  "levelname": "WARNING",
  "lineno": 133,
  "pathname": "/app/upload/views/legacy.py",
  "funcName": "post",
  "threadName": "MainThread",
  "taskName": null,
  "request_params": { "version": "github_enterprise", "token": null },
  "errors": "{'commit': [ErrorDetail(string='required field', code='invalid')], 'version': [ErrorDetail(string='unallowed value github_enterprise', code='invalid')]}",
  "logger.name": "upload.views.legacy",
  "logger.thread_name": "MainThread",
  "level": "WARNING"
}
{
  "message": "Bad Request: /upload/github_enterprise/MyOrg::::codecov-demo/upload-coverage",
  "asctime": "2025-01-28 21:31:31,499",
  "name": "django.request",
  "levelname": "WARNING",
  "lineno": 241,
  "pathname": "/usr/local/lib/python3.12/site-packages/django/utils/log.py",
  "funcName": "log_response",
  "threadName": "MainThread",
  "taskName": null,
  "status_code": 400,
  "request": "<WSGIRequest: POST '/upload/github_enterprise/MyOrg::::codecov-demo/upload-coverage'>",
  "logger.name": "django.request",
  "logger.thread_name": "MainThread",
  "level": "WARNING"
}

[nikosatwork]

Hi @michaelfedell. What version of Codecov are you on?

[nikosatwork]

Also, try
enterprise-url: https://codecov.mycompany.com
instead of
url: https://codecov.mycompany.com

See Usage

Though I am not familiar with the GitHub action. We use CLI directly in our job/action.

[michaelfedell]

This is a brand new installation; we are running 24.10.1 image tags.

enterprise-url is not supported by the github action, but it does appear to be passed down to the CLI call appropriately. Logs from the github action included below.

As I was going to pull these logs and redact our platform url, I noticed that there was a trailing slash missing (company.comupload-coverage/ instead of company.com/upload-coverage). Fixing this did not have any impact on the original error though; the upload still fails to process and the same errors show up in the api logs.

I think next I will try using the CLI directly for more control.

     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/
                            Wrapper-0.1.0
                                  
==> Detected linux
 -> Downloading https://cli.codecov.io/latest/linux/codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  1 27.4M    1  397k    0     0  1243k      0  0:00:22 --:--:--  0:00:22 1239k
100 27.4M  100 27.4M    0     0  55.0M      0 --:--:-- --:--:-- --:--:-- 55.0M
==> Finishing downloading linux:latest
      Version: v10.0.1
 
gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" imported
gpg: Total number processed: 1
==> Verifying GPG signature integrity
gpg:               imported: 1
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM.sig
 
gpg: Signature made Wed Jan 22 18:31:00 2025 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2703 4E7F DB85 0E0B BC2C  62FF 806B B28A ED77 9869
codecov: OK
==> CLI integrity verified

 -> Token of length 36 detected
==> Running upload-coverage
      ./codecov --enterprise-url https://codecov-dev.<company>.comupload-coverage/ -t <redacted> --git-service github_enterprise --sha f847611ecc3a7d2046c3ba157a361c2c46ea423f --gcov-executable gcov
info - 2025-01-28 23:03:26,101 -- ci service found: github-actions
warning - 2025-01-28 23:03:26,109 -- No config file could be found. Ignoring config.
warning - 2025-01-28 23:03:26,114 -- xcrun is not installed or can't be found.
warning - 2025-01-28 23:03:26,115 -- gcov is not installed or can't be found.
info - 2025-01-28 23:03:26,115 -- Generating coverage.xml report in /home/runner/_work/codecov-demo/codecov-demo
info - 2025-01-28 23:03:26,330 -- Wrote XML report to coverage.xml
info - 2025-01-28 23:03:26,347 -- Found 1 coverage files to report
info - 2025-01-28 23:03:26,347 -- > /home/runner/_work/codecov-demo/codecov-demo/coverage.xml
info - 2025-01-28 23:03:26,427 -- Process Upload complete
error - 2025-01-28 23:03:26,427 -- Upload failed: Invalid request parameters

[nikosatwork]

Latest version is 25.1.16 - see Codecov in docker

[nikosatwork]

Our CLI call looks like this:

./codecov --enterprise-url https://codecov.self-hosted.com --verbose upload-process --disable-search --disable-file-fixes --plugin none --fail-on-error --git-service github_enterprise --file *.xml --dir CodeCoverage/Raw -t *** --branch main --sha beb431c1ba241028a70e35ef728e484e64ed8396 --name Tests --flag Tests

[nikosatwork]

upload-coverage should be separate

i.e. ./codecov --enterprise-url https://codecov-dev.company.com upload-coverage -t <redacted> --git-service github_enterprise --sha f847611ecc3a7d2046c3ba157a361c2c46ea423f --gcov-executable gcov

[michaelfedell]

I have upgraded to the latest 25.1.16 images and have also set the token explicitly in our use of the github action (instead of using CODECOV_TOKEN env var. Seeing a different error now (500's on the upload) which I'll dig into later this afternoon. The api logs are pretty noisy, but will go spelunking there to find where the 500's are coming from.

Edit: took a quick glance and seems it may be an issue with minio/s3 configuration so I may have some infra problems to look at now. Will update here later.

==> Running upload-coverage
      ./codecov --enterprise-url https://codecov-dev.<company>.com/upload-coverage -t <redacted> --git-service github_enterprise --sha d7bc2d5d0caa884ca42eedffe830673d67f7cdaf --gcov-executable gcov
info - 2025-01-30 17:23:22,108 -- ci service found: github-actions
warning - 2025-01-30 17:23:22,116 -- No config file could be found. Ignoring config.
warning - 2025-01-30 17:23:22,119 -- xcrun is not installed or can't be found.
warning - 2025-01-30 17:23:22,120 -- gcov is not installed or can't be found.
info - 2025-01-30 17:23:22,120 -- Generating coverage.xml report in /home/runner/_work/codecov-demo/codecov-demo
info - 2025-01-30 17:23:22,257 -- Wrote XML report to coverage.xml
info - 2025-01-30 17:23:22,267 -- Found 1 coverage files to report
info - 2025-01-30 17:23:22,267 -- > /home/runner/_work/codecov-demo/codecov-demo/coverage.xml
warning - 2025-01-30 17:23:22,783 -- Response status code was 500. --- {"retry": 0}
warning - 2025-01-30 17:23:22,783 -- Request failed. Retrying --- {"retry": 0}
warning - 2025-01-30 17:23:23,765 -- Response status code was 500. --- {"retry": 1}
warning - 2025-01-30 17:23:23,765 -- Request failed. Retrying --- {"retry": 1}
warning - 2025-01-30 17:23:25,221 -- Response status code was 500. --- {"retry": 2}
warning - 2025-01-30 17:23:25,221 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
  File "codecov_cli/main.py", line 97, in <module>
  File "codecov_cli/main.py", line 93, in run
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "click/decorators.py", line 33, in new_func
  File "codecov_cli/commands/upload_coverage.py", line 82, in upload_coverage
  File "click/core.py", line 783, in invoke
  File "codecov_cli/services/upload_coverage/__init__.py", line 50, in upload_coverage_logic
  File "codecov_cli/services/upload/__init__.py", line 135, in do_upload_logic
  File "codecov_cli/services/upload/upload_sender.py", line 95, in send_upload_data
  File "codecov_cli/helpers/request.py", line 82, in wrapper
Exception: Request failed after too many retries
[PYI-410:ERROR] Failed to execute script 'main' due to unhandled exception!
==> Failed to run upload-coverage

[michaelfedell]

Hey @nikosatwork, thanks for all the input!

Turned out to be something simple.
Newer version of codecov gave better logs which showed that minio was failing to hit S3 due to a malformed token which is what I usually see when we've got a partition mismatch (our s3 bucket is in us-gov and I think minio is probably trying to hit commercial endpoint by default).

Finally found that there was a default env var being set in our deployment for services__minio__external_host=s3.amazonaws.com. Overriding this to be the proper regional endpoint in gov partition solved our upload problem and I've got the demo working now!