Skip to content

Commit

Permalink
[8.18] [Fleet] Fix unattended Transforms in integration packages not …
Browse files Browse the repository at this point in the history
…automatically restarting after reauthorizing (#210217) (#211807)

# Backport

This will backport the following commits from `main` to `8.18`:
- [[Fleet] Fix unattended Transforms in integration packages not
automatically restarting after reauthorizing
(#210217)](#210217)

<!--- Backport version: 9.6.4 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Quynh Nguyen
(Quinn)","email":"43350163+qn895@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-02-13T05:49:58Z","message":"[Fleet]
Fix unattended Transforms in integration packages not automatically
restarting after reauthorizing (#210217)\n\n## Summary\n\nThis PR
partially addresses an issue
with\nhttps://github.com/elastic/integrations/issues/12486 where the
transform\ndoesn't \"restart\" immediately after reauthorizing. This is
because for\nunattended transform, calling `_start` will come back with
409 transform\nalready started error. So this PR tracks if the transform
has\n`settings.unattended: true`, if yes, stop the transform first
before\nstarting.\n\nWithout this step, the transform will retry again
and become healthy\nagain anyway but it takes longer for that retry to
happen, so this PR\nspeeds up the process of retrying.\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] Any text
added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\nDoes this PR introduce any risks? For example,
consider risks like hard\nto test bugs, performance regression,
potential of data loss.\n\nDescribe the risk, its severity, and
mitigation for each identified\nrisk. Invite stakeholders and evaluate
how to proceed before merging.\n\n- [ ] [See some
risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n-
[ ] ...\n\n---------\n\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"e710f09d0c1159a9018ff6b9dfbac151509213f9","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Team:Fleet","v9.0.0","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3","v8.16.5"],"title":"[Fleet]
Fix unattended Transforms in integration packages not automatically
restarting after
reauthorizing","number":210217,"url":"https://github.com/elastic/kibana/pull/210217","mergeCommit":{"message":"[Fleet]
Fix unattended Transforms in integration packages not automatically
restarting after reauthorizing (#210217)\n\n## Summary\n\nThis PR
partially addresses an issue
with\nhttps://github.com/elastic/integrations/issues/12486 where the
transform\ndoesn't \"restart\" immediately after reauthorizing. This is
because for\nunattended transform, calling `_start` will come back with
409 transform\nalready started error. So this PR tracks if the transform
has\n`settings.unattended: true`, if yes, stop the transform first
before\nstarting.\n\nWithout this step, the transform will retry again
and become healthy\nagain anyway but it takes longer for that retry to
happen, so this PR\nspeeds up the process of retrying.\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] Any text
added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\nDoes this PR introduce any risks? For example,
consider risks like hard\nto test bugs, performance regression,
potential of data loss.\n\nDescribe the risk, its severity, and
mitigation for each identified\nrisk. Invite stakeholders and evaluate
how to proceed before merging.\n\n- [ ] [See some
risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n-
[ ] ...\n\n---------\n\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"e710f09d0c1159a9018ff6b9dfbac151509213f9"}},"sourceBranch":"main","suggestedTargetBranches":["8.18","8.x","8.17","8.16"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210950","number":210950,"state":"MERGED","mergeCommit":{"sha":"a8d9319cdd70a1ead8228d0da858c71bd709efe6","message":"[9.0]
[Fleet] Fix unattended Transforms in integration packages not
automatically restarting after reauthorizing (#210217) (#210950)\n\n#
Backport\n\nThis will backport the following commits from `main` to
`9.0`:\n- [[Fleet] Fix unattended Transforms in integration packages
not\nautomatically restarting after
reauthorizing\n(#210217)](https://github.com/elastic/kibana/pull/210217)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Quynh
Nguyen\n(Quinn)\",\"email\":\"43350163+qn895@users.noreply.github.com\"},\"sourceCommit\":{\"committedDate\":\"2025-02-13T05:49:58Z\",\"message\":\"[Fleet]\nFix
unattended Transforms in integration packages not
automatically\nrestarting after reauthorizing (#210217)\\n\\n##
Summary\\n\\nThis PR\npartially addresses an
issue\nwith\\nhttps://github.com/elastic/integrations/issues/12486 where
the\ntransform\\ndoesn't \\\"restart\\\" immediately after
reauthorizing. This is\nbecause for\\nunattended transform, calling
`_start` will come back with\n409 transform\\nalready started error. So
this PR tracks if the transform\nhas\\n`settings.unattended: true`, if
yes, stop the transform first\nbefore\\nstarting.\\n\\nWithout this
step, the transform will retry again\nand become healthy\\nagain anyway
but it takes longer for that retry to\nhappen, so this PR\\nspeeds up
the process of retrying.\\n\\n\\n###\nChecklist\\n\\nCheck the PR
satisfies following conditions. \\n\\nReviewers\nshould verify this PR
satisfies this list as well.\\n\\n- [ ] Any text\nadded follows
[EUI's\nwriting\\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),\nuses\\nsentence
case text and
includes\n[i18n\\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\\n-\n[\n]\\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\\nwas\nadded
for features that require explanation or tutorials\\n- [ ] [Unit
or\nfunctional\\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\\nwere\nupdated
or added to match the most common scenarios\\n- [ ] If a
plugin\nconfiguration key changed, check if it needs to be\\nallowlisted
in the\ncloud and added to
the\n[docker\\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\\n-\n[
] This was checked for breaking HTTP API changes, and
any\nbreaking\\nchanges have been approved by the breaking-change
committee.\nThe\\n`release_note:breaking` label should be applied in
these\nsituations.\\n- [ ]
[Flaky\nTest\\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)\nwas\\nused
on any tests changed\\n- [ ] The PR description includes
the\nappropriate Release Notes section,\\nand the correct
`release_note:*`\nlabel is applied
per\nthe\\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\\n\\n###\nIdentify
risks\\n\\nDoes this PR introduce any risks? For example,\nconsider
risks like hard\\nto test bugs, performance regression,\npotential of
data loss.\\n\\nDescribe the risk, its severity, and\nmitigation for
each identified\\nrisk. Invite stakeholders and evaluate\nhow to proceed
before merging.\\n\\n- [ ] [See
some\nrisk\\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\\n-\n[
] ...\\n\\n---------\\n\\nCo-authored-by: Elastic
Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"e710f09d0c1159a9018ff6b9dfbac151509213f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\":ml\",\"Team:Fleet\",\"v9.0.0\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\",\"v8.16.5\"],\"title\":\"[Fleet]\nFix
unattended Transforms in integration packages not
automatically\nrestarting
after\nreauthorizing\",\"number\":210217,\"url\":\"https://github.com/elastic/kibana/pull/210217\",\"mergeCommit\":{\"message\":\"[Fleet]\nFix
unattended Transforms in integration packages not
automatically\nrestarting after reauthorizing (#210217)\\n\\n##
Summary\\n\\nThis PR\npartially addresses an
issue\nwith\\nhttps://github.com/elastic/integrations/issues/12486 where
the\ntransform\\ndoesn't \\\"restart\\\" immediately after
reauthorizing. This is\nbecause for\\nunattended transform, calling
`_start` will come back with\n409 transform\\nalready started error. So
this PR tracks if the transform\nhas\\n`settings.unattended: true`, if
yes, stop the transform first\nbefore\\nstarting.\\n\\nWithout this
step, the transform will retry again\nand become healthy\\nagain anyway
but it takes longer for that retry to\nhappen, so this PR\\nspeeds up
the process of retrying.\\n\\n\\n###\nChecklist\\n\\nCheck the PR
satisfies following conditions. \\n\\nReviewers\nshould verify this PR
satisfies this list as well.\\n\\n- [ ] Any text\nadded follows
[EUI's\nwriting\\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),\nuses\\nsentence
case text and
includes\n[i18n\\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\\n-\n[\n]\\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\\nwas\nadded
for features that require explanation or tutorials\\n- [ ] [Unit
or\nfunctional\\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\\nwere\nupdated
or added to match the most common scenarios\\n- [ ] If a
plugin\nconfiguration key changed, check if it needs to be\\nallowlisted
in the\ncloud and added to
the\n[docker\\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\\n-\n[
] This was checked for breaking HTTP API changes, and
any\nbreaking\\nchanges have been approved by the breaking-change
committee.\nThe\\n`release_note:breaking` label should be applied in
these\nsituations.\\n- [ ]
[Flaky\nTest\\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)\nwas\\nused
on any tests changed\\n- [ ] The PR description includes
the\nappropriate Release Notes section,\\nand the correct
`release_note:*`\nlabel is applied
per\nthe\\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\\n\\n###\nIdentify
risks\\n\\nDoes this PR introduce any risks? For example,\nconsider
risks like hard\\nto test bugs, performance regression,\npotential of
data loss.\\n\\nDescribe the risk, its severity, and\nmitigation for
each identified\\nrisk. Invite stakeholders and evaluate\nhow to proceed
before merging.\\n\\n- [ ] [See
some\nrisk\\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\\n-\n[
] ...\\n\\n---------\\n\\nCo-authored-by: Elastic
Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"e710f09d0c1159a9018ff6b9dfbac151509213f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\",\"8.16\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/210217\",\"number\":210217,\"mergeCommit\":{\"message\":\"[Fleet]\nFix
unattended Transforms in integration packages not
automatically\nrestarting after reauthorizing (#210217)\\n\\n##
Summary\\n\\nThis PR\npartially addresses an
issue\nwith\\nhttps://github.com/elastic/integrations/issues/12486 where
the\ntransform\\ndoesn't \\\"restart\\\" immediately after
reauthorizing. This is\nbecause for\\nunattended transform, calling
`_start` will come back with\n409 transform\\nalready started error. So
this PR tracks if the transform\nhas\\n`settings.unattended: true`, if
yes, stop the transform first\nbefore\\nstarting.\\n\\nWithout this
step, the transform will retry again\nand become healthy\\nagain anyway
but it takes longer for that retry to\nhappen, so this PR\\nspeeds up
the process of retrying.\\n\\n\\n###\nChecklist\\n\\nCheck the PR
satisfies following conditions. \\n\\nReviewers\nshould verify this PR
satisfies this list as well.\\n\\n- [ ] Any text\nadded follows
[EUI's\nwriting\\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),\nuses\\nsentence
case text and
includes\n[i18n\\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\\n-\n[\n]\\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\\nwas\nadded
for features that require explanation or tutorials\\n- [ ] [Unit
or\nfunctional\\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\\nwere\nupdated
or added to match the most common scenarios\\n- [ ] If a
plugin\nconfiguration key changed, check if it needs to be\\nallowlisted
in the\ncloud and added to
the\n[docker\\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\\n-\n[
] This was checked for breaking HTTP API changes, and
any\nbreaking\\nchanges have been approved by the breaking-change
committee.\nThe\\n`release_note:breaking` label should be applied in
these\nsituations.\\n- [ ]
[Flaky\nTest\\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)\nwas\\nused
on any tests changed\\n- [ ] The PR description includes
the\nappropriate Release Notes section,\\nand the correct
`release_note:*`\nlabel is applied
per\nthe\\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\\n\\n###\nIdentify
risks\\n\\nDoes this PR introduce any risks? For example,\nconsider
risks like hard\\nto test bugs, performance regression,\npotential of
data loss.\\n\\nDescribe the risk, its severity, and\nmitigation for
each identified\\nrisk. Invite stakeholders and evaluate\nhow to proceed
before merging.\\n\\n- [ ] [See
some\nrisk\\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\\n-\n[
] ...\\n\\n---------\\n\\nCo-authored-by: Elastic
Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"e710f09d0c1159a9018ff6b9dfbac151509213f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.16\",\"label\":\"v8.16.5\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Quynh Nguyen (Quinn)
<43350163+qn895@users.noreply.github.com>"}},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/210217","number":210217,"mergeCommit":{"message":"[Fleet]
Fix unattended Transforms in integration packages not automatically
restarting after reauthorizing (#210217)\n\n## Summary\n\nThis PR
partially addresses an issue
with\nhttps://github.com/elastic/integrations/issues/12486 where the
transform\ndoesn't \"restart\" immediately after reauthorizing. This is
because for\nunattended transform, calling `_start` will come back with
409 transform\nalready started error. So this PR tracks if the transform
has\n`settings.unattended: true`, if yes, stop the transform first
before\nstarting.\n\nWithout this step, the transform will retry again
and become healthy\nagain anyway but it takes longer for that retry to
happen, so this PR\nspeeds up the process of retrying.\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] Any text
added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\nDoes this PR introduce any risks? For example,
consider risks like hard\nto test bugs, performance regression,
potential of data loss.\n\nDescribe the risk, its severity, and
mitigation for each identified\nrisk. Invite stakeholders and evaluate
how to proceed before merging.\n\n- [ ] [See some
risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n-
[ ] ...\n\n---------\n\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"e710f09d0c1159a9018ff6b9dfbac151509213f9"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.16","label":"v8.16.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
  • Loading branch information
qn895 and elasticmachine authored Feb 19, 2025
1 parent f28c44a commit c01a902
Showing 1 changed file with 51 additions and 33 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,16 @@ import type { SavedObjectsClientContract } from '@kbn/core-saved-objects-api-ser
import { sortBy, uniqBy } from 'lodash';
import { isPopulatedObject } from '@kbn/ml-is-populated-object';
import type { ErrorResponseBase } from '@elastic/elasticsearch/lib/api/types';
import pMap from 'p-map';

import type { SecondaryAuthorizationHeader } from '../../../../../common/types/models/transform_api_key';
import { updateEsAssetReferences } from '../../packages/es_assets_reference';
import type { Installation } from '../../../../../common';
import { ElasticsearchAssetType, PACKAGES_SAVED_OBJECT_TYPE } from '../../../../../common';

import { retryTransientEsErrors } from '../retry';

const MAX_CONCURRENT_TRANSFORMS_OPERATIONS = 5;

interface FleetTransformMetadata {
fleet_transform_version?: string;
order?: number;
Expand All @@ -30,6 +32,7 @@ interface FleetTransformMetadata {
last_authorized_by?: string;
run_as_kibana_system?: boolean;
transformId: string;
unattended?: boolean;
}

const isErrorResponse = (arg: unknown): arg is ErrorResponseBase =>
Expand All @@ -41,13 +44,15 @@ async function reauthorizeAndStartTransform({
transformId,
secondaryAuth,
meta,
shouldStopBeforeStart,
}: {
esClient: ElasticsearchClient;
logger: Logger;
transformId: string;
secondaryAuth?: SecondaryAuthorizationHeader;
shouldInstallSequentially?: boolean;
meta?: object;
shouldStopBeforeStart?: boolean;
}): Promise<{ transformId: string; success: boolean; error: null | any }> {
try {
await retryTransientEsErrors(
Expand All @@ -69,6 +74,18 @@ async function reauthorizeAndStartTransform({
}

try {
// For unattended transforms, we need to stop the transform before starting it
// otherwise, starting transform will fail with a 409 error
if (shouldStopBeforeStart) {
await retryTransientEsErrors(
() =>
esClient.transform.stopTransform(
{ transform_id: transformId, wait_for_completion: true },
{ ignore: [404, 409] }
),
{ logger, additionalResponseStatuses: [400] }
);
}
const startedTransform = await retryTransientEsErrors(
() => esClient.transform.startTransform({ transform_id: transformId }, { ignore: [409] }),
{ logger, additionalResponseStatuses: [400] }
Expand Down Expand Up @@ -119,27 +136,23 @@ export async function handleTransformReauthorizeAndStart({
);
}

const transformInfos = await Promise.all(
transforms.map(({ transformId }) =>
retryTransientEsErrors(
() =>
esClient.transform.getTransform(
{
transform_id: transformId,
},
{ ...(secondaryAuth ? secondaryAuth : {}), ignore: [404] }
),
{ logger, additionalResponseStatuses: [400] }
)
)
const transformInfos = await retryTransientEsErrors(
() =>
esClient.transform.getTransform(
{
transform_id: transforms.map((t) => t.transformId).join(','),
},
{ ...(secondaryAuth ? secondaryAuth : {}), ignore: [404] }
),
{ logger, additionalResponseStatuses: [400] }
);

const transformsMetadata: FleetTransformMetadata[] = transformInfos
.flat()
.filter((t) => t.transforms !== undefined)
.map<FleetTransformMetadata>((t) => {
const transform = t.transforms?.[0];
return { ...transform._meta, transformId: transform?.id };
const transformsMetadata: FleetTransformMetadata[] = transformInfos.transforms
.map<FleetTransformMetadata>((transform) => {
return {
...transform._meta,
transformId: transform?.id,
unattended: Boolean(transform.settings?.unattended),
};
})
.filter((t) => t?.run_as_kibana_system === false);

Expand All @@ -155,30 +168,35 @@ export async function handleTransformReauthorizeAndStart({
(t) => t.order,
]);

for (const { transformId, ...meta } of sortedTransformsMetadata) {
for (const { transformId, unattended, ...meta } of sortedTransformsMetadata) {
const authorizedTransform = await reauthorizeAndStartTransform({
esClient,
logger,
transformId,
secondaryAuth,
meta: { ...meta, last_authorized_by: username },
shouldStopBeforeStart: unattended,
});

authorizedTransforms.push(authorizedTransform);
}
} else {
// Else, create & start all the transforms at once for speed
const transformsPromises = transformsMetadata.map(async ({ transformId, ...meta }) => {
return await reauthorizeAndStartTransform({
esClient,
logger,
transformId,
secondaryAuth,
meta: { ...meta, last_authorized_by: username },
});
});

authorizedTransforms = await Promise.all(transformsPromises).then((results) => results.flat());
authorizedTransforms = await pMap(
transformsMetadata,
async ({ transformId, unattended, ...meta }) =>
reauthorizeAndStartTransform({
esClient,
logger,
transformId,
secondaryAuth,
meta: { ...meta, last_authorized_by: username },
shouldStopBeforeStart: unattended,
}),
{
concurrency: MAX_CONCURRENT_TRANSFORMS_OPERATIONS,
}
).then((results) => results.flat());
}

const so = await savedObjectsClient.get<Installation>(PACKAGES_SAVED_OBJECT_TYPE, pkgName);
Expand Down

0 comments on commit c01a902

Please sign in to comment.