Merge pull request #7 from tier4/feat/annotationless-scenario-format

feat!: annotationless scenario format
tier4 · Jul 31, 2024 · 817f659 · 817f659
2 parents 718b1f7 + 08345d6
commit 817f659
Show file tree

Hide file tree

Showing 11 changed files with 98 additions and 1,210 deletions.
diff --git a/docs/use_case/annotationless_perception.en.md b/docs/use_case/annotationless_perception.en.md
@@ -75,117 +75,6 @@ Published topics:
 | ---------- | --------- |
 | N/A        | N/A       |
 
-### Method of specifying conditions
-
-The conditions can be given in two ways
-
-#### Describe in scenario
-
-```yaml
-Evaluation:
-  UseCaseName: annotationless_perception
-  UseCaseFormatVersion: 0.3.0
-  Conditions:
-    ClassConditions:
-      # Describe the conditions for each class. If a class with no conditions is output, only the metrics are calculated. It does not affect the evaluation.
-      # In the sample data, the class of TRUCK is also output, but the condition is not described, so TRUCK is always Success.
-      # When specifying conditions from result.jsonl, only keys described here will be updated.
-      # Even though TRUCK metrics appear in result.jsonl, they are not added to the evaluation condition because the TRUCK key is not specified in this example.
-      CAR: # classification key
-        Threshold:
-          # Keys not described will not be evaluated (will always be a success)
-          lateral_deviation: { max: 0.4, mean: 0.019 }
-          yaw_deviation: { max: 3.1411, mean: 0.05 }
-          predicted_path_deviation_5.00: { max: 16.464, mean: 1.8 }
-          total_objects_count_r60.00_h10.00: { metric_value: 10 }
-        PassRange:
-          min: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          max: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          mean: 0.5-2.0 # lower[<=1.0]-upper[>=1.0]
-          metric_value: 0.9-1.1
-      BUS: # classification key
-        Threshold:
-          # Only lateral_deviation is evaluated.
-          yaw_rate: { max: 0.05 } # Only max is evaluated.
-        PassRange:
-          min: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          max: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          mean: 0.5-2.0 # lower[<=1.0]-upper[>=1.0]
-          metric_value: 0.9-1.1
-```
-
-#### Specify by launch argument
-
-This method is assumed to be used mainly.
-
-If the file path of result.jsonl output from past tests is specified, the metrics values from past tests are used as threshold values.
-The values are updated from result.jsonl only for the thresholds listed in the scenario.
-
-The passing range can also be specified as an argument.
-
-An image of its use is shown below.
-
-![threshold](./images/annotationless_threshold.drawio.svg)
-
-##### driving-log-replayer-cli
-
-```shell
-dlr simulation run -p annotationless_perception -l annotationless_threshold_file:=${previous_test_result.jsonl_path} -l 'annotationless_pass_range:={"KEY1":VALUE1"[,"KEY2":"VALUE2"...]}'
-
-# example
-dlr simulation run -p annotationless_perception -l annotationless_threshold_file:=$HOME/out/annotationless/2024-0314-155106/sample/result.jsonl -l 'annotationless_pass_range:={"CAR":{"min":"0.0-1.1","max":"0.0-1.2","mean":"0.5-1.3"},"BUS":{"min":"0.0-1.1","max":"0.0-1.2","mean":"0.5-1.3"}}'
-```
-
-##### WebAutoCLI
-
-```shell
-webauto ci scenario run --project-id ${project-id} --scenario-id ${scenario-id} --scenario-version-id ${scenario-version-id} --simulator-parameter-overrides 'annotationless_threshold_file=${previous_test_result.jsonl_path},annotationless_pass_range:={"KEY1":VALUE1"[,"KEY2":"VALUE2"...]}'
-```
-
-##### Autoware Evaluator
-
-Add to parameters in the simulator configuration in `.webauto-ci.yml`.
-
-```yaml
-simulations:
-  - name: annotationless_perception
-    type: annotationless_perception
-    simulator:
-      deployment:
-        type: container
-        artifact: main
-      runtime:
-        type: simulator/standard1/amd64/medium
-      parameters:
-        annotationless_threshold_file: ${previous_test_result.jsonl_path}
-        annotationless_pass_range:
-          KEY1: VALUE1
-          KEY2: VALUE2
-```
-
-#### How to update scenario conditions
-
-The driving-log-replayer-cli has the ability to run multiple scenarios in succession that exist under the data_directory of a profile.
-On the other hand, when evaluation conditions are given as arguments, the same arguments are applied to multiple scenarios, which is inconvenient.
-
-In the case of local testing using driving-log-replayer-cli, instead of specifying arguments, the following commands are provided so that scenario conditions can be updated as needed.
-
-- update-condition command to manually update scenario conditions
-- run's -u option to automatically update scenario conditions after a simulation run
-
-There are two ways to update
-
-- existing Update only those items that appear in the scenario
-- all Update all values in the metrics
-
-```shell
-# manual update
-dlr simulation update-condition -s ${scenario_path} -r ${result.jsonl_path} -u ${existing|all}
-
-# automatically update scenario after simulation run
-dlr simulation run -p annotationless_perception -u ${existing|all}
-```
-
 ## Arguments passed to logging_simulator.launch
 
 - localization: false

diff --git a/docs/use_case/annotationless_perception.ja.md b/docs/use_case/annotationless_perception.ja.md
@@ -75,117 +75,6 @@ Published topics:
 | ---------- | --------- |
 | N/A        | N/A       |
 
-### 条件を指定する方法
-
-条件は以下の2通りの方法で与えることができる
-
-#### シナリオに記述する
-
-```yaml
-Evaluation:
-  UseCaseName: annotationless_perception
-  UseCaseFormatVersion: 0.3.0
-  Conditions:
-    ClassConditions:
-      # クラス毎の条件を記述する。条件を設定がないクラスが出力された場合はメトリクスだけ計算される。評価には影響しない
-      # サンプルデータではTRUCKのclassも出力されるが条件を記述してないので、TRUCKは必ずSuccessになる
-      # result.jsonlから条件指定を行う場合、ここに記述があるキーのみ更新される。
-      # result.jsonlにTRUCKのメトリクスが出ていても、この例ではTRUCKのキーを指定していないので評価条件には追加されない。
-      CAR: # classification key
-        Threshold:
-          # 記述のないキーについては評価されない（必ず成功になる）
-          lateral_deviation: { max: 0.4, mean: 0.019 }
-          yaw_deviation: { max: 3.1411, mean: 0.05 }
-          predicted_path_deviation_5.00: { max: 16.464, mean: 1.8 }
-          total_objects_count_r60.00_h10.00: { metric_value: 10 }
-        PassRange:
-          min: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          max: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          mean: 0.5-2.0 # lower[<=1.0]-upper[>=1.0]
-          metric_value: 0.9-1.1
-      BUS: # classification key
-        Threshold:
-          # Only lateral_deviation is evaluated.
-          yaw_rate: { max: 0.05 } # Only max is evaluated.
-        PassRange:
-          min: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          max: 0.0-2.0 # lower[<=1.0]-upper[>=1.0]
-          mean: 0.5-2.0 # lower[<=1.0]-upper[>=1.0]
-          metric_value: 0.9-1.1
-```
-
-#### launch引数で指定する
-
-こちらの方法をメインに使う想定。
-
-過去のテストで出力されたresult.jsonlのファイルパスを指定すると、過去のテストのメトリクス値を閾値として利用する。
-シナリオに記載のある閾値だけresult.jsonlから値が更新される。
-
-また合格範囲も引数で指定可能。
-
-利用イメージを以下に示す。
-
-![threshold](./images/annotationless_threshold.drawio.svg)
-
-##### driving-log-replayer-cli
-
-```shell
-dlr simulation run -p annotationless_perception -l annotationless_threshold_file:=${previous_test_result.jsonl_path} -l 'annotationless_pass_range:={"KEY1":VALUE1"[,"KEY2":"VALUE2"...]}'
-
-# example
-dlr simulation run -p annotationless_perception -l annotationless_threshold_file:=$HOME/out/annotationless/2024-0314-155106/sample/result.jsonl -l 'annotationless_pass_range:={"CAR":{"min":"0.0-1.1","max":"0.0-1.2","mean":"0.5-1.3"},"BUS":{"min":"0.0-1.1","max":"0.0-1.2","mean":"0.5-1.3"}}'
-```
-
-##### WebAutoCLI
-
-```shell
-webauto ci scenario run --project-id ${project-id} --scenario-id ${scenario-id} --scenario-version-id ${scenario-version-id} --simulator-parameter-overrides 'annotationless_threshold_file=${previous_test_result.jsonl_path},annotationless_pass_range:={"KEY1":VALUE1"[,"KEY2":"VALUE2"...]}'
-```
-
-##### Autoware Evaluator
-
-.webauto-ci.ymlのsimulatorの設定でparametersに追加する。
-
-```yaml
-simulations:
-  - name: annotationless_perception
-    type: annotationless_perception
-    simulator:
-      deployment:
-        type: container
-        artifact: main
-      runtime:
-        type: simulator/standard1/amd64/medium
-      parameters:
-        annotationless_threshold_file: ${previous_test_result.jsonl_path}
-        annotationless_pass_range:
-          KEY1: VALUE1
-          KEY2: VALUE2
-```
-
-#### シナリオの条件を更新する方法
-
-driving-log-replayer-cliではプロファイルのdata_directory配下に存在する複数のシナリオを連続で実行する機能を備えている。
-一方で、評価条件を引数で与える場合、複数のシナリオに対して同じ引数が適用されてしまい不都合がある。
-
-driving-log-replayer-cliを用いたローカルテストの場合は、引数指定ではなく、シナリオの条件を随時更新して利用できるように、以下のようなコマンドが用意されている。
-
-- 手動でシナリオの条件を更新するupdate-conditionコマンド
-- シミュレーション実行後に自動でシナリオの条件を更新するrunの-uのオプション
-
-更新方法は以下の2通りがある
-
-- existing シナリオに記述のある項目のみ更新
-- all メトリクスに出ている値で全て更新
-
-```shell
-# manual update
-dlr simulation update-condition -s ${scenario_path} -r ${result.jsonl_path} -u ${existing|all}
-
-# automatically update scenario after simulation run
-dlr simulation run -p annotationless_perception -u ${existing|all}
-```
-
 ## logging_simulator.launch に渡す引数
 
 - localization: false

diff --git a/log_evaluator/launch/log_evaluator.launch.py b/log_evaluator/launch/log_evaluator.launch.py
@@ -128,10 +128,6 @@ def ensure_arg_compatibility(context: LaunchContext) -> list:
             conf[component] = use_case_launch_arg[component]
         launch_component[component] = conf.get(component, "true")
 
-    # annotationless
-    conf["annotationless_threshold_file"] = ""
-    conf["annotationless_pass_range"] = ""
-
     # create output directory
     output_dir.mkdir(exist_ok=True, parents=True)