Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Proposal] Tests for all envs #2767

Closed
jjshoots opened this issue Apr 20, 2022 · 2 comments
Closed

[Proposal] Tests for all envs #2767

jjshoots opened this issue Apr 20, 2022 · 2 comments

Comments

@jjshoots
Copy link
Contributor

Proposal

There should be more comprehensive tests for all envs. Probably the easiest test would be similar to that in PettingZoo where every state of a rollout is asserted to be contained in the observation space.

There should also be tests that assert the behaviour of environments where this can be tested. One example is the heuristic landing for Lunar Lander (which currently only works for the non-winded case).
@pseudo-rnd-thoughts
Copy link
Contributor

pseudo-rnd-thoughts commented Apr 26, 2022 •
edited
Loading

Proposed changes

  • Update thespec_list to remove atari and provide a full spec list and a no-mujoco spec list
  • Remove all atari tests - Currently atari is not being installed so there is no point in the tests being there. [Proposal] Add test coverage #2734 (comment)
  • Upgrade the action dim test - Check that environments do not allow out of bounds actions, i.e. extra dims, outside observation space
  • Upgrade the environment determinism - Improve the number of time steps that we environment step equivalence. This will also test the environments can step / rollout
  • Simplify the observation shape check - This could be part of the determinism check to help improve performance and reduce test time
  • Test render - All of the render modes for environments return the expected output
  • Test kwargs for classic control, box2d, and toy text, some of these environments have kwargs, hardcore, randomize, etc. Currently, there are no tests that the kwargs affect the environments "correctly" however this is difficult to test. There is a frozenlake DFS map generation test already for this
  • Mujoco - There is a mujoco v2 to v3 tests however it is unclear if this is unneeded anymore with v4 coming out. Possibly add new v4 tests
  • Registration - These tests are actually in a good condition, I believe so these will be kept

@gianlucadecola
Copy link
Contributor

I'm going to take a look at some of these points, I'll start with the action dim tests

This was referenced May 5, 2022
Merged
Closed
Merged
jkterry1 pushed a commit that referenced this issue May 12, 2022
@gianlucadecola
Upgrade test_determinism. Refer to #2767 (#2814)
350a082
jkterry1 pushed a commit that referenced this issue May 16, 2022
@gianlucadecola
Upgrade test_action_dim. Refer to #2767 (#2802)
6112b0d 
* Add action in bound checks.

* add correct dtype to clip_action test.

* ensure correct dtype in tests.

* action check in car_racing no more needed.

* Prune PR. parametrize tests with spec_list.

* revert changes.

* remove file.

* remove magic number.

* test both upper and lower bound for every action.

* assert oob is really out of bound. also bugfix in action performed

* valide car racing discrete action. Add docstrings. Include car racing discrete in tests.

Co-authored-by: Gianluca De Cola <gianluca.decola@ags-it.com>
@gianlucadecola gianlucadecola mentioned this issue May 26, 2022
Merged
jkterry1 pushed a commit that referenced this issue Jun 2, 2022
@gianlucadecola
Test BipedalWalker hardcore kwargs. Refer to #2767 (#2841)
83dac2c 
* test hardcore environment creation.

* add type hint.

* typo.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pseudo-rnd-thoughts @jkterry1 @jjshoots @gianlucadecola