Flaky specs

[filipefurtad0]

This epic groups all flaky specs.

Some causes for flaky specs:

1. Moving files around. Some files have been moved in the context of the migration from feature into system specs, and - as point 2 mentions - this can cause flakyness. The first approach therefore should be to update/move all feature specs into system specs. Some guidelines for this:

https://app.gitbook.com/@ofn-user-guide/s/ofn-testing-handbook/automated-testing-handbook/system-specs

2. Sometimes the order of specs matters for flakiness, as pointed out here. Here's a list of reasons why this is so:

• The database cleaner doesn't touch countries and states. If they get modified, that persists.
• Some queries like Spree preferences are cached in memory. Resetting the database doesn't help.
• Some code uses class level variables. If they are modified...
• If code is monkey patched...
• If code is loaded in a different order, e.g. require statements are executed in a different way...

3. Discrepancy in system vs. local times. Some specs fail on the last day of the month, see this case.

4. ? (WIP)

Some references on tackling flaky specs

i) in a system spec context:
https://www.jessespevack.com/2017-10-10-talk-rspec-to-me/

ii) https://nopecode.com/2021/03/03/your-flaky-system-test-again.html

iii) ? (WIP)

Issue list

• [Flaky] spec/system/admin/order_cycles/complex_creating_specific_time_spec.rb #9817
• [Flaky] Ferrum::ProcessTimeoutError #10018
• [Flaky] ./spec/system/admin/order_cycles/complex_editing_multiple_product_pages_spec.rb:37 #10027
• [Flaky] Ferrum::DeadBrowserError: Browser is dead or given window is closed #10220
• [Flaky] spec/system/admin/adjustments_spec.rb #10346
• [Flaky] spec/system/admin/orders_spec.rb #10408
• [Flaky] spec/system/admin/bulk_order_cancellation_spec.rb #10735
• [Flaky] rspec ./spec/system/admin/reports/sales_tax/sales_tax_totals_by_order_spec.rb:433 #10816
• [Flaky] rspec ./spec/system/admin/orders_spec.rb:322 #10953
• [Flaky] rspec ./spec/system/admin/reports_spec.rb #10977
• [Flaky] rspec ./spec/system/consumer/caching/shops_caching_spec.rb:17 #11010
• [Flaky] rspec ./spec/controllers/admin/bulk_line_items_controller_spec.rb:55 #11016
• [Flaky] rspec ./spec/system/admin/payments_spec.rb #11027
• [Flaky] rspec ./spec/system/admin/bulk_product_update_spec.rb:203 #11028
• [Flaky] rspec ./spec/system/admin/order_cycles/simple_spec.rb:572 #11037
• [Flaky] ./spec/system/admin/order_cycles/complex_updating_specific_time_spec.rb (commented out) #11266
• [Flaky] /spec/system/admin/orders_spec.rb #11462
• [Flaky] ./spec/system/consumer/shopping/unit_price_spec.rb #11558
• [Flaky] ./spec/system/admin/order_cycles_complex_nav_check_spec.rb #11669
• [Flaky] spec/system/consumer/white_label_spec.rb #11681
• [Flaky] /spec/system/admin/bulk_order_management_spec #11984
• [Flaky] ./spec/system/admin/reports/orders_and_fulfillment_spec.rb #11581
• [Flaky] ./spec/system/admin/enterprise_fees_spec.rb:136 #10760
• [Flaky] ./spec/requests/api/orders_spec.rb #12388
• [Flaky] ./spec/system/admin/reports/enterprise_fee_summaries_spec.rb #12560
• [Flaky] spec/system/consumer/checkout/tax_not_incl_spec.rb #12645