Remove initialization optimizations (#226)

This affects:
- `cons`
- `cycle`
- `iter`
- `numbers`
- `seq-index`

Remove optimizations that use iterations variables more directly by initializing
them to the value expected during the first iteration step. Doing that can
change behavior, as iteration variables could be updated again before the loop
is expected to exit, which currently happens for `cl-loop`.

```emacs-lisp
;; => (4 (1 2 3))
(cl-loop for elem in (list 1 2 3)
         for i from 1
         collect i into is
         finally return (list i is))
```

Loopy currently copies that behavior, which could introduce unexpected results
by default. For example, SBCL gives a different result.

```commonlisp
;; => (3 (1 2 3))
(loop for elem in (list 1 2 3)
      for num from 1
      collect num into nums
      finally (return (list num nums)))
```

Making this optimization optional (as we currently do via use of the `with`
special macro argument) means that users would have to remember the internal
implementations of each command with optimization to remember how they change.

See also #204.

If one were to add an `iter-opt` special macro argument and change default
behavior to be expected (more like SBCL), that would only move the burden to
needing to remember when to use `iter-opt`.

As mention in #204, using `iter-opt` requires the user to:

    1. Read the documentation to know that a command supports it

    2. Always remember that the command supports it while using the macro

    3. Always remember how it affects the command while using it


all for a slight gain in speed and marginally less typing.

For those reasons, instead of adding `iter-opt`, it is better to remove these
few optimized behaviors in favor of more standard behavior. As noted in #204,
instead of writing

```emacs-lisp
;; => (5 (1 2 3 4))
(loopy (iter-opt (numbers i))
       (list elem '(1 2 3 4))
       (numbers i :from 1 :to 10)
       (collect i)
       (finally-return i loopy-result))
```

one could instead write

```emacs-lisp
;; => (5 (1 2 3 4))
(loopy (with (i 1))
       (list elem '(1 2 3 4))
       (while (<= i 10))
       (collect i)
       (set i (1+ i))
       (finally-return i loopy-result))
```

which is basically the same length while requiring less knowledge of each
optimized iteration command's internals.

There are already cases where Loopy is not hyper-optimized, such as explicit
accumulation variables. However, there, the behavior of `accum-opt` is much less
of a burden to remember, because it affects every accumulation command in the
same way. There are no special cases to remember.

For these optimizations, it is the case that Loopy's flexibility and consistency
has a small cost, but they are not the only case of that (consider how `cl-loop`
moves some variable setting to `and` expressions while we settle for `catch` and
`throw`) and we do not question the value of consistency and predictability in
those cases.

Author: okamsn

CHANGELOG.md

@@ -72,6 +72,46 @@ version is needed for generic sequences.
   - Allow the `unique` keyword argument of the commands `map` and `map-ref` to be
     evaluable at run time, instead of just checked at compile time ([#209]).
 
+- Remove the initialization optimizations that produced faster code but
+  could change final results ([#226, #204]).  For example, consider the
+  difference results between `cl-loop` and SBCL's `loop`:
+
+  ``` emacs-lisp
+  ;; => (4 (1 2 3))
+  (cl-loop for elem in (list 1 2 3)
+           for i from 1
+           collect i into is
+           finally return (list i is))
+  ```
+
+
+  ``` common-lisp
+  ;; => (3 (1 2 3))
+  (loop for elem in (list 1 2 3)
+        for num from 1
+        collect num into nums
+        finally (return (list num nums)))
+  ```
+
+  Loopy would give the same result as `cl-loop` when using the optimization and
+  would given the same result as SBCL's `loop` when not using the optimization.
+
+  Working around having different results based on unstated (though documented)
+  settings would mean requiring users to fully know the implementations of each
+  loop command and how they could change when optimized.  That position also
+  argues against making use of the optimization more explicit via an added
+  `iter-opt` special macro argument, as discussed in [#226] and [#204].
+
+  Therefore, these optimizations are being removed and Loopy is reverting to its
+  previous behavior of initializing the iteration variables to `nil` by default
+  for the following commands:
+  - `cons`
+  - `cycle`
+  - `iter`
+  - `numbers`
+  - `seq-index`
+  - `substream`
+
 ### Improvements
 
 - The `map` and `map-ref` commands now check for duplicate keys step by step,
@@ -107,6 +147,7 @@ version is needed for generic sequences.
 [#179]: https://github.com/okamsn/loopy/issues/179
 [#184]: https://github.com/okamsn/loopy/issues/184
 [#203]: https://github.com/okamsn/loopy/pull/203
+[#204]: https://github.com/okamsn/loopy/issues/204
 [#205]: https://github.com/okamsn/loopy/pull/205
 [#206]: https://github.com/okamsn/loopy/pull/206
 [#207]: https://github.com/okamsn/loopy/pull/207
@@ -117,6 +158,7 @@ version is needed for generic sequences.
 [#213]: https://github.com/okamsn/loopy/pull/213
 [#215]: https://github.com/okamsn/loopy/pull/215
 [#217]: https://github.com/okamsn/loopy/pull/217
+[#226]: https://github.com/okamsn/loopy/pull/226
 
 ## 0.13.0
 

README.org

@@ -53,6 +53,10 @@ please let me know.
        beginning of the loop.
      - The =:on-failure= argument of the =find= command is now evaluated at the
        beginning of the loop.
+   - Changed back to the old, slightly slower behavior of always initializing
+     iteration variables to ~nil~, instead of sometimes initializing to the
+     expected value during the first iteration step.  This affects =cons=,
+     =cycle=, =iter=, =numbers=, =seq-index=, and =substream=.
  - Version 0.13.0:
    - The deprecated =:init= keyword argument has been removed.  Use the =with=
      special macro argument instead.

doc/loopy-doc.org

@@ -1436,14 +1436,6 @@ In ~loopy~, iteration commands are named after what they iterate through.  For
 example, the =array= and =list= commands iterate through the elements of arrays
 and lists, respectively.
 
-#+ATTR_TEXINFO: :tag Note
-#+begin_quote
-In general, iteration variables (such as the ~i~ and ~j~ above) are initialized
-to ~nil~.  For efficiency, some commands do not do this.  In such cases, the
-initial value of an iteration variable can be set using the =with= special macro
-argument, but this can result in less efficient code.
-#+end_quote
-
 Because some iteration commands use their variable to manage state, it is an
 error to use the same iteration variable for multiple iteration commands.
 
@@ -1454,6 +1446,85 @@ error to use the same iteration variable for multiple iteration commands.
          (finally-return t))
 #+end_src
 
+Iteration variables are initialized to ~nil~ and they are updated at the point
+in the loop body corresponding to the loop command's position in the macro's
+arguments.
+
+#+begin_src emacs-lisp
+  ;; `elem' retains its value from the previous
+  ;; iteration until it is updated again:
+  ;;
+  ;; => (((1 . nil) ; before
+  ;;      (2 . 1)
+  ;;      (3 . 2)
+  ;;      (4 . 3))
+  ;;     ((1 . 1) ; after
+  ;;      (2 . 2)
+  ;;      (3 . 3)
+  ;;      (4 . 4)))
+  (loopy (numbers nth :from 1)
+         (collect elem-before (cons nth elem))
+         (list elem '(1 2 3 4))
+         (collect elem-after (cons nth elem))
+         (finally-return elem-before
+                         elem-after))
+#+end_src
+
+Be aware that ~cl-loop~ does not consistently initialize its iteration variables
+to nil.  For some of ~cl-loop~'s iteration (=for=) statements, the variable is
+initialized to its value for the first iteration step and is manipulated
+directly at the end of the iteration step.  Loopy avoids this, as seen in the
+below example, but that can result in unnecessary indirection for some use
+cases, which has a minor speed cost.
+
+#+begin_src emacs-lisp
+  ;; => (5 (1 2 3 4) (1 2 3 4))
+  (cl-loop for elem in (list 1 2 3 4)
+           collect num into nums-1
+           for num from 1
+           collect num into nums-2
+           finally return (list num nums-1 nums-2))
+
+  ;; => (4 (nil 1 2 3) (1 2 3 4))
+  (loopy (list elem (list 1 2 3 4))
+         (collect nums-1 num)
+         (numbers num :from 1)
+         (collect nums-2 num)
+         (finally-return num nums-1 nums-2))
+#+end_src
+
+Generally, iteration commands with conditions check whether to terminate the
+loop /before/ the next iteration is run.  They do not check their conditions
+while running the current iteration step.  In the below example, note that the
+final value of ~i~ is 2 and not 3, even though the =do= command (similar to
+~cl-loop~'s =do= keyword) is placed before the =list= command.  Even though ~i~
+is updated before ~elem~ is updated, the decision whether to continue the loop,
+based on the =list= command's condition, is made /before/ the code in the =do=
+command is run.
+
+#+begin_src emacs-lisp
+  ;; => 2, not 3
+  (let ((i 0))
+    (loopy (do (setq i (1+ i)))
+           (list elem '(0 1)))
+    i)
+#+end_src
+
+If you do wish to conditionally leave the loop during an iteration, consider
+using the =leave= and =leave-from= commands ([[#exiting-the-loop-early]]).
+
+#+begin_src emacs-lisp
+  ;; => (3 (0 1))
+  (loopy (with (some-list (list 0 1))
+               (i 0))
+         (do (setq i (1+ i)))
+         (when (null some-list)
+           (leave))
+         (collect elems (car some-list))
+         (do (setq some-list (cdr some-list)))
+         (finally-return i elems))
+#+end_src
+
 Unlike ~cl-loop~ and like Common Lisp's ~iterate~, arguments of the iteration
 commands are evaluated only once.  For example, while iterating through numbers,
 you can't suddenly change the direction of the iteration in the middle of the
@@ -1472,12 +1543,15 @@ loop.  This restriction allows for producing more efficient code.
 #+findex: cycling
 #+findex: repeat
 #+findex: repeating
-- =(cycle|repeat [VAR] EXPR)= :: Run the loop for =EXPR= iterations.  If
-  specified, =VAR= starts at 0, and is incremented by 1 at the end of each step
-  in the loop.  If =EXPR= is 0, then the loop isn't run.
+- =(cycle|repeat [VAR] EXPR)= :: Run the loop for =EXPR= iterations.
+
+  If given, then during the loop, =VAR= is set to the number of iteration steps
+  that have been run (0 for the first iteration step).
+
+  If =EXPR= is 0, then the loop isn't run.
 
-  For efficiency, =VAR= is not initialized to ~nil~.  This can be overridden
-  using the =with= special macro argument, which can result in slower code.
+  =(cycle VAR EXPR)= works the same as =(numbers VAR :from 0 :below EXPR)=
+  ([[#numeric-iteration]]).
 
   This command also has the aliases =cycling= and =repeating=.
 
@@ -1493,6 +1567,14 @@ loop.  This restriction allows for producing more efficient code.
            (collect i)
            (collect j))
 
+    ;; Same as above:
+    ;;
+    ;; => (10 0 10 1 10 2)
+    (loopy (with (i 10))
+           (numbers j :from 0 :below 3)
+           (collect i)
+           (collect j))
+
     ;; An argument of 0 stops the loop from running:
     ;; => nil
     (loopy (cycle 0)
@@ -1519,13 +1601,6 @@ loop.  This restriction allows for producing more efficient code.
   once, =yield-result= is an expression which is substituted into the loop body.
   Therefore, =yield-result= can be used to repeatedly call functions.
 
-  For efficiency, when possible, =VAR= is bound to the yielded value before each
-  step of the loop, which is used to detect whether the iterator signals that it
-  is finished.  This is not possible when destructuring.  You can override this
-  behavior by using the =with= special macro argument, which can result in
-  slower code and tells the macro that the initial value of =VAR= is meaningful
-  and to update =VAR= during the loop.
-
   This command also has the name =iterating=.
 
   #+begin_src emacs-lisp
@@ -1632,11 +1707,6 @@ variants =numbers-up= and =numbers-down=.
   =(list i (number-sequence 1 10))=, and =(numbers i 3)= is similar to
   =(set i 3 (1+ i))=.
 
-  For efficiency, _=VAR= is initialized to the starting numeric value_, not
-  ~nil~, and is updated at the end of each step of the loop.  This can be
-  overridden using the =with= special macro argument, which can result in slower
-  code.
-
   In its most basic form, =numbers= iterates from a starting value to an
   inclusive ending value using the =:from= and =:to= keywords, respectively.
 
@@ -1646,6 +1716,34 @@ variants =numbers-up= and =numbers-down=.
            (collect i))
   #+end_src
 
+  Unlike ~cl-loop~, =VAR= is not initialized to the starting value given.
+  Instead, =VAR= is updated during the loop, like in other iteration
+  commands. This avoids unexpectedly changing the value of =VAR= after the
+  iteration step, as happens with some implementations of Common Lisp's ~loop~
+  macro (such ~cl-loop~).
+
+  #+begin_src emacs-lisp
+    ;; => (4 (1 2 3 4))
+    (loopy (list elem (list 1 2 3 4))
+           (numbers num :from 1)
+           (collect nums num)
+           (finally-return num nums))
+
+    ;; => (5 (1 2 3 4))
+    (cl-loop for elem in (list 1 2 3 4)
+             for num from 1
+             collect num into nums
+             finally return (list num nums))
+
+    ;; SBCL returns 4, not 5:
+    ;;
+    ;; => (4 (1 2 3 4))
+    (loop for elem in (list 1 2 3 4)
+          for num from 1
+          collect num into nums
+          finally (return (list num nums)))
+  #+end_src
+
   If the ending value is not given, then the value is incremented by 1 without
   end.
 
@@ -1929,11 +2027,6 @@ source sequences.
 - =(cons|conses VAR EXPR &key by)= :: Loop through the cons cells of =EXPR=.
   Optionally, find the cons cells via the function =by= instead of =cdr=.
 
-  For efficiency, when possible, =VAR= is initialized to the value of =EXPR=,
-  not ~nil~, and is updated at the end of each step in the loop.  This is not
-  possible when destructuring.  Such initialization can be overridden by using
-  the =with= special macro argument, which can result in slower code.
-
   This command also has the alias =consing=.
 
   #+BEGIN_SRC emacs-lisp
@@ -2214,11 +2307,6 @@ source sequences.
   and are compatible with features from the built-in =seq= library, such as
   ~seq-elt~ and ~seq-do~.
 
-  For efficiency, when possible, =VAR= is initialized to the value of =EXPR=,
-  not ~nil~, and is updated at the end of each step in the loop.  This is not
-  possible when destructuring.  Such initialization can be overridden by using
-  the =with= special macro argument, which can result in slower code.
-
   Sub-streams can only be destructured using the =&seq= feature of the default
   destructuring method ([[#basic-destructuring][Basic Destructuring]]), or by using the =seq= flag
   ([[#flags][Using Flags]]).  Streams are neither lists nor arrays.
@@ -2313,11 +2401,6 @@ iterate.
   With =numbers=, one would first need to explicitly calculate the length of the
   sequence.
 
-  Similar to =numbers=, for efficiency, =VAR= is initialized to the starting
-  index value, not ~nil~, and is updated at the end of each step of the loop.
-  This can be overridden using the =with= special macro argument, which can
-  result in slower code.
-
   #+begin_src emacs-lisp
     ;; => (97 98 99 100 101 102)
     (loopy (with (my-string "abcdef"))