ToList as Memory<T>

[clipperhouse]

Before, we’d allocate a new List, and then call ToArray on each token. The List is new, and each ToArray is a copy of the ReadOnlySpan representing the token.

In this PR, we clone the original input, and create Memory pointing to each token.

We want the semantics that by calling ToList, the new is now entirely independent of the original input. Copies in all cases. This is to avoid gotchas where the underlying original data may be mutated. I think this is what users expect.

[clipperhouse]

@kevin-montrose @bretcope I don’t know if this is better, see description above. Is it better?

[clipperhouse]

A little faster, but more allocation:

after: 

| Method              | Mean       | Error    | StdDev   | Gen0     | Gen1     | Gen2     | Allocated   |
| ToList              | 1,484.0 us | 15.28 us | 14.29 us | 382.8125 | 373.0469 | 248.0469 | 2,155,937 B |

before:

| Method              | Mean       | Error    | StdDev   | Gen0     | Gen1     | Gen2     | Allocated   |
| ToList              | 1,847.3 us | 6.59 us  | 5.84 us  | 234.3750 | 201.1719 | 105.4688 | 1,522,633 B |

[clipperhouse]

Calling it: never mind, given the above results and the simplicity of the original version.