Preserve bad header values as strings rather than losing them

[chris-morgan]

At present, HeaderConvertible::from_stream returns Option<Self> and no track is kept of the bytes that have been read, with the result that bad header values are lost to the world. This is probably not an ideal state of things, although being able to read the header value iteratively without an extra allocation is good for performance. All up, I'm afraid it's probably best to read the entire header value up front. from_stream can continue to return Option<Self>, but something, somewhere, (the HeaderValueByteIterator, perhaps?) should retain the bytes that have been read so that the bad value can be obtained.

The resulting value that goes in the HeaderCollection (currently Option<T>) needs to be something like this:

pub enum HeaderValue<T> {
    Missing,
    InvalidValue(~str),
    ValidValue(T),
}

Another option would be Option<Result<T, ~str>>.

This probably suggests a total redesign of HeaderValueByteIterator.

An interesting reference from RFC 2616, §14.21 (about the Expires header), which demonstrates how invalid values must be treated as distinct from missing values:

HTTP/1.1 clients and caches MUST treat other invalid date formats,
especially including the value "0", as in the past (i.e., "already
expired").

(This does not mandate retaining the invalid value, but I think it's probably going to be best.)

I think we should expect people to normally treat an invalid value and a missing value the same, only using the valid value, but there are clearly certain cases where we need to at least identify that there was an invalid value.

[eerden]

I started working on this.

https://github.com/eerden/rust-http/tree/preserve_bad_header