The decoder allows reading individual elements from a file without loading all file data to memory all at once.
+This decreases memory usage, which is especially useful for large files.
+Elements can also be read all at once, and corrupted files can be read until the first decoding error occurs.
+
+
The class internally uses BinaryStreamDecoder to encode the individual elements,
+which can also be used independently to decode the data for more complex operations.
+
+
Handling corrupted data
+
+
The binary format does not necessarily allow detection of data corruption, and various errors can occur
+as the result of added, changed, or missing bytes. Additional measures should be applied if there is an
+increased risk of data corruption.
+
+
As an example, consider the simple encoding of a String inside a struct, which consists of a key
+followed by the length of the string in bytes, and the string content. The length of the string is encoded using
+variable length encoding, so a single bit flip (in the MSB of the length byte) could result in a very large length being decoded,
+causing the decoder to wait for a very large number of bytes to decode the string. This simple error would cause much
+data to be skipped. At the same time, it is not possible to determine with certainty where the error occured.
+
+
The library does therefore only provide hints about the decoding errors likely occuring from non-conformance to the binary format
+or version incompatibility, which are not necessarily the true causes of the failures when data corruption is present.
Stream decoding can be used when either the data is not regularly available completely (e.g. when loading data over a network connection),
+or when the binary data should not be loaded into memory all at once (e.g. when parsing a file).
+
+
Each stream decoder handles only elements of a single type.
+The elements are handed to a handler passed to the decoder upon initialization.
+The byte stream is managed by a provider also specified on object creation.
+The decoder can then attempt to read elements by reading bytes from the provider, until a complete element can be decoded.
+Buffering is handled internally, freeing the stream provider from this responsibility.
+Each completely decoded element is immediatelly passed to handler for further processing.
Encode elements sequentially into a binary data stream.
+
+
A stream encoder is used to encode individual elements of the same type to a continuous binary stream,
+which can be decoded sequentially.
+
+
The encoding behaviour is different to BinaryEncoder, where the full data must be present to successfully decode.
+Additional information is embedded into the stream to facilitate this behaviour.
+The binary data produced by a stream encoder is not compatible with BinaryDecoder and can only be decoded using
+BinaryStreamDecoder.
+
+
The special data format of an encoded stream also allows joining sequences of encoded data, where:
+encode([a,b]) + encode([c,d]) == encode([a,b,c,d]) and decode(encode([a]) + encode([b])) == [a,b]
Add a set of indices for nil values in unkeyed containers.
+
+
This option changes the encoding of unkeyed sequences like arrays with optional values.
+
+
If this option is set to true, then the encoded binary data first contains a list of indexes for each position where nil is encoded.
+After this data the remaining (non-nil) values are added.
+If this option is false, then each value is prepended with a byte 1 for non-nil values, and a byte 0 for nil values.
+
+
Using an index set is generally more efficient, expect for large sequences with many nil values.
+An index set is encoded using first the number of elements, and then each element, all encoded as var-ints.
+
+
One benefit of this option is that top-level sequences can be joined using their binary data, where encoded([a,b]) | encoded([c,d]) == encoded([a,b,c,d]).
+
+
Note
+ This option defaults to true
+
+
+
Note
+ To decode successfully, the decoder must use the same setting for containsNilIndexSetForUnkeyedContainers.
+
+
The decoder allows reading individual elements from a file without loading all file data to memory all at once.
+This decreases memory usage, which is especially useful for large files.
+Elements can also be read all at once, and corrupted files can be read until the first decoding error occurs.
+
+
The class internally uses BinaryStreamDecoder to encode the individual elements,
+which can also be used independently to decode the data for more complex operations.
+
+
Handling corrupted data
+
+
The binary format does not necessarily allow detection of data corruption, and various errors can occur
+as the result of added, changed, or missing bytes. Additional measures should be applied if there is an
+increased risk of data corruption.
+
+
As an example, consider the simple encoding of a String inside a struct, which consists of a key
+followed by the length of the string in bytes, and the string content. The length of the string is encoded using
+variable length encoding, so a single bit flip (in the MSB of the length byte) could result in a very large length being decoded,
+causing the decoder to wait for a very large number of bytes to decode the string. This simple error would cause much
+data to be skipped. At the same time, it is not possible to determine with certainty where the error occured.
+
+
The library does therefore only provide hints about the decoding errors likely occuring from non-conformance to the binary format
+or version incompatibility, which are not necessarily the true causes of the failures when data corruption is present.
Stream decoding can be used when either the data is not regularly available completely (e.g. when loading data over a network connection),
+or when the binary data should not be loaded into memory all at once (e.g. when parsing a file).
+
+
Each stream decoder handles only elements of a single type.
+The elements are handed to a handler passed to the decoder upon initialization.
+The byte stream is managed by a provider also specified on object creation.
+The decoder can then attempt to read elements by reading bytes from the provider, until a complete element can be decoded.
+Buffering is handled internally, freeing the stream provider from this responsibility.
+Each completely decoded element is immediatelly passed to handler for further processing.
Encode elements sequentially into a binary data stream.
+
+
A stream encoder is used to encode individual elements of the same type to a continuous binary stream,
+which can be decoded sequentially.
+
+
The encoding behaviour is different to BinaryEncoder, where the full data must be present to successfully decode.
+Additional information is embedded into the stream to facilitate this behaviour.
+The binary data produced by a stream encoder is not compatible with BinaryDecoder and can only be decoded using
+BinaryStreamDecoder.
+
+
The special data format of an encoded stream also allows joining sequences of encoded data, where:
+encode([a,b]) + encode([c,d]) == encode([a,b,c,d]) and decode(encode([a]) + encode([b])) == [a,b]
Call this function to convert an element into binary data whenever new elements are available.
+The data provided as the result of this function should be processed (e.g. stored or transmitted) while conserving the
+order of the chunks, so that decoding can work reliably.
+
+
Pass the encoded data back into an instance of BinaryStreamDecoder to convert each chunk back to an element.
+
+
Note
+ Data encoded by this function can only be decoded by an appropriate BinaryStreamDecoder.
+Decoding using a simple BinaryDecoder will not be successful.
+
+
Note: The BinaryCodable format is optimized for size, but does not go all-out to create the smallest binary sizes possible. If this is your goal, then simply using Codable with it’s key-value approach will not be the best solution. An unkeyed format optimized for the actually encoded data will be more suitable. But if you’re really looking into this kind of efficiency, then you probably know this already.
+
Note: The BinaryCodable format is optimized for size, but does not go all-out to create the smallest binary sizes possible.
+The binary format, while being efficient, needs to serve as a general-purpose encoding, which will never be as efficient than a custom format optimized for a very specific use case.
+If this is your goal, then simply using Codable with it’s key-value approach will not be the best solution. An unkeyed format optimized for the actually encoded data will be more suitable. But if you’re really looking into this kind of efficiency, then you probably know this already.
The encoding format used by BinaryCodable is similar to Google’s Protocol Buffers in some aspects, but provides much more flexibility regarding the different types which can be encoded, including the ability to encode Optional, Set, single values, multidimensional Arrays, and more.
Integer encoding
@@ -188,7 +202,18 @@
Arrays
Arrays (and other sequences) are encoded by converting each item to binary data, and concatenating the results. Elements with variable length (like String) are prepended with their length encoded as a Varint. Each encoded array has at least one byte prepended to it, in order to support optional values.
Arrays of Optionals
-
It is possible to encode arrays where the elements are Optional, e.g. [Bool?]. Due to constraints regarding Apple’s implementation of Encoder and Decoder, it is not consistently possible to infer if optionals are present in unkeyed containers. BinaryCodable therefore encodes optionals using a different strategy: Each array binary representation is prepended with a “nil index set”. It first consists of the number of nil elements in the sequence, encoded as a Varint. Then follow the indices in the array where nil values are present, each encoded as a Varint. The decoder can then first parse this nil set, and return the appropriate value for each position where a nil value is encoded. This approach is fairly efficient while only few nil values are encoded, or while the sequence doesn’t contain a large number of elements. For arrays that don’t contain optionals, only a single byte (0) is prepended to the binary representation, to signal that there are no nil indices in the sequence.
+
It is possible to encode arrays where the elements are Optional, e.g. [Bool?]. Due to constraints regarding Apple’s implementation of Encoder and Decoder, it is not consistently possible to infer if optionals are present in unkeyed containers. BinaryCodable therefore encodes optionals using a different strategy: Each array binary representation is prepended with a “nil index set”. It first consists of the number of nil elements in the sequence, encoded as a Varint. Then follow the indices in the array where nil values are present, each encoded as a Varint. The decoder can then first parse this nil set, and return the appropriate value for each position where a nil value is encoded. This approach is fairly efficient while only few nil values are encoded, or while the sequence doesn’t contain a large number of elements.
+For arrays that don’t contain optionals, only a single byte (0) is prepended to the binary representation, to signal that there are no nil indices in the sequence.
+
+
If the prependNilIndexSetForUnkeyedContainers is set to false, then this behaviour is changed.
+The encoder then omits the nil index set, and instead adds a single byte 0x01 before each non-nil element, and a single 0x00 byte to signal nil.
+
+
Note
+ One benefit of this option is that top-level sequences can be joined using their binary data, where encoded([a,b]) | encoded([c,d]) == encoded([a,b,c,d]).
+
+
+
+
More efficient ways could be devised to handle arrays of optionals, like specifying the number of nil or non-nil elements following one another, but the increased encoding and decoding complexity don’t justify these gains in communication efficiency.
Structs
Structs are encoded using Codable‘s KeyedEncodingContainer, which uses String or Int coding keys to distinguish the properties of the types.
@@ -449,11 +474,27 @@
Dictionaries with Intege
Dictionaries with String keys
For dictionaries with String keys ([String: ...]), the process is similar to the above, except with CodingKeys having the stringValue of the key. There is another weird exception though: Whenever a String can be represented by an integer (i.e. when String(key) != nil), then the corresponding CodingKey will have its integerValue also set. This means that for dictionaries with integer keys, there may be a mixture of integer and string keys present in the binary data, depending on the input values. But don’t worry, BinaryCodable will also handle these cases correctly.
+
Stream encoding
+
+
The encoding for data streams is only differs from standard encoding in two key aspects.
+
Added length information
+
+
Each top-level element is encoded as if it is part of an unkeyed container (which it essentially is), meaning that each element has the necessary length information prepended to determine it’s size.
+Only types with data type variable length have their length prepended using varint encoding.
+This concerns String and Data, as well as complex types like structs and arrays, among others.
+
Optionals
+
+
Normally, Optional values in unkeyed containers are tracked using nil index sets, which is prepended to the list of non-optionals.
+This approach is not possible for streams, requiring additional information for each element in the stream.
+A single byte is prepended to each Optional element, where binary 0x01 is used to indicate a non-optional value, and 0x00 is used to signal an optional value.
+nil values have no additional data, so each is encoded using one byte.
The decoder allows reading individual elements from a file without loading all file data to memory all at once.
+This decreases memory usage, which is especially useful for large files.
+Elements can also be read all at once, and corrupted files can be read until the first decoding error occurs.
+
+
The class internally uses BinaryStreamDecoder to encode the individual elements,
+which can also be used independently to decode the data for more complex operations.
+
+
Handling corrupted data
+
+
The binary format does not necessarily allow detection of data corruption, and various errors can occur
+as the result of added, changed, or missing bytes. Additional measures should be applied if there is an
+increased risk of data corruption.
+
+
As an example, consider the simple encoding of a String inside a struct, which consists of a key
+followed by the length of the string in bytes, and the string content. The length of the string is encoded using
+variable length encoding, so a single bit flip (in the MSB of the length byte) could result in a very large length being decoded,
+causing the decoder to wait for a very large number of bytes to decode the string. This simple error would cause much
+data to be skipped. At the same time, it is not possible to determine with certainty where the error occured.
+
+
The library does therefore only provide hints about the decoding errors likely occuring from non-conformance to the binary format
+or version incompatibility, which are not necessarily the true causes of the failures when data corruption is present.
Stream decoding can be used when either the data is not regularly available completely (e.g. when loading data over a network connection),
+or when the binary data should not be loaded into memory all at once (e.g. when parsing a file).
+
+
Each stream decoder handles only elements of a single type.
+The elements are handed to a handler passed to the decoder upon initialization.
+The byte stream is managed by a provider also specified on object creation.
+The decoder can then attempt to read elements by reading bytes from the provider, until a complete element can be decoded.
+Buffering is handled internally, freeing the stream provider from this responsibility.
+Each completely decoded element is immediatelly passed to handler for further processing.
Encode elements sequentially into a binary data stream.
+
+
A stream encoder is used to encode individual elements of the same type to a continuous binary stream,
+which can be decoded sequentially.
+
+
The encoding behaviour is different to BinaryEncoder, where the full data must be present to successfully decode.
+Additional information is embedded into the stream to facilitate this behaviour.
+The binary data produced by a stream encoder is not compatible with BinaryDecoder and can only be decoded using
+BinaryStreamDecoder.
+
+
The special data format of an encoded stream also allows joining sequences of encoded data, where:
+encode([a,b]) + encode([c,d]) == encode([a,b,c,d]) and decode(encode([a]) + encode([b])) == [a,b]
Add a set of indices for nil values in unkeyed containers.
+
+
This option changes the encoding of unkeyed sequences like arrays with optional values.
+
+
If this option is set to true, then the encoded binary data first contains a list of indexes for each position where nil is encoded.
+After this data the remaining (non-nil) values are added.
+If this option is false, then each value is prepended with a byte 1 for non-nil values, and a byte 0 for nil values.
+
+
Using an index set is generally more efficient, expect for large sequences with many nil values.
+An index set is encoded using first the number of elements, and then each element, all encoded as var-ints.
+
+
One benefit of this option is that top-level sequences can be joined using their binary data, where encoded([a,b]) | encoded([c,d]) == encoded([a,b,c,d]).
+
+
Note
+ This option defaults to true
+
+
+
Note
+ To decode successfully, the decoder must use the same setting for containsNilIndexSetForUnkeyedContainers.
+
+
The decoder allows reading individual elements from a file without loading all file data to memory all at once.
+This decreases memory usage, which is especially useful for large files.
+Elements can also be read all at once, and corrupted files can be read until the first decoding error occurs.
+
+
The class internally uses BinaryStreamDecoder to encode the individual elements,
+which can also be used independently to decode the data for more complex operations.
+
+
Handling corrupted data
+
+
The binary format does not necessarily allow detection of data corruption, and various errors can occur
+as the result of added, changed, or missing bytes. Additional measures should be applied if there is an
+increased risk of data corruption.
+
+
As an example, consider the simple encoding of a String inside a struct, which consists of a key
+followed by the length of the string in bytes, and the string content. The length of the string is encoded using
+variable length encoding, so a single bit flip (in the MSB of the length byte) could result in a very large length being decoded,
+causing the decoder to wait for a very large number of bytes to decode the string. This simple error would cause much
+data to be skipped. At the same time, it is not possible to determine with certainty where the error occured.
+
+
The library does therefore only provide hints about the decoding errors likely occuring from non-conformance to the binary format
+or version incompatibility, which are not necessarily the true causes of the failures when data corruption is present.
Stream decoding can be used when either the data is not regularly available completely (e.g. when loading data over a network connection),
+or when the binary data should not be loaded into memory all at once (e.g. when parsing a file).
+
+
Each stream decoder handles only elements of a single type.
+The elements are handed to a handler passed to the decoder upon initialization.
+The byte stream is managed by a provider also specified on object creation.
+The decoder can then attempt to read elements by reading bytes from the provider, until a complete element can be decoded.
+Buffering is handled internally, freeing the stream provider from this responsibility.
+Each completely decoded element is immediatelly passed to handler for further processing.
Encode elements sequentially into a binary data stream.
+
+
A stream encoder is used to encode individual elements of the same type to a continuous binary stream,
+which can be decoded sequentially.
+
+
The encoding behaviour is different to BinaryEncoder, where the full data must be present to successfully decode.
+Additional information is embedded into the stream to facilitate this behaviour.
+The binary data produced by a stream encoder is not compatible with BinaryDecoder and can only be decoded using
+BinaryStreamDecoder.
+
+
The special data format of an encoded stream also allows joining sequences of encoded data, where:
+encode([a,b]) + encode([c,d]) == encode([a,b,c,d]) and decode(encode([a]) + encode([b])) == [a,b]
Call this function to convert an element into binary data whenever new elements are available.
+The data provided as the result of this function should be processed (e.g. stored or transmitted) while conserving the
+order of the chunks, so that decoding can work reliably.
+
+
Pass the encoded data back into an instance of BinaryStreamDecoder to convert each chunk back to an element.
+
+
Note
+ Data encoded by this function can only be decoded by an appropriate BinaryStreamDecoder.
+Decoding using a simple BinaryDecoder will not be successful.
+
+
Note: The BinaryCodable format is optimized for size, but does not go all-out to create the smallest binary sizes possible. If this is your goal, then simply using Codable with it’s key-value approach will not be the best solution. An unkeyed format optimized for the actually encoded data will be more suitable. But if you’re really looking into this kind of efficiency, then you probably know this already.
+
Note: The BinaryCodable format is optimized for size, but does not go all-out to create the smallest binary sizes possible.
+The binary format, while being efficient, needs to serve as a general-purpose encoding, which will never be as efficient than a custom format optimized for a very specific use case.
+If this is your goal, then simply using Codable with it’s key-value approach will not be the best solution. An unkeyed format optimized for the actually encoded data will be more suitable. But if you’re really looking into this kind of efficiency, then you probably know this already.
The encoding format used by BinaryCodable is similar to Google’s Protocol Buffers in some aspects, but provides much more flexibility regarding the different types which can be encoded, including the ability to encode Optional, Set, single values, multidimensional Arrays, and more.
Integer encoding
@@ -188,7 +202,18 @@
Arrays
Arrays (and other sequences) are encoded by converting each item to binary data, and concatenating the results. Elements with variable length (like String) are prepended with their length encoded as a Varint. Each encoded array has at least one byte prepended to it, in order to support optional values.
Arrays of Optionals
-
It is possible to encode arrays where the elements are Optional, e.g. [Bool?]. Due to constraints regarding Apple’s implementation of Encoder and Decoder, it is not consistently possible to infer if optionals are present in unkeyed containers. BinaryCodable therefore encodes optionals using a different strategy: Each array binary representation is prepended with a “nil index set”. It first consists of the number of nil elements in the sequence, encoded as a Varint. Then follow the indices in the array where nil values are present, each encoded as a Varint. The decoder can then first parse this nil set, and return the appropriate value for each position where a nil value is encoded. This approach is fairly efficient while only few nil values are encoded, or while the sequence doesn’t contain a large number of elements. For arrays that don’t contain optionals, only a single byte (0) is prepended to the binary representation, to signal that there are no nil indices in the sequence.
+
It is possible to encode arrays where the elements are Optional, e.g. [Bool?]. Due to constraints regarding Apple’s implementation of Encoder and Decoder, it is not consistently possible to infer if optionals are present in unkeyed containers. BinaryCodable therefore encodes optionals using a different strategy: Each array binary representation is prepended with a “nil index set”. It first consists of the number of nil elements in the sequence, encoded as a Varint. Then follow the indices in the array where nil values are present, each encoded as a Varint. The decoder can then first parse this nil set, and return the appropriate value for each position where a nil value is encoded. This approach is fairly efficient while only few nil values are encoded, or while the sequence doesn’t contain a large number of elements.
+For arrays that don’t contain optionals, only a single byte (0) is prepended to the binary representation, to signal that there are no nil indices in the sequence.
+
+
If the prependNilIndexSetForUnkeyedContainers is set to false, then this behaviour is changed.
+The encoder then omits the nil index set, and instead adds a single byte 0x01 before each non-nil element, and a single 0x00 byte to signal nil.
+
+
Note
+ One benefit of this option is that top-level sequences can be joined using their binary data, where encoded([a,b]) | encoded([c,d]) == encoded([a,b,c,d]).
+
+
+
+
More efficient ways could be devised to handle arrays of optionals, like specifying the number of nil or non-nil elements following one another, but the increased encoding and decoding complexity don’t justify these gains in communication efficiency.
Structs
Structs are encoded using Codable‘s KeyedEncodingContainer, which uses String or Int coding keys to distinguish the properties of the types.
@@ -449,11 +474,27 @@
Dictionaries with Intege
Dictionaries with String keys
For dictionaries with String keys ([String: ...]), the process is similar to the above, except with CodingKeys having the stringValue of the key. There is another weird exception though: Whenever a String can be represented by an integer (i.e. when String(key) != nil), then the corresponding CodingKey will have its integerValue also set. This means that for dictionaries with integer keys, there may be a mixture of integer and string keys present in the binary data, depending on the input values. But don’t worry, BinaryCodable will also handle these cases correctly.
+
Stream encoding
+
+
The encoding for data streams is only differs from standard encoding in two key aspects.
+
Added length information
+
+
Each top-level element is encoded as if it is part of an unkeyed container (which it essentially is), meaning that each element has the necessary length information prepended to determine it’s size.
+Only types with data type variable length have their length prepended using varint encoding.
+This concerns String and Data, as well as complex types like structs and arrays, among others.
+
Optionals
+
+
Normally, Optional values in unkeyed containers are tracked using nil index sets, which is prepended to the list of non-optionals.
+This approach is not possible for streams, requiring additional information for each element in the stream.
+A single byte is prepended to each Optional element, where binary 0x01 is used to indicate a non-optional value, and 0x00 is used to signal an optional value.
+nil values have no additional data, so each is encoded using one byte.
All possible errors occuring during encoding produce BinaryEncodingError errors, while unsuccessful decoding produces BinaryDecodingErrors.
Both are enums with several cases describing the nature of the error.
See the documentation of the types to learn more about the different error conditions.
+
Handling corrupted data
+
+
The binary format provides no provisions to detect data corruption, and various errors can occur as the result of added, changed, or missing bytes and bits.
+Additional external measures (checksums, error-correcting codes, …) should be applied if there is an increased risk of data corruption.
+
+
As an example, consider the simple encoding of a String inside a struct, which consists of a key followed by the length of the string in bytes, and the string content.
+The length of the string is encoded using variable-length encoding, so a single bit flip (in the MSB of the length byte) could result in a very large length being decoded, causing the decoder to wait for a very large number of bytes to decode the string.
+This simple error would cause much data to be skipped, potentially corrupting the data stream indefinitely.
+At the same time, it is not possible to determine with certainty where the error occured, making error recovery difficult without additional information about boundaries between elements.
+
+
The decoding errors provided by the library are therefore only hints about error likely occuring from non-conformance to the binary format or version incompatibility, which are not necessarily the true causes of the failures when data corruption is present.
Coding Keys
The Codable protocol uses CodingKey definitions to identify properties of instances. By default, coding keys are generated using the string values of the property names.
The BinaryEncoder provides the sortKeysDuringEncoding option, which forces fields in “keyed” containers, such as struct properties (and some dictionaries), to be sorted in the binary data. This sorting is done by using either the integer keys (if defined), or the property names. Dictionaries with Int or String keys are also sorted.
Sorting the binary data does not influence decoding, but introduces a computation penalty during encoding. It should therefore only be used if the binary data must be consistent across multiple invocations.
-
Note: The sortKeysDuringEncoding option does not guarantee deterministic binary data, and should be used with care.
+
Note: The sortKeysDuringEncoding option does not neccessarily guarantee deterministic binary data, and should be used with care.
+
Encoding optionals in arrays
+
+
Sequences of Optional values (like arrays, sets, …) are normally encoded using a nil index set.
+The index of each nil element in the sequence is recorded, and only non-nil values are encoded.
+The indices of nil elements are then prepended to the data as an array of integers.
+During decoding, this index set is checked to place nil values between the non-nil elements at the appropriate indices.
+
+
This encoding scheme is usually more efficient than, e.g. indicating for each element whether the value is non-optional using an additional byte.
+There can be specific cases where nil index sets become less efficient, e.g. when storing very large arrays of mostly nil values.
+
+
In these cases, the encoder option prependNilIndexSetForUnkeyedContainers can be set to false, causing the encoder to omit the nil index set in favour of an additional byte before each element.
+The decoder must then have containsNilIndexSetForUnkeyedContainers set to false, so that the data can be successfully decoded.
+
Stream encoding and decoding
+
+
The library provides the option to perform encoding and decoding of continuous streams, such as when writing sequences of elements to a file, or when transmitting data over a network.
+This functionality can be used through BinaryStreamEncoder and BinaryStreamDecoder, causing the encoder to embed additional information into the data to allow continuous decoding (mostly length information).
+Encoding and decoding is always done with sequences of one specific type, since multiple types in one stream could not be distinguished from one another.
+
+
Encoding of a stream works similarly to normal encoding:
The decoder has an internal buffer, so incomplete data can be inserted into the decoder as it becomes available. The output of decode(_ data:) will be empty until the next complete element is processed.
+
File encoding and decoding
+
+
Writing data streams to files is a common use case, so the library also provides wrappers around BinaryStreamEncoder and BinaryStreamDecoder to perform these tasks.
+The BinaryFileEncoder can be used to sequentially write elements to a file:
+
letencoder=BinaryFileEncoder<DataElement>(fileAt:url)
+tryencoder.write(element1)
+tryencoder.write(element2)
+...
+tryencoder.close()// Close the file
+
+
+
Elements will always be appended to the end of file, so existing files can be updated with additional data.
+
+
Decoding works in a similar way, except with a callback to handle each element as it is decoded:
+
letdecoder=BinaryFileDecoder<DataElement>(fileAt:url)
+trydecoder.read{elementin
+ // Process each element
+}
+
+
+
There is also the possibility to read all elements at once using readAll(), or to read only one element at a time (readElement()).
Protocol Buffer compatibility
Achieving Protocol Buffer compatibility is described in ProtobufSupport.md.
An integer value encoded as a Base128 Varint.","parent_name":"DataType"},"Enums/DataType.html#/s:13BinaryCodable8DataTypeO4byteyA2CmF":{"name":"byte","abstract":"
"}}
\ No newline at end of file
+{"Structs/SignedValue.html#/s:s27ExpressibleByIntegerLiteralP0cD4TypeQa":{"name":"IntegerLiteralType","parent_name":"SignedValue"},"Structs/SignedValue.html#/s:13BinaryCodable11SignedValueV07wrappedD0xvp":{"name":"wrappedValue","abstract":"
An integer value encoded as a Base128 Varint.","parent_name":"DataType"},"Enums/DataType.html#/s:13BinaryCodable8DataTypeO4byteyA2CmF":{"name":"byte","abstract":"
All possible errors occuring during encoding produce BinaryEncodingError errors, while unsuccessful decoding produces BinaryDecodingErrors.
Both are enums with several cases describing the nature of the error.
See the documentation of the types to learn more about the different error conditions.
+
Handling corrupted data
+
+
The binary format provides no provisions to detect data corruption, and various errors can occur as the result of added, changed, or missing bytes and bits.
+Additional external measures (checksums, error-correcting codes, …) should be applied if there is an increased risk of data corruption.
+
+
As an example, consider the simple encoding of a String inside a struct, which consists of a key followed by the length of the string in bytes, and the string content.
+The length of the string is encoded using variable-length encoding, so a single bit flip (in the MSB of the length byte) could result in a very large length being decoded, causing the decoder to wait for a very large number of bytes to decode the string.
+This simple error would cause much data to be skipped, potentially corrupting the data stream indefinitely.
+At the same time, it is not possible to determine with certainty where the error occured, making error recovery difficult without additional information about boundaries between elements.
+
+
The decoding errors provided by the library are therefore only hints about error likely occuring from non-conformance to the binary format or version incompatibility, which are not necessarily the true causes of the failures when data corruption is present.
Coding Keys
The Codable protocol uses CodingKey definitions to identify properties of instances. By default, coding keys are generated using the string values of the property names.
The BinaryEncoder provides the sortKeysDuringEncoding option, which forces fields in “keyed” containers, such as struct properties (and some dictionaries), to be sorted in the binary data. This sorting is done by using either the integer keys (if defined), or the property names. Dictionaries with Int or String keys are also sorted.
Sorting the binary data does not influence decoding, but introduces a computation penalty during encoding. It should therefore only be used if the binary data must be consistent across multiple invocations.
-
Note: The sortKeysDuringEncoding option does not guarantee deterministic binary data, and should be used with care.
+
Note: The sortKeysDuringEncoding option does not neccessarily guarantee deterministic binary data, and should be used with care.
+
Encoding optionals in arrays
+
+
Sequences of Optional values (like arrays, sets, …) are normally encoded using a nil index set.
+The index of each nil element in the sequence is recorded, and only non-nil values are encoded.
+The indices of nil elements are then prepended to the data as an array of integers.
+During decoding, this index set is checked to place nil values between the non-nil elements at the appropriate indices.
+
+
This encoding scheme is usually more efficient than, e.g. indicating for each element whether the value is non-optional using an additional byte.
+There can be specific cases where nil index sets become less efficient, e.g. when storing very large arrays of mostly nil values.
+
+
In these cases, the encoder option prependNilIndexSetForUnkeyedContainers can be set to false, causing the encoder to omit the nil index set in favour of an additional byte before each element.
+The decoder must then have containsNilIndexSetForUnkeyedContainers set to false, so that the data can be successfully decoded.
+
Stream encoding and decoding
+
+
The library provides the option to perform encoding and decoding of continuous streams, such as when writing sequences of elements to a file, or when transmitting data over a network.
+This functionality can be used through BinaryStreamEncoder and BinaryStreamDecoder, causing the encoder to embed additional information into the data to allow continuous decoding (mostly length information).
+Encoding and decoding is always done with sequences of one specific type, since multiple types in one stream could not be distinguished from one another.
+
+
Encoding of a stream works similarly to normal encoding:
The decoder has an internal buffer, so incomplete data can be inserted into the decoder as it becomes available. The output of decode(_ data:) will be empty until the next complete element is processed.
+
File encoding and decoding
+
+
Writing data streams to files is a common use case, so the library also provides wrappers around BinaryStreamEncoder and BinaryStreamDecoder to perform these tasks.
+The BinaryFileEncoder can be used to sequentially write elements to a file:
+
letencoder=BinaryFileEncoder<DataElement>(fileAt:url)
+tryencoder.write(element1)
+tryencoder.write(element2)
+...
+tryencoder.close()// Close the file
+
+
+
Elements will always be appended to the end of file, so existing files can be updated with additional data.
+
+
Decoding works in a similar way, except with a callback to handle each element as it is decoded:
+
letdecoder=BinaryFileDecoder<DataElement>(fileAt:url)
+trydecoder.read{elementin
+ // Process each element
+}
+
+
+
There is also the possibility to read all elements at once using readAll(), or to read only one element at a time (readElement()).
Protocol Buffer compatibility
Achieving Protocol Buffer compatibility is described in ProtobufSupport.md.
An integer value encoded as a Base128 Varint.","parent_name":"DataType"},"Enums/DataType.html#/s:13BinaryCodable8DataTypeO4byteyA2CmF":{"name":"byte","abstract":"
"}}
\ No newline at end of file
+{"Structs/SignedValue.html#/s:s27ExpressibleByIntegerLiteralP0cD4TypeQa":{"name":"IntegerLiteralType","parent_name":"SignedValue"},"Structs/SignedValue.html#/s:13BinaryCodable11SignedValueV07wrappedD0xvp":{"name":"wrappedValue","abstract":"
An integer value encoded as a Base128 Varint.","parent_name":"DataType"},"Enums/DataType.html#/s:13BinaryCodable8DataTypeO4byteyA2CmF":{"name":"byte","abstract":"
View on GitHub
+ BinaryFileDecoder Class Reference
+ 
