For Developers
This section contains various information that may be useful for developing and understanding the package in more detail.
Data Flow
The following diagram illustrates the process of data handling in OhMyCH, including iteration, deserialization, and returning data to the user:
+-------------------+
| Input Data |
| (Vector{UInt8}) |
+-------------------+
|
+----------------------+-----------------------+
| |
+---------------------------+ +------------------------------------+
| RowBinary | | RowBinaryWithNamesAndTypes |
+---------------------------+ +------------------------------------+
| |
v v
+---------------------------+ +------------------------------------+
| User type `T` | | Parse Column Names and Types |
| (e.g., `T` or `Tuple`) | | |
+---------------------------+ +------------------------------------+
| |
+----------------------+-----------------------+
|
v
+-------------------+
+---------> | Start Iteration |
| +-------------------+
| |
| v
| +-------------------+
| | Read Binary Chunk |
| +-------------------+
| |
| v
| +-------------------+
| | Deserialize Row |
| +-------------------+
| |
| v
| +-------------------+
| | Return Row to User|
| +-------------------+
| |
| v
| +-------------------+
| | Check for EOF |
| | - More Data? |
| +-------------------+
| |
+--------------------+
|
v
+-------------------+
| End Iteration |
+-------------------+
Serializer
OhMyCH.Serializer
— TypeSerializer{I<:IO}
Custom IO wrapper for (de)serializing binary data streams.
OhMyCH.serialize
— Functionserialize(s::Serializer, value::T) -> Int
Writes the byte representation of value
to buffer s
.
Examples
julia> using ClickHouse: Serializer, serialize, deserialize
julia> s = Serializer();
julia> serialize(s, UInt8, UInt8(10))
1
julia> serialize(s, Int32, Int32(100))
4
julia> serialize(s, Float64, Float64(1000.0))
8
julia> serialize(s, String, "Hello, World!!!")
16
julia> seekstart(s.io);
julia> deserialize(s, UInt8)
0x0a
julia> deserialize(s, Int32)
100
julia> deserialize(s, Float64)
1000.0
julia> deserialize(s, String)
"Hello, World!!!"
OhMyCH.deserialize
— Functiondeserialize(s::Serializer, T::Type) -> T
Reads a sequence of bytes corresponding to type T
from buffer s
, and then creates an object from them.
Examples
julia> using ClickHouse: Serializer, serialize, deserialize
julia> s = Serializer([0x0a, 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x8f, 0x40]);
julia> deserialize(s, UInt8)
0x0a
julia> deserialize(s, Int32)
100
julia> deserialize(s, Float64)
1000.0
Binary formats
OhMyCH.RowBinaryResult
— TypeRowBinaryResult
Abstract supertype for all row binary data results.
This type is a base for binary data representations, such as RowBinary
and RowBinaryWithNamesAndTypes
.
See Also
OhMyCH.RowBinary
— TypeRowBinary <: RowBinaryResult
This format represents binary rows of data, without specifying their types and names. Data can be extracted using the eachrow
or collect
methods with the column types specified.
See more in ClickHouse Docs
.
OhMyCH.RowBinaryWithNamesAndTypes
— TypeRowBinaryWithNamesAndTypes <: RowBinaryResult
This format represents binary rows of data containing information about column types and their names. Data can be extracted using the eachrow
or collect
methods.
See more in ClickHouse Docs
.
Column types
OhMyCH.parse_column_type
— Functionparse_column_type(type_str::String)
Parses a ClickHouse-like type string and returns the corresponding Julia type.
See also Supported column types
.
Examples
julia> parse_column_type("LowCardinality(String)")
String
julia> parse_column_type("Nullable(Decimal(9,2))")
Union{Nothing,Decimal{9,2}}
julia> parse_column_type("Array(Tuple(Int8, UInt16))")
Vector{Tuple{Int8,UInt16}}
Binary iterators
OhMyCH.RowToBinaryIter
— TypeRowToBinaryIter{T}
Iterator for converting rows into serialized binary data in chunks.
Fields
items::AbstractVector{T}
: Collection of rows to be serialized.chunk_size::Int
: Maximum size of serialized chunk in bytes.serializer::Serializer
: Serializer used to encode rows.current_start::Int
: Current position in the list of items.items_per_chunk::Int
: Number of rows (items) included in a single chunk.
OhMyCH.BinaryToRowIter
— TypeBinaryToRowIter{F<:RowBinaryResult}
Iterator for row data in format F
(See supported formats).
Fields
row_type::Type
: Data type whose fields define row cell types.binary::F
: Binary format containing the serialized data.column_names::NTuple{N,Symbol}
: Column names.column_types::NTuple{N,Type}
: Column types.
Content encoding
OhMyCH.Lz4
— TypeLz4 <: Codec
Represents the LZ4 codec used for compressing and decompressing data.
OhMyCH.encode
— Functionencode(::Type{Lz4}, x::Vector{UInt8}) -> Vector{UInt8}
Compresses the data in the byte vector x
using the LZ4 codec and returns a vector of the compressed data.
Examples
julia> original_data = collect(UInt8, "Hello") # "Hello" in UInt8 array
5-element Vector{UInt8}:
0x48
0x65
0x6c
0x6c
0x6f
julia> compressed_data = encode(Lz4, original_data);
20-element Vector{UInt8}:
0x04
0x22
⋮
0x00
0x00
OhMyCH.decode
— Functiondecode(::Type{Lz4}, x::Vector{UInt8}) -> Vector{UInt8}
Decompresses LZ4-compressed data in the byte vector x
and returns a vector of the original data.
Examples
julia> original_data = collect(UInt8, "Hello")
5-element Vector{UInt8}:
0x48
0x65
0x6c
0x6c
0x6f
julia> compressed_data = encode(Lz4, original_data);
julia> decode(Lz4, compressed_data) # "Hello" in UInt8 array
5-element Vector{UInt8}:
0x48
0x65
0x6c
0x6c
0x6f