GitHub

For Developers

This section contains various information that may be useful for developing and understanding the package in more detail.

Data Flow

The following diagram illustrates the process of data handling in OhMyCH, including iteration, deserialization, and returning data to the user:

                              +-------------------+
                              |   Input Data      |
                              |  (Vector{UInt8})  |
                              +-------------------+
                                       |
                +----------------------+-----------------------+
                |                                              |
 +---------------------------+               +------------------------------------+
 |         RowBinary         |               |     RowBinaryWithNamesAndTypes     |
 +---------------------------+               +------------------------------------+
                |                                              |
                v                                              v
 +---------------------------+               +------------------------------------+
 |        User type `T`      |               |    Parse Column Names and Types    |
 |   (e.g., `T` or `Tuple`)  |               |                                    |
 +---------------------------+               +------------------------------------+
                |                                              |
                +----------------------+-----------------------+
                                       |
                                       v
                              +-------------------+
                  +---------> |  Start Iteration  |
                  |           +-------------------+
                  |                    |
                  |                    v
                  |           +-------------------+
                  |           | Read Binary Chunk |
                  |           +-------------------+
                  |                    |
                  |                    v
                  |           +-------------------+
                  |           |  Deserialize Row  |
                  |           +-------------------+
                  |                    |
                  |                    v
                  |           +-------------------+
                  |           | Return Row to User|
                  |           +-------------------+
                  |                    |
                  |                    v
                  |           +-------------------+
                  |           |   Check for EOF   |
                  |           |   - More Data?    |
                  |           +-------------------+
                  |                    |
                  +--------------------+
                                       |
                                       v
                             +-------------------+
                             |   End Iteration   |
                             +-------------------+

Serializer

OhMyCH.serialize — Function
serialize(s::Serializer, value::T) -> Int

Writes the byte representation of value to buffer s.

Examples

julia> using ClickHouse: Serializer, serialize, deserialize

julia> s = Serializer();

julia> serialize(s, UInt8, UInt8(10))
1

julia> serialize(s, Int32, Int32(100))
4

julia> serialize(s, Float64, Float64(1000.0))
8

julia> serialize(s, String, "Hello, World!!!")
16

julia> seekstart(s.io);

julia> deserialize(s, UInt8)
0x0a

julia> deserialize(s, Int32)
100

julia> deserialize(s, Float64)
1000.0

julia> deserialize(s, String)
"Hello, World!!!"
source
OhMyCH.deserialize — Function
deserialize(s::Serializer, T::Type) -> T

Reads a sequence of bytes corresponding to type T from buffer s, and then creates an object from them.

Examples

julia> using ClickHouse: Serializer, serialize, deserialize

julia> s = Serializer([0x0a, 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x8f, 0x40]);

julia> deserialize(s, UInt8)
0x0a

julia> deserialize(s, Int32)
100

julia> deserialize(s, Float64)
1000.0
source

Binary formats

OhMyCH.RowBinary — Type
RowBinary <: RowBinaryResult

This format represents binary rows of data, without specifying their types and names. Data can be extracted using the eachrow or collect methods with the column types specified.

See more in ClickHouse Docs.

source

Column types

OhMyCH.parse_column_type — Function
parse_column_type(type_str::String)

Parses a ClickHouse-like type string and returns the corresponding Julia type.

See also Supported column types.

Examples

julia> parse_column_type("LowCardinality(String)")
String

julia> parse_column_type("Nullable(Decimal(9,2))")
Union{Nothing,Decimal{9,2}}

julia> parse_column_type("Array(Tuple(Int8, UInt16))")
Vector{Tuple{Int8,UInt16}}
source

Binary iterators

OhMyCH.RowToBinaryIter — Type
RowToBinaryIter{T}

Iterator for converting rows into serialized binary data in chunks.

Fields

  • items::AbstractVector{T}: Collection of rows to be serialized.
  • chunk_size::Int: Maximum size of serialized chunk in bytes.
  • serializer::Serializer: Serializer used to encode rows.
  • current_start::Int: Current position in the list of items.
  • items_per_chunk::Int: Number of rows (items) included in a single chunk.
source
OhMyCH.BinaryToRowIter — Type
BinaryToRowIter{F<:RowBinaryResult}

Iterator for row data in format F (See supported formats).

Fields

  • row_type::Type: Data type whose fields define row cell types.
  • binary::F: Binary format containing the serialized data.
  • column_names::NTuple{N,Symbol}: Column names.
  • column_types::NTuple{N,Type}: Column types.
source

Content encoding

OhMyCH.Lz4 — Type
Lz4 <: Codec

Represents the LZ4 codec used for compressing and decompressing data.

source
OhMyCH.encode — Function
encode(::Type{Lz4}, x::Vector{UInt8}) -> Vector{UInt8}

Compresses the data in the byte vector x using the LZ4 codec and returns a vector of the compressed data.

Examples

julia> original_data = collect(UInt8, "Hello") # "Hello" in UInt8 array
5-element Vector{UInt8}:
 0x48
 0x65
 0x6c
 0x6c
 0x6f

julia> compressed_data = encode(Lz4, original_data);
20-element Vector{UInt8}:
 0x04
 0x22
    ⋮
 0x00
 0x00
source
OhMyCH.decode — Function
decode(::Type{Lz4}, x::Vector{UInt8}) -> Vector{UInt8}

Decompresses LZ4-compressed data in the byte vector x and returns a vector of the original data.

Examples

julia> original_data = collect(UInt8, "Hello")
5-element Vector{UInt8}:
 0x48
 0x65
 0x6c
 0x6c
 0x6f

julia> compressed_data = encode(Lz4, original_data);

julia> decode(Lz4, compressed_data) # "Hello" in UInt8 array
5-element Vector{UInt8}:
 0x48
 0x65
 0x6c
 0x6c
 0x6f
source