conversions

Utility module for various conversions around PrimitiveType implementations.

This module enables

• Converting partition strings to built-in python objects.
• Converting a value to a byte buffer.
• Converting a byte buffer to a value.
• Converting a json-single field serialized field

Note

Conversion logic varies based on the PrimitiveType implementation. Therefore conversion functions are defined here as generic functions using the @singledispatch decorator. For each PrimitiveType implementation, a concrete function is registered for each generic conversion function. For PrimitiveType implementations that share the same conversion logic, registrations can be stacked.

_(_, val)

Convert JSON WKT string into WKB bytes per Iceberg spec.

Note: This requires WKT to WKB conversion which is not yet implemented. The Iceberg spec requires geography values to be represented as WKT strings in JSON, but PyIceberg stores geography as WKB bytes at runtime.

Raises:

Type	Description
NotImplementedError	WKT to WKB conversion is not yet supported.

Source code in pyiceberg/conversions.py

@from_json.register(GeographyType)
def _(_: GeographyType, val: str | bytes) -> bytes:
    """Convert JSON WKT string into WKB bytes per Iceberg spec.

    Note: This requires WKT to WKB conversion which is not yet implemented.
    The Iceberg spec requires geography values to be represented as WKT strings
    in JSON, but PyIceberg stores geography as WKB bytes at runtime.

    Raises:
        NotImplementedError: WKT to WKB conversion is not yet supported.
    """
    if isinstance(val, bytes):
        # Already WKB bytes, return as-is
        return val
    raise NotImplementedError(
        "Geography JSON deserialization requires WKT to WKB conversion, which is not yet implemented. "
        "See https://iceberg.apache.org/spec/#json-single-value-serialization for spec details."
    )

from_bytes(primitive_type, b)

Convert bytes to a built-in python value.

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
b	bytes	The bytes to convert.	required

Source code in pyiceberg/conversions.py

@singledispatch  # type: ignore
def from_bytes(primitive_type: PrimitiveType, b: bytes) -> L:  # type: ignore
    """Convert bytes to a built-in python value.

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        b (bytes): The bytes to convert.
    """
    raise TypeError(f"Cannot deserialize bytes, type {primitive_type} not supported: {b!r}")

from_json(primitive_type, val)

Convert JSON value types into built-in python values.

https://iceberg.apache.org/spec/#json-single-value-serialization

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
val	Any	The arbitrary JSON value to convert into the right form	required

Source code in pyiceberg/conversions.py

@singledispatch  # type: ignore
def from_json(primitive_type: PrimitiveType, val: Any) -> L:  # type: ignore
    """Convert JSON value types into built-in python values.

    https://iceberg.apache.org/spec/#json-single-value-serialization

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        val (Any): The arbitrary JSON value to convert into the right form
    """
    raise TypeError(f"Cannot deserialize bytes, type {primitive_type} not supported: {str(val)}")

handle_none(func)

Handle cases where partition values are None or "HIVE_DEFAULT_PARTITION".

Parameters:

Name	Type	Description	Default
func	Callable	A function registered to the singledispatch function partition_to_py.	required

Source code in pyiceberg/conversions.py

def handle_none(func: Callable) -> Callable:  # type: ignore
    """Handle cases where partition values are `None` or "__HIVE_DEFAULT_PARTITION__".

    Args:
        func (Callable): A function registered to the singledispatch function `partition_to_py`.
    """

    def wrapper(primitive_type: PrimitiveType, value_str: str | None) -> Any:
        if value_str is None:
            return None
        elif value_str == "__HIVE_DEFAULT_PARTITION__":
            return None
        return func(primitive_type, value_str)

    return wrapper

partition_to_py(primitive_type, value_str)

Convert a partition string to a python built-in.

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
value_str	str	A string representation of a partition value.	required

Source code in pyiceberg/conversions.py

@singledispatch
def partition_to_py(primitive_type: PrimitiveType, value_str: str) -> int | float | str | uuid.UUID | bytes | Decimal:
    """Convert a partition string to a python built-in.

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        value_str (str): A string representation of a partition value.
    """
    raise TypeError(f"Cannot convert '{value_str}' to unsupported type: {primitive_type}")

to_bytes(primitive_type, _)

Convert a built-in python value to bytes.

This conversion follows the serialization scheme for storing single values as individual binary values defined in the Iceberg specification that can be found at https://iceberg.apache.org/spec/#appendix-d-single-value-serialization

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
_	bool | bytes | Decimal | date | datetime | float | int | str | time | UUID	The value to convert to bytes (The type of this value depends on which dispatched function is used--check dispatchable functions for type hints).	required

Source code in pyiceberg/conversions.py

@singledispatch
def to_bytes(
    primitive_type: PrimitiveType, _: bool | bytes | Decimal | date | datetime | float | int | str | time | uuid.UUID
) -> bytes:
    """Convert a built-in python value to bytes.

    This conversion follows the serialization scheme for storing single values as individual binary values
    defined in the Iceberg specification that can be found at
    https://iceberg.apache.org/spec/#appendix-d-single-value-serialization

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        _: The value to convert to bytes (The type of this value depends on which dispatched function is
            used--check dispatchable functions for type hints).
    """
    raise TypeError(f"scale does not match {primitive_type}")

to_json(primitive_type, val)

Convert built-in python values into JSON value types.

https://iceberg.apache.org/spec/#json-single-value-serialization

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
val	Any	The arbitrary built-in value to convert into the right form	required

Source code in pyiceberg/conversions.py

@singledispatch  # type: ignore
def to_json(primitive_type: PrimitiveType, val: Any) -> L:  # type: ignore
    """Convert built-in python values into JSON value types.

    https://iceberg.apache.org/spec/#json-single-value-serialization

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        val (Any): The arbitrary built-in value to convert into the right form
    """
    raise TypeError(f"Cannot deserialize bytes, type {primitive_type} not supported: {val}")