conversions

Utility module for various conversions around PrimitiveType implementations.

This module enables

• Converting partition strings to built-in python objects.
• Converting a value to a byte buffer.
• Converting a byte buffer to a value.
• Converting a json-single field serialized field

Note

Conversion logic varies based on the PrimitiveType implementation. Therefore conversion functions are defined here as generic functions using the @singledispatch decorator. For each PrimitiveType implementation, a concrete function is registered for each generic conversion function. For PrimitiveType implementations that share the same conversion logic, registrations can be stacked.

_(_, val)

Convert JSON string into Python UUID.

Source code in pyiceberg/conversions.py

@from_json.register(UUIDType)
def _(_: UUIDType, val: Union[str, bytes, uuid.UUID]) -> uuid.UUID:
    """Convert JSON string into Python UUID."""
    if isinstance(val, str):
        return uuid.UUID(val)
    elif isinstance(val, bytes):
        return uuid.UUID(bytes=val)
    else:
        return val

from_bytes(primitive_type, b)

Convert bytes to a built-in python value.

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
b	bytes	The bytes to convert.	required

Source code in pyiceberg/conversions.py

@singledispatch  # type: ignore
def from_bytes(primitive_type: PrimitiveType, b: bytes) -> L:  # type: ignore
    """Convert bytes to a built-in python value.

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        b (bytes): The bytes to convert.
    """
    raise TypeError(f"Cannot deserialize bytes, type {primitive_type} not supported: {b!r}")

from_json(primitive_type, val)

Convert JSON value types into built-in python values.

https://iceberg.apache.org/spec/#json-single-value-serialization

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
val	Any	The arbitrary JSON value to convert into the right form	required

Source code in pyiceberg/conversions.py

@singledispatch  # type: ignore
def from_json(primitive_type: PrimitiveType, val: Any) -> L:  # type: ignore
    """Convert JSON value types into built-in python values.

    https://iceberg.apache.org/spec/#json-single-value-serialization

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        val (Any): The arbitrary JSON value to convert into the right form
    """
    raise TypeError(f"Cannot deserialize bytes, type {primitive_type} not supported: {str(val)}")

handle_none(func)

Handle cases where partition values are None or "HIVE_DEFAULT_PARTITION".

Parameters:

Name	Type	Description	Default
func	Callable	A function registered to the singledispatch function partition_to_py.	required

Source code in pyiceberg/conversions.py

def handle_none(func: Callable) -> Callable:  # type: ignore
    """Handle cases where partition values are `None` or "__HIVE_DEFAULT_PARTITION__".

    Args:
        func (Callable): A function registered to the singledispatch function `partition_to_py`.
    """

    def wrapper(primitive_type: PrimitiveType, value_str: Optional[str]) -> Any:
        if value_str is None:
            return None
        elif value_str == "__HIVE_DEFAULT_PARTITION__":
            return None
        return func(primitive_type, value_str)

    return wrapper

partition_to_py(primitive_type, value_str)

Convert a partition string to a python built-in.

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
value_str	str	A string representation of a partition value.	required

Source code in pyiceberg/conversions.py

@singledispatch
def partition_to_py(primitive_type: PrimitiveType, value_str: str) -> Union[int, float, str, uuid.UUID, bytes, Decimal]:
    """Convert a partition string to a python built-in.

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        value_str (str): A string representation of a partition value.
    """
    raise TypeError(f"Cannot convert '{value_str}' to unsupported type: {primitive_type}")

to_bytes(primitive_type, _)

Convert a built-in python value to bytes.

This conversion follows the serialization scheme for storing single values as individual binary values defined in the Iceberg specification that can be found at https://iceberg.apache.org/spec/#appendix-d-single-value-serialization

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
_	Union[bool, bytes, Decimal, date, datetime, float, int, str, time, UUID]	The value to convert to bytes (The type of this value depends on which dispatched function is used--check dispatchable functions for type hints).	required

Source code in pyiceberg/conversions.py

@singledispatch
def to_bytes(
    primitive_type: PrimitiveType, _: Union[bool, bytes, Decimal, date, datetime, float, int, str, time, uuid.UUID]
) -> bytes:
    """Convert a built-in python value to bytes.

    This conversion follows the serialization scheme for storing single values as individual binary values defined in the Iceberg specification that
    can be found at https://iceberg.apache.org/spec/#appendix-d-single-value-serialization

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        _: The value to convert to bytes (The type of this value depends on which dispatched function is
            used--check dispatchable functions for type hints).
    """
    raise TypeError(f"scale does not match {primitive_type}")

to_json(primitive_type, val)

Convert built-in python values into JSON value types.

https://iceberg.apache.org/spec/#json-single-value-serialization

Parameters:

Name	Type	Description	Default
primitive_type	PrimitiveType	An implementation of the PrimitiveType base class.	required
val	Any	The arbitrary built-in value to convert into the right form	required

Source code in pyiceberg/conversions.py

@singledispatch  # type: ignore
def to_json(primitive_type: PrimitiveType, val: Any) -> L:  # type: ignore
    """Convert built-in python values into JSON value types.

    https://iceberg.apache.org/spec/#json-single-value-serialization

    Args:
        primitive_type (PrimitiveType): An implementation of the PrimitiveType base class.
        val (Any): The arbitrary built-in value to convert into the right form
    """
    raise TypeError(f"Cannot deserialize bytes, type {primitive_type} not supported: {val}")