Tracking
This section covers the task tracking and monitoring system in the Kailash SDK.
Overview
The tracking system provides comprehensive monitoring and analytics for workflow execution:
Task Tracking: Monitor individual node executions
Run Management: Track complete workflow runs
Metrics Collection: Gather performance and resource metrics
Storage Backends: Persist tracking data
Analytics: Analyze execution patterns and performance
TaskManager
The central component for tracking workflow execution.
- class kailash.tracking.manager.TaskManager(storage_backend: StorageBackend | None = None)[source]
Bases:
objectManages task tracking for workflow executions.
- Parameters:
storage_backend (StorageBackend | None)
- __init__(storage_backend: StorageBackend | None = None)[source]
Initialize task manager.
- Parameters:
storage_backend (StorageBackend | None) – Storage backend for persistence. Defaults to SQLiteStorage.
- Raises:
TaskException – If initialization fails
- create_run(workflow_name: str, metadata: dict[str, Any] | None = None) str[source]
Create a new workflow run.
- update_run_status(run_id: str, status: str, error: str | None = None) None[source]
Update workflow run status.
- create_task(node_id: str, input_data: dict[str, Any] | None = None, metadata: dict[str, Any] | None = None, run_id: str = 'test-run-id', node_type: str = 'default-node-type', dependencies: list[str] | None = None, started_at: datetime | None = None) TaskRun[source]
Create a new task.
- Parameters:
node_id (str) – Node ID in the workflow
input_data (dict[str, Any] | None) – Input data for the task
run_id (str) – Associated run ID (defaults to “test-run-id” for backward compatibility)
node_type (str) – Type of node (defaults to “default-node-type” for backward compatibility)
dependencies (list[str] | None) – List of task IDs this task depends on
started_at (datetime | None) – When the task started
- Returns:
TaskRun instance
- Raises:
TaskException – If task creation fails
StorageException – If storage operation fails
- Return type:
TaskRun
- update_task_status(task_id: str, status: TaskStatus, result: dict[str, Any] | None = None, error: str | None = None, ended_at: datetime | None = None, metadata: dict[str, Any] | None = None) None[source]
Update task status.
- Parameters:
- Raises:
TaskException – If task not found
StorageException – If storage operation fails
TaskStateError – If status transition is invalid
- Return type:
None
- get_run(run_id: str) WorkflowRun | None[source]
Get workflow run by ID.
- Parameters:
run_id (str) – Run ID
- Returns:
WorkflowRun instance or None
- Raises:
StorageException – If storage operation fails
- Return type:
WorkflowRun | None
- get_task(task_id: str) TaskRun | None[source]
Get task by ID.
- Parameters:
task_id (str) – Task ID
- Returns:
TaskRun instance or None
- Raises:
StorageException – If storage operation fails
- Return type:
TaskRun | None
- list_runs(workflow_name: str | None = None, status: str | None = None, limit: int | None = None) list[RunSummary][source]
List workflow runs.
- list_tasks(run_id: str, node_id: str | None = None, status: TaskStatus | None = None) list[TaskSummary][source]
List tasks for a run.
- get_run_summary(run_id: str) RunSummary | None[source]
Get summary for a specific run.
- Parameters:
run_id (str) – Run ID
- Returns:
RunSummary or None
- Raises:
StorageException – If storage operation fails
- Return type:
RunSummary | None
- complete_task(task_id: str, output_data: dict[str, Any] | None = None) None[source]
Complete a task successfully.
- retry_task(task_id: str) TaskRun[source]
Create a new task as a retry of an existing task.
- Parameters:
task_id (str) – Original task ID
- Returns:
New task instance
- Raises:
TaskException – If task not found
StorageException – If storage operation fails
- Return type:
TaskRun
- delete_task(task_id: str) None[source]
Delete a task.
- Parameters:
task_id (str) – Task ID
- Raises:
TaskException – If task not found
StorageException – If storage operation fails
- Return type:
None
- get_tasks_by_status(status: TaskStatus) list[TaskRun][source]
Get tasks by status.
- Parameters:
status (TaskStatus) – Status to filter by
- Returns:
List of matching tasks
- Raises:
StorageException – If storage operation fails
- Return type:
list[TaskRun]
- get_task_history(task_id: str) list[TaskRun][source]
Get task history (original task and all retries).
- get_tasks_by_timerange(start_time: datetime, end_time: datetime) list[TaskRun][source]
Get tasks created between start_time and end_time.
- get_task_statistics() dict[str, Any][source]
Get task statistics.
- Returns:
total_tasks: Total number of tasks
by_status: Count of tasks by status
by_node: Count of tasks by node ID
- Return type:
Dictionary with statistics
- Raises:
StorageException – If storage operation fails
- update_task_metrics(task_id: str, metrics: TaskMetrics) None[source]
Update task metrics.
- Parameters:
task_id (str) – Task ID
metrics (TaskMetrics) – Metrics to update
- Raises:
TaskException – If task not found
StorageException – If storage operation fails
- Return type:
None
- get_running_tasks() list[TaskRun][source]
Get all currently running tasks.
- Returns:
List of running tasks
- Raises:
StorageException – If storage operation fails
- Return type:
list[TaskRun]
- get_task_dependencies(task_id: str) list[TaskRun][source]
Get tasks that are dependencies for the given task.
- save_task(task: TaskRun) None[source]
Save a task to storage.
This is a convenience method that directly saves a task instance to storage. For new tasks, prefer using create_task() instead.
- Parameters:
task (TaskRun) – TaskRun instance to save
- Raises:
StorageException – If storage operation fails
- Return type:
None
- set_search_attributes(run_id: str, attributes: dict[str, Any]) None[source]
Set (upsert) search attributes on a workflow run.
- search_runs(filters: dict[str, Any], order_by: str = 'created_at DESC', limit: int = 100, offset: int = 0) list[dict][source]
Search workflow runs by attribute filters.
- Parameters:
- Returns:
List of workflow run dicts matching all filters.
- Raises:
StorageException – If storage operation fails.
- Return type:
- get_execution_audit_trail(run_id: str) list[dict][source]
Get a comprehensive execution audit trail for a run.
Combines task records with timing and audit events into a chronological list of everything that happened during the run.
Basic Usage:
from kailash.tracking import TaskManager
from kailash import Workflow
from kailash.workflow.runner import WorkflowRunner
# Create task manager
task_manager = TaskManager()
# Use with workflow runner
workflow = Workflow("my_workflow")
runner = WorkflowRunner(workflow, task_manager)
# Execute and track
results = runner.run()
# Access tracking data
run_id = results["run_id"]
run_info = task_manager.get_run(run_id)
print(f"Execution took: {run_info.duration}s")
Tracking Models
WorkflowRun
Represents a complete workflow execution.
- class kailash.tracking.models.WorkflowRun(*, run_id: str = <factory>, workflow_name: str, status: str = 'running', started_at: datetime = <factory>, ended_at: datetime | None = None, tasks: list[str] = <factory>, metadata: dict[str, ~typing.Any]=<factory>, error: str | None = None)[source]
Bases:
BaseModelModel for a workflow execution run.
- Parameters:
- add_task(task_id: str) None[source]
Add a task to this run.
- Parameters:
task_id (str) – Task ID to add
- Raises:
TaskException – If task_id is invalid
- Return type:
None
- get_duration() float | None[source]
Get run duration in seconds.
- Returns:
Duration in seconds, or None if not completed
- Return type:
float | None
- classmethod __get_pydantic_json_schema__(core_schema: CoreSchema, handler: GetJsonSchemaHandler, /) JsonSchemaValue
Hook into generating the model’s JSON schema.
- Parameters:
core_schema (CoreSchema) – A pydantic-core CoreSchema. You can ignore this argument and call the handler with a new CoreSchema, wrap this CoreSchema ({‘type’: ‘nullable’, ‘schema’: current_schema}), or just call the handler with the original schema.
handler (GetJsonSchemaHandler) – Call into Pydantic’s internal JSON schema generation. This will raise a pydantic.errors.PydanticInvalidForJsonSchema if JSON schema generation fails. Since this gets called by BaseModel.model_json_schema you can override the schema_generator argument to that function to change JSON schema generation globally for a type.
- Returns:
A JSON schema, as a Python object.
- Return type:
JsonSchemaValue
- __init__(**data: Any) None
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
- Parameters:
data (Any)
- Return type:
None
- __pretty__(fmt: Callable[[Any], Any], **kwargs: Any) Generator[Any]
Used by devtools (https://python-devtools.helpmanual.io/) to pretty print objects.
- classmethod __pydantic_init_subclass__(**kwargs: Any) None
This is intended to behave just like __init_subclass__, but is called by ModelMetaclass only after basic class initialization is complete. In particular, attributes like model_fields will be present when this is called, but forward annotations are not guaranteed to be resolved yet, meaning that creating an instance of the class may fail.
This is necessary because __init_subclass__ will always be called by type.__new__, and it would require a prohibitively large refactor to the ModelMetaclass to ensure that type.__new__ was called in such a manner that the class would already be sufficiently initialized.
This will receive the same kwargs that would be passed to the standard __init_subclass__, namely, any kwargs passed to the class definition that aren’t used internally by Pydantic.
- Parameters:
**kwargs (Any) – Any keyword arguments passed to the class definition that aren’t used internally by Pydantic.
- Return type:
None
Note
You may want to override [__pydantic_on_complete__()][pydantic.main.BaseModel.__pydantic_on_complete__] instead, which is called once the class and its fields are fully initialized and ready for validation.
- classmethod __pydantic_on_complete__() None
This is called once the class and its fields are fully initialized and ready to be used.
This typically happens when the class is created (just before [__pydantic_init_subclass__()][pydantic.main.BaseModel.__pydantic_init_subclass__] is called on the superclass), except when forward annotations are used that could not immediately be resolved. In that case, it will be called later, when the model is rebuilt automatically or explicitly using [model_rebuild()][pydantic.main.BaseModel.model_rebuild].
- Return type:
None
- __rich_repr__() RichReprResult
Used by Rich (https://rich.readthedocs.io/en/stable/pretty.html) to pretty print objects.
- Return type:
RichReprResult
- copy(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, update: Dict[str, Any] | None = None, deep: bool = False) Self
Returns a copy of the model.
- !!! warning “Deprecated”
This method is now deprecated; use model_copy instead.
If you need include or exclude, use:
`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `- Parameters:
include (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to include in the copied model.
exclude (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to exclude in the copied model.
update (Dict[str, Any] | None) – Optional dictionary of field-value pairs to override field values in the copied model.
deep (bool) – If True, the values of fields that are Pydantic models will be deep-copied.
- Returns:
A copy of the model with included, excluded and updated fields as specified.
- Return type:
Self
- dict(*, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, by_alias: bool = False, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) Dict[str, Any]
- Parameters:
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)
- Return type:
- json(*, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, by_alias: bool = False, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Callable[[Any], Any] | None = PydanticUndefined, models_as_dict: bool = PydanticUndefined, **dumps_kwargs: Any) str
- Parameters:
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)
models_as_dict (bool)
dumps_kwargs (Any)
- Return type:
- model_computed_fields = {}
- model_config = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- classmethod model_construct(_fields_set: set[str] | None = None, **values: Any) Self
Creates a new instance of the Model class with validated data.
Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.
- !!! note
model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.
- Parameters:
_fields_set (set[str] | None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.
values (Any) – Trusted or pre-validated data dictionary.
- Returns:
A new instance of the Model class with validated data.
- Return type:
- model_copy(*, update: Mapping[str, Any] | None = None, deep: bool = False) Self
- !!! abstract “Usage Documentation”
[model_copy](../concepts/models.md#model-copy)
Returns a copy of the model.
- !!! note
The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).
- model_dump(*, mode: Literal['json', 'python'] | str = 'python', include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, context: Any | None = None, by_alias: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, exclude_computed_fields: bool = False, round_trip: bool = False, warnings: bool | Literal['none', 'warn', 'error'] = True, fallback: Callable[[Any], Any] | None = None, serialize_as_any: bool = False, polymorphic_serialization: bool | None = None) dict[str, Any]
- !!! abstract “Usage Documentation”
[model_dump](../concepts/serialization.md#python-mode)
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
- Parameters:
mode (Literal['json', 'python'] | str) – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to include in the output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to exclude from the output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to use the field’s alias in the dictionary key if defined.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization (bool | None) – Whether to use model and dataclass polymorphic serialization for this call.
- Returns:
A dictionary representation of the model.
- Return type:
- model_dump_json(*, indent: int | None = None, ensure_ascii: bool = False, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, context: Any | None = None, by_alias: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, exclude_computed_fields: bool = False, round_trip: bool = False, warnings: bool | Literal['none', 'warn', 'error'] = True, fallback: Callable[[Any], Any] | None = None, serialize_as_any: bool = False, polymorphic_serialization: bool | None = None) str
- !!! abstract “Usage Documentation”
[model_dump_json](../concepts/serialization.md#json-mode)
Generates a JSON representation of the model using Pydantic’s to_json method.
- Parameters:
indent (int | None) – Indentation to use in the JSON output. If None is passed, the output will be compact.
ensure_ascii (bool) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to include in the JSON output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to exclude from the JSON output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to serialize using field aliases.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization (bool | None) – Whether to use model and dataclass polymorphic serialization for this call.
- Returns:
A JSON string representation of the model.
- Return type:
- property model_extra: dict[str, Any] | None
Get extra fields set during validation.
- Returns:
A dictionary of extra fields, or None if config.extra is not set to “allow”.
- model_fields = {'ended_at': FieldInfo(annotation=Union[datetime, NoneType], required=False, default=None), 'error': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'metadata': FieldInfo(annotation=dict[str, Any], required=False, default_factory=dict), 'run_id': FieldInfo(annotation=str, required=False, default_factory=<lambda>), 'started_at': FieldInfo(annotation=datetime, required=False, default_factory=<lambda>), 'status': FieldInfo(annotation=str, required=False, default='running', description='Run status'), 'tasks': FieldInfo(annotation=list[str], required=False, default_factory=list, description='Task IDs'), 'workflow_name': FieldInfo(annotation=str, required=True, description='Name of the workflow')}
- property model_fields_set: set[str]
Returns the set of fields that have been explicitly set on this model instance.
- Returns:
- A set of strings representing the fields that have been set,
i.e. that were not filled from defaults.
- classmethod model_parametrized_name(params: tuple[type[Any], ...]) str
Compute the class name for parametrizations of generic classes.
This method can be overridden to achieve a custom naming scheme for generic BaseModels.
- Parameters:
params (tuple[type[Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.
- Returns:
String representing the new class where params are passed to cls as type variables.
- Raises:
TypeError – Raised when trying to generate concrete names for non-generic models.
- Return type:
- model_post_init(context: Any, /) None
Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.
- Parameters:
context (Any)
- Return type:
None
- classmethod model_rebuild(*, force: bool = False, raise_errors: bool = True, _parent_namespace_depth: int = 2, _types_namespace: MappingNamespace | None = None) bool | None
Try to rebuild the pydantic-core schema for the model.
This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.
- Parameters:
force (bool) – Whether to force the rebuilding of the model schema, defaults to False.
raise_errors (bool) – Whether to raise errors, defaults to True.
_parent_namespace_depth (int) – The depth level of the parent namespace, defaults to 2.
_types_namespace (MappingNamespace | None) – The types namespace, defaults to None.
- Returns:
Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.
- Return type:
bool | None
- classmethod model_validate(obj: Any, *, strict: bool | None = None, extra: Literal['allow', 'ignore', 'forbid'] | None = None, from_attributes: bool | None = None, context: Any | None = None, by_alias: bool | None = None, by_name: bool | None = None) Self
Validate a pydantic model instance.
- Parameters:
obj (Any) – The object to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
from_attributes (bool | None) – Whether to extract data from object attributes.
context (Any | None) – Additional context to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.
- Raises:
ValidationError – If the object could not be validated.
- Returns:
The validated model instance.
- Return type:
- classmethod model_validate_json(json_data: str | bytes | bytearray, *, strict: bool | None = None, extra: Literal['allow', 'ignore', 'forbid'] | None = None, context: Any | None = None, by_alias: bool | None = None, by_name: bool | None = None) Self
- !!! abstract “Usage Documentation”
[JSON Parsing](../concepts/json.md#json-parsing)
Validate the given JSON data against the Pydantic model.
- Parameters:
json_data (str | bytes | bytearray) – The JSON data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.
- Returns:
The validated Pydantic model.
- Raises:
ValidationError – If json_data is not a JSON string or the object could not be validated.
- Return type:
- classmethod model_validate_strings(obj: Any, *, strict: bool | None = None, extra: Literal['allow', 'ignore', 'forbid'] | None = None, context: Any | None = None, by_alias: bool | None = None, by_name: bool | None = None) Self
Validate the given object with string data against the Pydantic model.
- Parameters:
obj (Any) – The object containing string data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.
- Returns:
The validated Pydantic model.
- Return type:
- classmethod parse_file(path: str | Path, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) Self
- classmethod parse_raw(b: str | bytes, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) Self
Attributes:
run_id: Unique identifier for the runworkflow_id: Identifier of the executed workflowstatus: Current status (pending, running, completed, failed)start_time: Execution start timestampend_time: Execution end timestampduration: Total execution timemetadata: Additional run metadata
Example Usage:
# Get run information
run = task_manager.get_run(run_id)
print(f"Run ID: {run.run_id}")
print(f"Status: {run.status}")
print(f"Duration: {run.duration}s")
print(f"Nodes executed: {len(run.tasks)}")
Task
Represents individual node execution within a workflow run.
- kailash.tracking.models.Task
alias of
TaskRun
Attributes:
task_id: Unique task identifierrun_id: Parent workflow run IDnode_id: ID of the executed nodestatus: Task status (pending, running, completed, failed, skipped)start_time: Task start timestampend_time: Task end timestampduration: Task execution timeerror: Error information if failedmetrics: Performance metrics
Example Usage:
# Get task details
tasks = task_manager.get_tasks(run_id)
for task in tasks:
print(f"Node: {task.node_id}")
print(f"Status: {task.status}")
print(f"Duration: {task.duration}s")
if task.error:
print(f"Error: {task.error}")
TaskMetrics
Performance and resource metrics for task execution.
- class kailash.tracking.models.TaskMetrics(*, duration: float | None = 0.0, memory_usage: float | None = 0.0, memory_usage_mb: float | None = 0.0, cpu_usage: float | None = 0.0, custom_metrics: dict[str, ~typing.Any]=<factory>)[source]
Bases:
BaseModelMetrics for task execution.
- Parameters:
- classmethod from_dict(data: dict[str, Any]) TaskMetrics[source]
Create metrics from dictionary representation.
- Parameters:
- Return type:
- classmethod __get_pydantic_json_schema__(core_schema: CoreSchema, handler: GetJsonSchemaHandler, /) JsonSchemaValue
Hook into generating the model’s JSON schema.
- Parameters:
core_schema (CoreSchema) – A pydantic-core CoreSchema. You can ignore this argument and call the handler with a new CoreSchema, wrap this CoreSchema ({‘type’: ‘nullable’, ‘schema’: current_schema}), or just call the handler with the original schema.
handler (GetJsonSchemaHandler) – Call into Pydantic’s internal JSON schema generation. This will raise a pydantic.errors.PydanticInvalidForJsonSchema if JSON schema generation fails. Since this gets called by BaseModel.model_json_schema you can override the schema_generator argument to that function to change JSON schema generation globally for a type.
- Returns:
A JSON schema, as a Python object.
- Return type:
JsonSchemaValue
- __pretty__(fmt: Callable[[Any], Any], **kwargs: Any) Generator[Any]
Used by devtools (https://python-devtools.helpmanual.io/) to pretty print objects.
- classmethod __pydantic_init_subclass__(**kwargs: Any) None
This is intended to behave just like __init_subclass__, but is called by ModelMetaclass only after basic class initialization is complete. In particular, attributes like model_fields will be present when this is called, but forward annotations are not guaranteed to be resolved yet, meaning that creating an instance of the class may fail.
This is necessary because __init_subclass__ will always be called by type.__new__, and it would require a prohibitively large refactor to the ModelMetaclass to ensure that type.__new__ was called in such a manner that the class would already be sufficiently initialized.
This will receive the same kwargs that would be passed to the standard __init_subclass__, namely, any kwargs passed to the class definition that aren’t used internally by Pydantic.
- Parameters:
**kwargs (Any) – Any keyword arguments passed to the class definition that aren’t used internally by Pydantic.
- Return type:
None
Note
You may want to override [__pydantic_on_complete__()][pydantic.main.BaseModel.__pydantic_on_complete__] instead, which is called once the class and its fields are fully initialized and ready for validation.
- classmethod __pydantic_on_complete__() None
This is called once the class and its fields are fully initialized and ready to be used.
This typically happens when the class is created (just before [__pydantic_init_subclass__()][pydantic.main.BaseModel.__pydantic_init_subclass__] is called on the superclass), except when forward annotations are used that could not immediately be resolved. In that case, it will be called later, when the model is rebuilt automatically or explicitly using [model_rebuild()][pydantic.main.BaseModel.model_rebuild].
- Return type:
None
- __rich_repr__() RichReprResult
Used by Rich (https://rich.readthedocs.io/en/stable/pretty.html) to pretty print objects.
- Return type:
RichReprResult
- copy(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, update: Dict[str, Any] | None = None, deep: bool = False) Self
Returns a copy of the model.
- !!! warning “Deprecated”
This method is now deprecated; use model_copy instead.
If you need include or exclude, use:
`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `- Parameters:
include (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to include in the copied model.
exclude (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to exclude in the copied model.
update (Dict[str, Any] | None) – Optional dictionary of field-value pairs to override field values in the copied model.
deep (bool) – If True, the values of fields that are Pydantic models will be deep-copied.
- Returns:
A copy of the model with included, excluded and updated fields as specified.
- Return type:
Self
- dict(*, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, by_alias: bool = False, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) Dict[str, Any]
- Parameters:
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)
- Return type:
- json(*, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, by_alias: bool = False, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Callable[[Any], Any] | None = PydanticUndefined, models_as_dict: bool = PydanticUndefined, **dumps_kwargs: Any) str
- Parameters:
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)
models_as_dict (bool)
dumps_kwargs (Any)
- Return type:
- model_computed_fields = {}
- model_config = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- classmethod model_construct(_fields_set: set[str] | None = None, **values: Any) Self
Creates a new instance of the Model class with validated data.
Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.
- !!! note
model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.
- Parameters:
_fields_set (set[str] | None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.
values (Any) – Trusted or pre-validated data dictionary.
- Returns:
A new instance of the Model class with validated data.
- Return type:
- model_copy(*, update: Mapping[str, Any] | None = None, deep: bool = False) Self
- !!! abstract “Usage Documentation”
[model_copy](../concepts/models.md#model-copy)
Returns a copy of the model.
- !!! note
The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).
- model_dump(*, mode: Literal['json', 'python'] | str = 'python', include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, context: Any | None = None, by_alias: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, exclude_computed_fields: bool = False, round_trip: bool = False, warnings: bool | Literal['none', 'warn', 'error'] = True, fallback: Callable[[Any], Any] | None = None, serialize_as_any: bool = False, polymorphic_serialization: bool | None = None) dict[str, Any]
- !!! abstract “Usage Documentation”
[model_dump](../concepts/serialization.md#python-mode)
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
- Parameters:
mode (Literal['json', 'python'] | str) – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to include in the output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to exclude from the output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to use the field’s alias in the dictionary key if defined.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization (bool | None) – Whether to use model and dataclass polymorphic serialization for this call.
- Returns:
A dictionary representation of the model.
- Return type:
- model_dump_json(*, indent: int | None = None, ensure_ascii: bool = False, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, context: Any | None = None, by_alias: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, exclude_computed_fields: bool = False, round_trip: bool = False, warnings: bool | Literal['none', 'warn', 'error'] = True, fallback: Callable[[Any], Any] | None = None, serialize_as_any: bool = False, polymorphic_serialization: bool | None = None) str
- !!! abstract “Usage Documentation”
[model_dump_json](../concepts/serialization.md#json-mode)
Generates a JSON representation of the model using Pydantic’s to_json method.
- Parameters:
indent (int | None) – Indentation to use in the JSON output. If None is passed, the output will be compact.
ensure_ascii (bool) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to include in the JSON output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to exclude from the JSON output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to serialize using field aliases.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization (bool | None) – Whether to use model and dataclass polymorphic serialization for this call.
- Returns:
A JSON string representation of the model.
- Return type:
- property model_extra: dict[str, Any] | None
Get extra fields set during validation.
- Returns:
A dictionary of extra fields, or None if config.extra is not set to “allow”.
- model_fields = {'cpu_usage': FieldInfo(annotation=Union[float, NoneType], required=False, default=0.0), 'custom_metrics': FieldInfo(annotation=dict[str, Any], required=False, default_factory=dict), 'duration': FieldInfo(annotation=Union[float, NoneType], required=False, default=0.0), 'memory_usage': FieldInfo(annotation=Union[float, NoneType], required=False, default=0.0), 'memory_usage_mb': FieldInfo(annotation=Union[float, NoneType], required=False, default=0.0)}
- property model_fields_set: set[str]
Returns the set of fields that have been explicitly set on this model instance.
- Returns:
- A set of strings representing the fields that have been set,
i.e. that were not filled from defaults.
- classmethod model_parametrized_name(params: tuple[type[Any], ...]) str
Compute the class name for parametrizations of generic classes.
This method can be overridden to achieve a custom naming scheme for generic BaseModels.
- Parameters:
params (tuple[type[Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.
- Returns:
String representing the new class where params are passed to cls as type variables.
- Raises:
TypeError – Raised when trying to generate concrete names for non-generic models.
- Return type:
- model_post_init(context: Any, /) None
Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.
- Parameters:
context (Any)
- Return type:
None
- classmethod model_rebuild(*, force: bool = False, raise_errors: bool = True, _parent_namespace_depth: int = 2, _types_namespace: MappingNamespace | None = None) bool | None
Try to rebuild the pydantic-core schema for the model.
This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.
- Parameters:
force (bool) – Whether to force the rebuilding of the model schema, defaults to False.
raise_errors (bool) – Whether to raise errors, defaults to True.
_parent_namespace_depth (int) – The depth level of the parent namespace, defaults to 2.
_types_namespace (MappingNamespace | None) – The types namespace, defaults to None.
- Returns:
Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.
- Return type:
bool | None
- classmethod model_validate(obj: Any, *, strict: bool | None = None, extra: Literal['allow', 'ignore', 'forbid'] | None = None, from_attributes: bool | None = None, context: Any | None = None, by_alias: bool | None = None, by_name: bool | None = None) Self
Validate a pydantic model instance.
- Parameters:
obj (Any) – The object to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
from_attributes (bool | None) – Whether to extract data from object attributes.
context (Any | None) – Additional context to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.
- Raises:
ValidationError – If the object could not be validated.
- Returns:
The validated model instance.
- Return type:
- classmethod model_validate_json(json_data: str | bytes | bytearray, *, strict: bool | None = None, extra: Literal['allow', 'ignore', 'forbid'] | None = None, context: Any | None = None, by_alias: bool | None = None, by_name: bool | None = None) Self
- !!! abstract “Usage Documentation”
[JSON Parsing](../concepts/json.md#json-parsing)
Validate the given JSON data against the Pydantic model.
- Parameters:
json_data (str | bytes | bytearray) – The JSON data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.
- Returns:
The validated Pydantic model.
- Raises:
ValidationError – If json_data is not a JSON string or the object could not be validated.
- Return type:
- classmethod model_validate_strings(obj: Any, *, strict: bool | None = None, extra: Literal['allow', 'ignore', 'forbid'] | None = None, context: Any | None = None, by_alias: bool | None = None, by_name: bool | None = None) Self
Validate the given object with string data against the Pydantic model.
- Parameters:
obj (Any) – The object containing string data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.
- Returns:
The validated Pydantic model.
- Return type:
- classmethod parse_file(path: str | Path, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) Self
- classmethod parse_raw(b: str | bytes, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) Self
Collected Metrics:
CPU Usage: Processor utilization percentage
Memory Usage: RAM consumption in bytes
Disk I/O: Read/write operations and bytes
Network I/O: Sent/received bytes
Custom Metrics: Application-specific measurements
Example Usage:
# Access task metrics
task = task_manager.get_task(task_id)
metrics = task.metrics
print(f"CPU Usage: {metrics.cpu_percent}%")
print(f"Memory: {metrics.memory_bytes / 1024 / 1024:.2f} MB")
print(f"Disk Read: {metrics.disk_read_bytes / 1024 / 1024:.2f} MB")
print(f"Network Sent: {metrics.network_sent_bytes / 1024:.2f} KB")
Storage Backends
FileSystemStorage
Default storage backend using the local filesystem.
- class kailash.tracking.storage.filesystem.FileSystemStorage(base_path: str | None = None)[source]
Bases:
StorageBackendFilesystem-based storage backend.
- Parameters:
base_path (str | None)
- __init__(base_path: str | None = None)[source]
Initialize filesystem storage.
- Parameters:
base_path (str | None) – Base directory for storage. Defaults to ~/.kailash/tracking
- save_run(run: WorkflowRun) None[source]
Save a workflow run.
- Parameters:
run (WorkflowRun)
- Return type:
None
- load_run(run_id: str) WorkflowRun | None[source]
Load a workflow run by ID.
- Parameters:
run_id (str)
- Return type:
WorkflowRun | None
- list_runs(workflow_name: str | None = None, status: str | None = None) list[WorkflowRun][source]
List workflow runs.
- Parameters:
- Return type:
- save_task(task: TaskRun) None[source]
Save a task.
- Parameters:
task (TaskRun) – TaskRun to save
- Raises:
KailashStorageError – If task cannot be saved
- Return type:
None
- get_task(task_id: str) TaskRun | None[source]
Load a task by ID.
- Parameters:
task_id (str) – Task ID to load
- Returns:
TaskRun or None if not found
- Raises:
KailashStorageError – If task cannot be loaded
- Return type:
TaskRun | None
- load_task(task_id: str) TaskRun | None[source]
Load a task by ID.
- Parameters:
task_id (str)
- Return type:
TaskRun | None
- list_tasks(run_id: str, node_id: str | None = None, status: TaskStatus | None = None) list[TaskRun][source]
List tasks for a run.
- update_task(task: TaskRun) None[source]
Update an existing task.
- Parameters:
task (TaskRun) – TaskRun to update
- Raises:
KailashStorageError – If task cannot be updated
- Return type:
None
- delete_task(task_id: str) None[source]
Delete a task.
- Parameters:
task_id (str) – Task ID to delete
- Raises:
KailashStorageError – If task cannot be deleted
- Return type:
None
- get_all_tasks() list[TaskRun][source]
Get all tasks.
- Returns:
List of all TaskRun objects
- Raises:
KailashStorageError – If tasks cannot be retrieved
- Return type:
list[TaskRun]
- query_tasks(node_id: str | None = None, status: TaskStatus | None = None, started_after: datetime | None = None, completed_before: datetime | None = None) list[TaskRun][source]
Query tasks with filters.
- Parameters:
- Returns:
List of matching TaskRun objects
- Raises:
KailashStorageError – If tasks cannot be queried
- Return type:
list[TaskRun]
Configuration:
from kailash.tracking.storage import FileSystemStorage
storage = FileSystemStorage(
base_path="/path/to/tracking/data",
format="json", # or "yaml"
compress=True, # Gzip compression
retention_days=30 # Auto-cleanup old data
)
task_manager = TaskManager(storage=storage)
Directory Structure:
tracking_data/
├── runs/
│ ├── 2024-01-01/
│ │ ├── run_123.json
│ │ └── run_456.json
│ └── 2024-01-02/
│ └── run_789.json
├── tasks/
│ └── run_123/
│ ├── task_001.json
│ └── task_002.json
└── metrics/
└── daily_summary.json
DatabaseStorage
Database backend for scalable storage.
- kailash.tracking.storage.database.DatabaseStorage
alias of
SQLiteStorage
Supported Databases:
PostgreSQL
MySQL
SQLite
MongoDB
Configuration:
from kailash.tracking.storage import DatabaseStorage
# PostgreSQL
storage = DatabaseStorage(
connection_string="postgresql://user:pass@localhost/tracking",
pool_size=10,
echo=False # SQL logging
)
# MongoDB
storage = DatabaseStorage(
connection_string="mongodb://localhost:27017/",
database="kailash_tracking",
collection_prefix="tracking_"
)
Custom Storage Backend
Implement custom storage by extending the base class:
from kailash.tracking.storage.base import BaseStorage
import redis
class RedisStorage(BaseStorage):
"""Redis-based tracking storage."""
def __init__(self, redis_url: str):
self.client = redis.from_url(redis_url)
def save_run(self, run: WorkflowRun) -> None:
key = f"run:{run.run_id}"
self.client.setex(
key,
86400, # 24 hour TTL
run.json()
)
def get_run(self, run_id: str) -> WorkflowRun:
key = f"run:{run_id}"
data = self.client.get(key)
if data:
return WorkflowRun.parse_raw(data)
return None
def save_task(self, task: Task) -> None:
key = f"task:{task.task_id}"
self.client.setex(key, 86400, task.json())
def get_tasks(self, run_id: str) -> List[Task]:
# Implementation for retrieving tasks
pass
Tracking Configuration
Environment Variables
Configure tracking behavior:
# Storage location
export KAILASH_TRACKING_PATH=/var/kailash/tracking
# Storage backend
export KAILASH_TRACKING_BACKEND=filesystem
# Database connection (if using database backend)
export KAILASH_TRACKING_DB=postgresql://localhost/tracking
# Metrics collection
export KAILASH_COLLECT_METRICS=true
export KAILASH_METRICS_INTERVAL=1.0 # seconds
# Retention policy
export KAILASH_TRACKING_RETENTION_DAYS=30
Configuration File
# ~/.kailash/tracking.yaml
tracking:
enabled: true
backend: filesystem
storage:
path: ~/.kailash/tracking
format: json
compress: true
metrics:
enabled: true
interval: 1.0
include:
- cpu
- memory
- disk_io
- network_io
retention:
days: 30
max_runs: 10000
Programmatic Configuration
from kailash.tracking import TaskManager, TrackingConfig
config = TrackingConfig(
enabled=True,
collect_metrics=True,
metric_interval=0.5,
storage_backend="database",
storage_config={
"connection_string": "postgresql://localhost/tracking",
"pool_size": 20
}
)
task_manager = TaskManager(config=config)
Analytics and Reporting
Run Analytics
Analyze workflow execution patterns:
from kailash.tracking.analytics import RunAnalyzer
analyzer = RunAnalyzer(task_manager)
# Get run statistics
stats = analyzer.get_run_stats(
start_date="2024-01-01",
end_date="2024-01-31"
)
print(f"Total runs: {stats['total_runs']}")
print(f"Success rate: {stats['success_rate']:.2%}")
print(f"Average duration: {stats['avg_duration']:.2f}s")
print(f"Failed runs: {stats['failed_runs']}")
Performance Analysis
Identify performance bottlenecks:
# Analyze node performance
node_stats = analyzer.get_node_performance(workflow_id)
for node_id, stats in node_stats.items():
print(f"\nNode: {node_id}")
print(f" Executions: {stats['count']}")
print(f" Avg Duration: {stats['avg_duration']:.2f}s")
print(f" Max Duration: {stats['max_duration']:.2f}s")
print(f" Failure Rate: {stats['failure_rate']:.2%}")
Resource Usage Analysis
Monitor resource consumption:
# Get resource usage trends
resource_trends = analyzer.get_resource_trends(
run_id=run_id,
metric="memory",
interval="1min"
)
# Plot memory usage
import matplotlib.pyplot as plt
plt.plot(resource_trends['timestamps'], resource_trends['values'])
plt.xlabel('Time')
plt.ylabel('Memory (MB)')
plt.title('Memory Usage Over Time')
plt.show()
Custom Reports
Generate custom reports:
from kailash.tracking.reporting import ReportGenerator
generator = ReportGenerator(task_manager)
# Generate HTML report
report = generator.generate_html_report(
run_id=run_id,
include_metrics=True,
include_timeline=True,
include_errors=True
)
with open("execution_report.html", "w") as f:
f.write(report)
# Generate CSV summary
generator.export_run_summary(
output_path="run_summary.csv",
start_date="2024-01-01",
end_date="2024-01-31"
)
Real-time Monitoring
Live Tracking
Monitor workflows in real-time:
from kailash.tracking import LiveMonitor
monitor = LiveMonitor(task_manager)
# Start monitoring
monitor.start()
# Execute workflow
results = workflow.run()
# Get live statistics
live_stats = monitor.get_stats()
print(f"Active tasks: {live_stats['active_tasks']}")
print(f"Completed tasks: {live_stats['completed_tasks']}")
print(f"Failed tasks: {live_stats['failed_tasks']}")
Event Streaming
Stream tracking events:
from kailash.tracking import EventStream
stream = EventStream(task_manager)
# Subscribe to events
@stream.on("task.started")
def on_task_start(event):
print(f"Task started: {event['node_id']}")
@stream.on("task.completed")
def on_task_complete(event):
print(f"Task completed: {event['node_id']} in {event['duration']}s")
@stream.on("run.failed")
def on_run_failed(event):
print(f"Run failed: {event['error']}")
# Send alert
send_failure_alert(event)
# Start streaming
stream.start()
Webhooks
Send tracking events to external systems:
from kailash.tracking import WebhookHandler
webhook = WebhookHandler(
url="https://api.example.com/webhooks/kailash",
events=["run.completed", "run.failed"],
headers={"Authorization": "Bearer token"}
)
task_manager.add_handler(webhook)
Performance Metrics Collection
The SDK includes comprehensive performance metrics collection that automatically tracks resource usage during workflow execution.
MetricsCollector
Collects real-time performance metrics during node execution.
- class kailash.tracking.metrics_collector.MetricsCollector(sampling_interval: float = 0.1, enable_resource_monitoring: bool | None = None)[source]
Bases:
objectCollects performance metrics during task execution.
This class provides context managers for collecting detailed performance metrics during node execution, with support for both process-level and system-level monitoring.
- Usage:
>>> collector = MetricsCollector() >>> with collector.collect() as metrics: ... # Execute node code here ... pass >>> performance_data = metrics.result()
- __init__(sampling_interval: float = 0.1, enable_resource_monitoring: bool | None = None)[source]
Initialize metrics collector.
- Parameters:
sampling_interval (float) – How often to sample metrics (seconds)
enable_resource_monitoring (bool | None) – Whether to enable psutil-based resource monitoring.
None(default) auto-detects: enabled iff psutil is installed.Trueexplicitly opts in (warns if psutil is missing).Falsedisables; only duration is tracked. P0D-001: Dramatically reduces per-node overhead by avoiding thread creation/join.
Usage Example:
from kailash.tracking.metrics_collector import MetricsCollector
# Automatic collection in runtime
collector = MetricsCollector()
with collector.collect(node_id="process_data") as metrics:
# Your node execution code
result = process_data(input_data)
# Access collected metrics
performance = metrics.result()
print(f"Duration: {performance.duration}s")
print(f"CPU Usage: {performance.cpu_percent}%")
print(f"Memory: {performance.memory_mb}MB")
PerformanceMetrics
Comprehensive performance data collected during execution.
- class kailash.tracking.metrics_collector.PerformanceMetrics(duration: float = 0.0, cpu_percent: float = 0.0, memory_mb: float = 0.0, memory_delta_mb: float = 0.0, io_read_bytes: int = 0, io_write_bytes: int = 0, io_read_count: int = 0, io_write_count: int = 0, thread_count: int = 1, context_switches: int = 0, custom: dict[str, ~typing.Any]=<factory>)[source]
Bases:
objectContainer for comprehensive performance metrics.
- Variables:
duration (float) – Execution time in seconds
cpu_percent (float) – Average CPU usage percentage
memory_mb (float) – Peak memory usage in MB
memory_delta_mb (float) – Memory increase during execution
io_read_bytes (int) – Total bytes read during execution
io_write_bytes (int) – Total bytes written during execution
io_read_count (int) – Number of read operations
io_write_count (int) – Number of write operations
thread_count (int) – Number of threads used
context_switches (int) – Number of context switches
- Parameters:
- __init__(duration: float = 0.0, cpu_percent: float = 0.0, memory_mb: float = 0.0, memory_delta_mb: float = 0.0, io_read_bytes: int = 0, io_write_bytes: int = 0, io_read_count: int = 0, io_write_count: int = 0, thread_count: int = 1, context_switches: int = 0, custom: dict[str, ~typing.Any]=<factory>) None
Collected Metrics:
Timing: Start time, end time, duration
CPU: Usage percentage, user/system time
Memory: Current usage, peak usage, delta
I/O: Read/write bytes, operation counts
Network: Bytes sent/received (for API nodes)
Performance Visualization
Visualize and analyze performance metrics from workflow runs.
PerformanceVisualizer
Creates various performance visualizations from collected metrics.
- class kailash.visualization.performance.PerformanceVisualizer(task_manager: TaskManager)[source]
Bases:
objectCreates performance reports from task execution metrics.
Generates Markdown with tables and Mermaid bar charts — renders natively in GitHub, VS Code, JetBrains, and any Markdown viewer.
- Parameters:
task_manager (TaskManager)
- __init__(task_manager: TaskManager)[source]
- Parameters:
task_manager (TaskManager)
Visualization Types:
Execution Timeline - Gantt chart showing node execution order and duration
Resource Usage - Line charts of CPU and memory over time
Performance Comparison - Radar charts comparing multiple runs
I/O Analysis - Bar charts of read/write operations
Performance Heatmap - Visual bottleneck identification
Markdown Reports - Comprehensive performance analysis
Example Usage:
from kailash.visualization.performance import PerformanceVisualizer
from kailash.tracking import TaskManager
# Create visualizer
task_manager = TaskManager()
perf_viz = PerformanceVisualizer(task_manager)
# Generate performance report
outputs = perf_viz.create_run_performance_summary(
run_id="abc-123",
output_dir="performance_report"
)
# Compare multiple runs
perf_viz.compare_runs(
run_ids=["run-1", "run-2", "run-3"],
output_path="comparison.png"
)
Dashboard Creation:
from kailash.workflow.visualization import WorkflowVisualizer
# Create performance dashboard
workflow_viz = WorkflowVisualizer(workflow)
dashboard = workflow_viz.create_performance_dashboard(
run_id=run_id,
task_manager=task_manager,
output_dir="dashboard"
)
# Dashboard includes:
# - dashboard.html (interactive overview)
# - Timeline charts
# - Resource usage graphs
# - Performance heatmaps
# - Detailed metrics tables
Best Practices
Enable Tracking in Production
# Always use tracking in production
task_manager = TaskManager(
storage=DatabaseStorage(connection_string),
config=TrackingConfig(
enabled=True,
collect_metrics=True
)
)
Set Appropriate Retention
# Balance storage vs history
storage = FileSystemStorage(
retention_days=90, # 3 months
archive_old_data=True, # Compress old data
archive_path="/archive/tracking"
)
Monitor Key Metrics
# Focus on important metrics
key_metrics = analyzer.get_key_metrics(run_id)
if key_metrics['memory_peak'] > threshold:
alert("High memory usage detected")
if key_metrics['duration'] > sla_duration:
alert("SLA breach: execution too slow")
Use Appropriate Storage
Development: FileSystemStorage with JSON
Production: DatabaseStorage with PostgreSQL/MongoDB
High Volume: Time-series database or data warehouse
Implement Cleanup
# Regular cleanup job
from kailash.tracking.maintenance import cleanup_old_data
# Run daily
cleanup_old_data(
task_manager,
older_than_days=90,
keep_failed_runs=True
)