Eval

Defaults Eval Functions

eq

source

Signature: eq(a, b)

Compare two values for exact equality.

Parameter	Type	Default	Description
a	None	None	First value to compare
b	None	None	Second value to compare
:Returns:	float	-	Returns 0 if values are equal, infinity if they differ

safe_eval

source

Signature: safe_eval(out, expression)

Safely evaluate a Python expression with a provided value.

This function takes a value and a Python expression template, formats the expression with the value, and evaluates it safely. It handles various error cases and type conversions.

Parameter	Type	Default	Description
out	None	None	Value to insert into the expression template
expression	str	None	Python expression template with {} placeholder for the value
:Returns:	float	-	- For boolean results: 0 if True, infinity if False - For float results: the float value directly - For errors or invalid types: infinity

chat_eval

source

Signature: chat_eval(out: Any, expected: Any, model: str = 'gpt-4o-mini', system_prompt: str = None) -> float

Evaluate similarity between two values using a language model.

This function uses a language model to compare two values and return a similarity score. It can use either a default system prompt or a custom one to guide the evaluation.

Parameter	Type	Default	Description
out	typing.Any	None	First value to compare
expected	typing.Any	None	Second value to compare
model	<class ‘str’>	gpt-4o-mini	Language model to use. Defaults to “gpt-4o-mini”
system_prompt	<class ‘str’>	None	Custom system prompt for evaluation. Must contain{{out}} and {{expected}} jinja variables. If None, uses default prompt.
:Returns:	<class ‘float’>	-	Similarity score between 0 and 1, where: - 0 indicates lowest similarity - 1 indicates highest similarity

cosine_dist

source

Signature: cosine_dist(out: str, expected: str, model: str = 'text-embedding-3-small') -> float

Compute cosine distance between two strings using OpenAI embeddings.

This function converts two strings into embeddings using OpenAI’s embedding model and computes the cosine distance between them. The distance is normalized by the norm of the expected embedding.

Parameter	Type	Default	Description
out	<class ‘str’>	None	First string to compare
expected	<class ‘str’>	None	Second string to compare
model	<class ‘str’>	text-embedding-3-small	OpenAI embedding model to use. Defaults to ‘text-embedding-3-small’
:Returns:	<class ‘float’>	-	Cosine distance between the strings If out is not a string, returns inf

eval_any

source

Signature: eval_any(out: Any, expected: Any) -> float

Accept any value by always returning 0 distance.

This function is used to verify the existence of a key or value without caring about its actual content. It always returns 0, indicating a perfect match regardless of the input values.

Parameter	Type	Default	Description
out	typing.Any	None	First value (ignored)
expected	typing.Any	None	Second value (ignored)
:Returns:	<class ‘float’>	-	Always returns 0, indicating a perfect match

Test Case Schema

Here is the formal TestCase Schema

from stringdale.eval import TestCase

TestCase.model_json_schema()

{'$defs': {'Condition': {'properties': {'key': {'title': 'Key',
     'type': 'string'},
    'value': {'title': 'Value'},
    'func': {'anyOf': [{'type': 'string'}, {'type': 'null'}],
     'default': None,
     'title': 'Func'},
    'kwargs': {'additionalProperties': True,
     'default': {},
     'title': 'Kwargs',
     'type': 'object'},
    'aggregation': {'anyOf': [{'type': 'string'}, {'type': 'null'}],
     'default': None,
     'title': 'Aggregation'}},
   'required': ['key', 'value'],
   'title': 'Condition',
   'type': 'object'},
  'TestNode': {'properties': {'name': {'title': 'Name', 'type': 'string'},
    'label': {'anyOf': [{'type': 'string'}, {'type': 'null'}],
     'default': None,
     'title': 'Label'},
    'conditions': {'items': {'$ref': '#/$defs/Condition'},
     'title': 'Conditions',
     'type': 'array'},
    'before': {'anyOf': [{'items': {'type': 'string'}, 'type': 'array'},
      {'type': 'null'}],
     'title': 'Before'},
    'after': {'anyOf': [{'items': {'type': 'string'}, 'type': 'array'},
      {'type': 'null'}],
     'title': 'After'},
    'parallel': {'anyOf': [{'type': 'boolean'}, {'type': 'null'}],
     'default': False,
     'title': 'Parallel'}},
   'required': ['name', 'conditions'],
   'title': 'TestNode',
   'type': 'object'}},
 'properties': {'inputs': {'items': {}, 'title': 'Inputs', 'type': 'array'},
  'test_nodes': {'items': {'$ref': '#/$defs/TestNode'},
   'title': 'Test Nodes',
   'type': 'array'}},
 'required': ['inputs', 'test_nodes'],
 'title': 'TestCase',
 'type': 'object'}

Semaphor decorator

from stringdale.core import semaphore_decorator

semaphore_decorator

source

Signature: semaphore_decorator(concurrency: int = 0, method_name: str = None)

A decorator that limits concurrent execution of functions or class methods using an async semaphore.

This decorator can be applied to both async functions and classes. When applied to a class, it can limit concurrent executions of a specific method (defaults to call) across all instances of that class.

Parameter	Type	Default	Description
concurrency	<class ‘int’>	0	The maximum number of concurrent executions allowed. If 0, no concurrency limit is applied.
method_name	<class ‘str’>	None	When decorating a class, specifies which method to apply the semaphore to.Defaults to ‘call’. Must be None when decorating a function.