Skip to main content

Custom metric authoring

aip is a lean, dependency-light SDK for declaring ops on the AIP platform. This page covers @aip.metric, the decorator a custom-op author uses to declare a metric outside Resaro's built-in metrics libraries, and aip.data, which reads and writes the rows a running op is handed.

The @aip.metric decorator​

@aip.metric composes @aip.ops (kind="metric") with a metric's registry metadata, validating all of it at decoration time:

import aip

@aip.metric(
    name="acme.readability",
    version="1.0.0",
    scorer_contract="per_row",
    direction="higher_is_better",
    accepts=["gdi_text_v1"],
    required_columns=["input_id", "sut_response"],
)
def score_readability(*, sut_response: str, **_) -> float:
    ...

It is a decorator and nothing more: it validates, composes @aip.ops, attaches the validated MetricSpec to the function, and returns the function unchanged. A caller holding the function reads its metadata back with aip.metrics.get_spec(fn).

Why this lives in aip, not a family package​

Every other family (aip-data, aip-sdg, ...) layers its metadata onto @aip.ops via its own decorator in its own package. The metric family is the exception: @aip.metric lives in the lean aip package itself rather than in aip-metrics-api.

The reason is who authors a metric and what they need to import. A custom op — authored outside Resaro and built into a customer image — reaches the authoring surface with aip alone, instead of dragging in aip-metrics-api and the whole built-in scoring closure (six-family scorer dispatch, pycocotools, artifact assembly) for an image that will never run a built-in metric.

There is also no metric registry in aip itself. Name-keyed registries exist so score() can resolve one of the built-in metrics by name in a process that imported all of them; a customer image holds exactly one op and has nothing to resolve. Those registries stay in aip-metrics-api, so built-in metrics keep importing aip_metrics unchanged.

The metrics extra​

@aip.metric needs contract types from aip-core, so it is behind an extra:

pip install aip[metrics]

aip.metrics is resolved lazily (PEP 562), so a plain import aip still costs only pydantic — the property the rest of the package relies on. The cost is paid on first touch of aip.metric, by the code that wants it. Without the extra installed, that first touch raises an ImportError naming the extra.

That cost is not small, and the deferral moves it rather than removing it: aip-core's ops_registry reaches pandas and pandera through eager top-level imports, so an op image that authors a metric still pays for them. What the extra buys is that an image which only runs an op, or only imports aip.ops, does not.

Reading and writing data​

A declaration says what the op is. aip.data is how it gets its rows at runtime — three calls, all taking the payload the platform passed the handler, never a URL:

rows = aip.data.read(payload)          # every row this invocation must process
aip.data.write(payload, rows)          # publish the results

with aip.data.batches(payload) as rows:  # bounded memory, for a large dataset
    for batch in rows:
        rows.write(score(batch))

No storage client, no credentials, and no need to know how the rows are encoded. write omits the DataFrame index, because results are read by column.

batches() holds one batch in memory at a time. Leaving the with block normally publishes the result; leaving it because something raised discards the partial result and re-raises, so a half-scored artifact never appears where the platform would read it as complete. The batch size comes from the op's declared execution={"stream_batch_size": N}, or batch_size= at the call site, or a default of 5000.

Declare execution as a dict literal. The platform reads a handler without running it, so ExecutionHints(...) cannot be read. A custom op may set stream_batch_size, timeout_seconds and max_concurrency; anything else is refused at publish with an error explaining why.

read and write also accept a file:// URL or a plain path, so a handler can be exercised against local Parquet before it is published.

aip.data sits behind the same metrics extra as @aip.metric, resolved just as lazily — it needs pandas and the batched dataset reader, both of which come with aip-core.

This is ergonomics and correctness, not containment: the URLs stay on the payload and an author can still read them, because an op is the author's own code and the URL has to exist in that process for the rows to be fetched. What bounds exposure is that the op never receives a storage credential, only URLs scoped to single objects and signed with a secret it does not hold.

Publishing the op​

Publishing is a platform call rather than something aip does: POST /workspaces/{workspace_id}/custom_ops with the handler file, which builds the image, deploys it and registers the op. The CLI wraps it as aip op publish handler.py --workspace-id ....