Skip to content

UpdateDatasetMetadata API

Update the metadata associated with your Dataset

The UpdateDatasetMetadata API allows you to update the metadata for a dataset in a programmatic way.

Data Types

MetadataUpdate

The MetadataUpdate class holds the metadata attributes which you want to update. Only the following fields can be updated:

benchmark Optional[str]

The name of the benchmark. Must not be an empty string or a string containing only whitespace.

accuracy Optional[Union[float, int]]

The accuracy score. Must not be negative.

name Optional[str]

The name of the agent that achieved this accuracy score on the given benchmark. Must not be an empty string or a string containing only whitespace.

If benchmark=web-arena, accuracy=0.95, name=gpt-4o: this means that for the gpt-4o agent on the web-arena benchmark we achieved 95% accuracy.

UpdateDatasetMetadataRequest

The UpdateDatasetMetadataRequest class holds the request data for updating metadata.

dataset_name str

The name of the dataset.

replace_all bool

When replace_all is False:

  1. If an attribute doesn't exist or is set to None in metadata, it is ignored.

  2. For the other attributes in metadata, the values are used to update the metadata for the dataset.

When replace_all is True:

  1. If an attribute doesn't exist or is set to None in metadata, the corresponding key is deleted from the dataset metadata.

  2. For the other attributes in metadata, the values are used to update the metadata for the dataset.

If replace_all=True is passed but metadata is empty - that deletes all set metadata for the dataset.

metadata MetadataUpdate

The metadata class which contains attributes to update.

Updating metadata

There are two SDK methods to update metadata: update_dataset_metadata and create_request_and_update_dataset_metadata. The former accepts the UpdateDatasetMetadataRequest type as an argument and the latter accepts Python-native types as arguments.

update_dataset_metadata

The update_dataset_metadata method is used to update the metadata for a dataset.

request UpdateDatasetMetadataRequest

The request object containing information to update the metadata.

request_kwargs Optional[Dict[str, Any]]

Additional keyword arguments to pass to the requests method. Default is None.

Return Value
Dict

The response object from the Invariant API.

AsyncClient Example 

from invariant_sdk.async_client import AsyncClient
from invariant_sdk.types.update_dataset_metadata import UpdateDatasetMetadataRequest, MetadataUpdate

client = AsyncClient()

# Metadata state: {}

request_1 = UpdateDatasetMetadataRequest(
    dataset_name="some_name",
    metadata=MetadataUpdate(benchmark="some_benchmark")
)
response_1 = await client.update_dataset_metadata(request_1)

# Metadata state: {"benchmark": "some_benchmark"}

request_2 = UpdateDatasetMetadataRequest(
    dataset_name="some_name",
    metadata=MetadataUpdate(accuracy=5, name="xyz")
)

response_2 = await client.update_dataset_metadata(request_2)

# Metadata state: {"benchmark": "some_benchmark", "accuracy": 5, "name": "xyz"}

request_3 = UpdateDatasetMetadataRequest(
    dataset_name="some_name",
    replace_all=True
    metadata=MetadataUpdate(benchmark="new_benchmark")
)

response_3 = await client.update_dataset_metadata(request_3)

# Metadata state: {"benchmark": "new_benchmark"}

AsyncClient Example to clear all previously set metadata 

from invariant_sdk.async_client import AsyncClient
from invariant_sdk.types.update_dataset_metadata import UpdateDatasetMetadataRequest, MetadataUpdate

client = AsyncClient()

request = UpdateDatasetMetadataRequest(
    dataset_name="some_name",
    replace_all=True,
    metadata=MetadataUpdate()
)

response = await client.update_dataset_metadata(request)

create_request_and_update_dataset_metadata

The create_request_and_update_dataset_metadata method is used to update the metadata for a dataset.

dataset_name str

The name of the dataset.

replace_all bool

The replace_all field as described above.

metadata Optional[Dict]

The dictionary containing the metadata to be used for update. Default is None.

request_kwargs Optional[Mapping]

Additional keyword arguments to pass to the requests method. Default is None.

Return Value
Dict

The response object from the Invariant API.

AsyncClient Example 

from invariant_sdk.async_client import AsyncClient

client = AsyncClient()

# Metadata state: {}

response_1 = await client.create_request_and_update_dataset_metadata(
    dataset_name="some_name",
    metadata={"benchmark": "some_benchmark"}
)

# Metadata state: {"benchmark": "some_benchmark"}

response_2 = await client.create_request_and_update_dataset_metadata(
    dataset_name="some_name",
    metadata={"accuracy": 5, "name": "xyz"}
)

# Metadata state: {"benchmark": "some_benchmark", "accuracy": 5, "name": "xyz"}

response_3 = await client.create_request_and_update_dataset_metadata(
    dataset_name="some_name",
    replace_all=True,
    metadata={"benchmark": "new_benchmark"}
)

# Metadata state: {"benchmark": "new_benchmark"}

AsyncClient Example to clear all previously set metadata 

from invariant_sdk.async_client import AsyncClient

client = AsyncClient()

response = await client.create_request_and_update_dataset_metadata(
    dataset_name="some_name"
    replace_all=True,
)