l5kit.data package

class l5kit.data.ChunkedDataset(path: str, key: str = '')

Bases: object

ChunkedDataset is a dataset that lives on disk in compressed chunks, it has easy to use data loading and writing interfaces that involves making numpy-like slices.

Currently only .zarr directory stores are supported (i.e. the data will live in a folder on your local filesystem called <something>.zarr).

initialize(mode: str = 'w', num_scenes: int = 0, num_frames: int = 0, num_agents: int = 0, num_tl_faces: int = 0)l5kit.data.zarr_dataset.ChunkedDataset

Initializes a new zarr dataset, creating the underlying arrays.

Keyword Arguments
  • mode (str) – Mode to open dataset in, should be something that supports writing. (default: {“w”})

  • num_scenes (int) – pre-allocate this number of scenes

  • num_frames (int) – pre-allocate this number of frames

  • num_agents (int) – pre-allocate this number of agents

  • num_tl_faces (int) – pre-allocate this number of traffic lights

open(mode: str = 'r', cached: bool = True, cache_size_bytes: int = 1000000000)l5kit.data.zarr_dataset.ChunkedDataset

Opens a zarr dataset from disk from the path supplied in the constructor.

Keyword Arguments:

mode (str): Mode to open dataset in, default to read-only (default: {“r”}) cached (bool): Whether to cache files read from disk using a LRU cache. (default: {True}) cache_size_bytes (int): Size of cache in bytes (default: {1e9} (1GB))

Raises:

Exception: When any of the expected arrays (frames, agents, scenes) is missing or the store couldn’t be

opened.

class l5kit.data.DataManager

Bases: abc.ABC

abstract require(key: str) → str
class l5kit.data.LocalDataManager(local_data_folder: Optional[Union[str, pathlib.Path]] = None)

Bases: l5kit.data.local_data_manager.DataManager

LocalDataManager allows you to require data to be present in the subpath of a specific folder.

Example: Your data folder is set to "/tmp/my-data-folder", and you call local_data_manager.require("path/to/image.jpg"), it would check if "/tmp/my-data-folder/path/to/image.jpg" exists, and if so return that complete path ("/tmp/my-data-folder/path/to/image.jpg"), otherwise it raises an error.

In order of precedence, the local data folder is set by
  1. Passing in the path to the constructor of LocalDataManager

  2. Setting the L5KIT_DATA_FOLDER environment variable.

require(key: str) → str

Require checks whether the file with the given key is present in the local data folder, if it is not it raises an error. Returns the path to the file otherwise.

Parameters

key (str) – Path from the data folder where the file or folder should be present.

Returns

str – Filepath including the data folder where required key is present.

class l5kit.data.MapAPI(protobuf_map_path: str, world_to_ecef: numpy.ndarray)

Bases: object

get_crosswalk_coords(element_id: str) → dict

Get XYZ coordinates in world ref system for a crosswalk given its id lru_cached for O(1) access

Parameters

element_id (str) – crosswalk element id

Returns

a dict with the polygon coordinates as an (Nx3) XYZ array

Return type

dict

get_lane_coords(element_id: str) → dict

Get XYZ coordinates in world ref system for a lane given its id lru_cached for O(1) access

Parameters

element_id (str) – lane element id

Returns

a dict with the two boundaries coordinates as (Nx3) XYZ arrays

Return type

dict

static id_as_str(element_id: road_network_pb2.GlobalId) → str

Get the element id as a string. Elements ids are stored as a variable len sequence of bytes in the protobuf

Parameters

element_id (GlobalId) – the GlobalId in the protobuf

Returns

the id as a str

Return type

str

static is_crosswalk(element: road_network_pb2.MapElement) → bool

Check whether an element is a valid crosswalk

Parameters

element (MapElement) – a proto element

Returns

True if the element is a valid crosswalk

Return type

bool

static is_lane(element: road_network_pb2.MapElement) → bool

Check whether an element is a valid lane

Parameters

element (MapElement) – a proto element

Returns

True if the element is a valid lane

Return type

bool

is_traffic_face_colour(element_id: str, colour: str) → bool

Check if the element is a traffic light face of the given colour

Parameters
  • element_id (str) – the id (utf-8 encode) of the element

  • colour (str) – the colour to check

Returns

True if the element is a traffic light with the given colour

unpack_deltas_cm(dx: Sequence[int], dy: Sequence[int], dz: Sequence[int], frame: road_network_pb2.GeoFrame) → numpy.ndarray

Get coords in world reference system (local ENU->ECEF->world). See the protobuf annotations for additional information about how coordinates are stored

Parameters
  • dx (Sequence[int]) – X displacement in centimeters in local ENU

  • dy (Sequence[int]) – Y displacement in centimeters in local ENU

  • dz (Sequence[int]) – Z displacement in centimeters in local ENU

  • frame (GeoFrame) – geo-location information for the local ENU. It contains lat and long origin of the frame

Returns

array of shape (Nx3) with XYZ coordinates in world ref system

Return type

np.ndarray

l5kit.data.filter_agents_by_frames(frames: numpy.ndarray, agents: numpy.ndarray) → List[numpy.ndarray]

Get a list of agents array, one array per frame. Note that “agent_index_interval” is used to filter agents, so you should take care of re-setting it if you have previously sliced agents.

Parameters
  • frames (np.ndarray) – an array of frames

  • agents (np.ndarray) – an array of agents

Returns

List[np.ndarray] with the agents divided by frame

l5kit.data.filter_agents_by_labels(agents: numpy.ndarray, threshold: float = 0.5) → numpy.ndarray

Filters an agents array, keeping those agents that meet the threshold.

Parameters

agents (np.ndarray) – Agents array

Keyword Arguments

threshold (float) – probability threshold for filtering (default: {0.5})

Returns

np.ndarray – A subset of input agents array.

l5kit.data.filter_agents_by_track_id(agents: numpy.ndarray, track_id: int) → numpy.ndarray

Return all agent object (np.ndarray) of a given track_id.

Parameters
  • agents (np.ndarray) – agents array. NOTE: do NOT pass a zarr to this function, it can’t handle boolean indexing

  • track_id (int) – agent track id to select

Returns

np.ndarray – Selected agent.

l5kit.data.filter_tl_faces_by_frames(frames: numpy.ndarray, tl_faces: numpy.ndarray) → List[numpy.ndarray]

Get a list of traffic light faces arrays, one array per frame. This functions mimics filter_agents_by_frames for traffic light faces

Parameters
  • frames (np.ndarray) – an array of frames

  • tl_faces (np.ndarray) – an array of traffic light faces

Returns

List[np.ndarray] with the traffic light faces divided by frame

l5kit.data.filter_tl_faces_by_status(tl_faces: numpy.ndarray, status: str) → numpy.ndarray

Filter tl_faces and keep only active faces :param tl_faces: array of traffic faces :type tl_faces: np.ndarray :param status: status we want to keep TODO refactor for enum :type status: str

Returns

traffic light faces array with only faces with that status

Return type

np.ndarray

l5kit.data.get_agents_slice_from_frames(frame_a: numpy.ndarray, frame_b: Optional[numpy.ndarray] = None) → slice

Get a slice for indexing agents giving a start and end frame

Parameters
  • frame_a (np.ndarray) – the starting frame

  • frame_b (Optional[np.ndarray]) – the ending frame. If None, then frame_a end will be used

Returns

a slice object starting from the first agent in frame_a to the last one in frame_b

Return type

slice

l5kit.data.get_combined_scenes(scenes: numpy.ndarray) → numpy.ndarray

Takes as input an np.ndarray or zarr array with scenes, and combines scenes that follow up eachother perfectly (i.e. from a single recording by the same host). Returns an np.ndarray of combined scenes.

Arguments:

scenes (np.ndarray): scenes

Returns:

np.ndarray – combined scenes where followup scenes have been merged.

l5kit.data.get_frames_slice_from_scenes(scene_a: numpy.ndarray, scene_b: Optional[numpy.ndarray] = None) → slice

Get a slice for indexing frames giving a start and end scene

Parameters
  • scene_a (np.ndarray) – the starting scene

  • scene_b (Optional[np.ndarray]) – the ending scene. If None, then scene_a end will be used

Returns

a slice object starting from the first frame in scene_a to the last one in scene_b

Return type

slice

l5kit.data.get_tl_faces_slice_from_frames(frame_a: numpy.ndarray, frame_b: Optional[numpy.ndarray] = None) → slice

Get a slice for indexing traffic light faces giving a start and end frame

Parameters
  • frame_a (np.ndarray) – the starting frame

  • frame_b (Optional[np.ndarray]) – the ending frame. If None, then frame_a end will be used

Returns

a slice object starting from the first tl_face in frame_a to the last one in frame_b

Return type

slice

l5kit.data.zarr_concat(input_zarrs: List[str], output_zarr: str) → None

Concat many zarr into a single one. Takes care of updating indices for frames and agents.

Parameters
  • input_zarrs (List[str]) – a list of paths to input zarrs

  • output_zarr (str) – the path to the output zarr

Returns: