Trajectory Prediction Training Data: Why Forecasting What Objects Will Do Is the Hardest Problem in AV Perception
Every other layer in the AV perception stack has a verifiable ground truth in the current sensor frame: the object is there or it isn’t, the road boundary is here or it isn’t, the traffic signal is red or it isn’t. Trajectory prediction has no current-frame ground truth. Its ground truth is the future where an object actually went in the seconds following the observation. The model’s quality can only be measured after time has passed.
This makes trajectory prediction both the most theoretically challenging perception problem in autonomous driving and the training data problem that requires the most careful thinking about what ground truth means and how to collect it.
Why Trajectory Prediction Is Different From Other Perception Tasks
Detection and tracking models learn from the current sensor frame with reference to known ground truth. Trajectory prediction models learn from the history of an object’s behavior to forecast its future behavior, a fundamentally different learning task.
The inputs to a trajectory prediction model are the historical states of the objects in the scene: their positions, velocities, headings, and interaction histories over the past several seconds. The output is a distribution over possible future trajectories where the model predicts not a single future path but a set of plausible paths with associated probabilities.
The multimodal nature of trajectory prediction is what makes it hard. A vehicle approaching an intersection might turn left, turn right, or continue straight. These are three distinct outcomes, each with some probability given the vehicle’s current state and the context. A model that predicts only the single most likely outcome misrepresents the uncertainty inherent in the prediction and a planning system that relies on single-outcome predictions will be surprised by the outcomes it didn’t predict.
Training data for trajectory prediction needs to capture this multimodality: for each scenario type, the training data should include examples of all the plausible outcomes, with representation proportional to their actual frequency. A training dataset where all intersection-approaching vehicles turn right because the collection route always involves right turns at that intersection teaches the model that vehicles always turn right at intersections which is wrong.
The Training Data Components That Trajectory Prediction Requires
Annotated Historical Trajectories
The historical trajectory of each tracked object, its position, velocity, acceleration, and heading at each timestep over the preceding 1–5 seconds is the primary model input. This requires:
- Precise position tracking: Sub-meter position accuracy at each timestep for each tracked object, across the full scene. The tracking precision that object detection provides is sufficient for detection but may not be sufficient for the velocity and acceleration estimates that trajectory prediction relies on. High-precision differential GPS-equipped reference vehicles in the training data provide ground-truth trajectories for at least some objects.
- Consistent temporal sampling: The historical trajectory needs to be sampled at a consistent rate typically matching the sensor frame rate without missing timesteps. Gaps in the historical trajectory (from brief detection failures) that are filled with interpolation rather than actual observations introduce uncertainty into the prediction input that the model wasn’t trained to handle.
- State completeness: Beyond position, the historical trajectory needs velocity and heading at each timestep. Velocity is computed from the position sequence (with associated uncertainty), and heading is estimated from the velocity direction (also with associated uncertainty). The quality of these derived state estimates bounds the prediction model’s performance.
Future Trajectory Ground Truth
The ground truth that trajectory prediction models train against is the actual future trajectory of each object what it actually did in the seconds after the observation point. Collecting this requires continuing to track objects in the annotated data after the prediction horizon:
- Track completion: Every trajectory in the training data needs to be followed through the end of its presence in the recording, not just to the prediction point. A vehicle that is tracked up to a prediction point and then drops out of the annotation (because the annotators focused on the prediction point context) provides no future trajectory ground truth.
- Consistent ground truth format: The future trajectory ground truth needs to be in the same coordinate frame and state representation as the historical trajectory input. Future trajectories expressed in the vehicle’s current reference frame (which rotates with the vehicle’s heading) need to account for the vehicle’s motion during the prediction horizon if the planning system operates in a world-frame coordinate system.
- Multi-modal ground truth representation: For scenarios where multiple outcomes are possible, the training data captures only what actually happened, which is one sample from the distribution of possible outcomes. Training on single outcomes for inherently multimodal scenarios requires either collecting many examples of each scenario type (so that the distribution of outcomes is represented across examples) or using explicit multi-outcome labeling for the predicted distribution.
Map Context and Semantic Labels
Trajectory prediction accuracy improves substantially when the model can reference the semantic structure of the environment, road topology, lane connectivity, intersection types, traffic control context. This map-conditioned prediction requires the AV Perception Model Training Data training data to include:
- HD map alignment: Each training scene annotated with the corresponding HD map elements visible in that scene lane boundaries, lane connectivity, intersection geometry, traffic sign locations.
- Agent-map relationship labels: Which lane is each vehicle in? Is a pedestrian on the sidewalk or in the road? Is a cyclist using the bike lane or the vehicle lane? These agent-map relationship labels give the prediction model the semantic context that improves its forecasting of lane-adherent versus lane-changing behavior.
- Traffic control state: Traffic signal states at the prediction point which signals are red and which are green contextualizes vehicle behavior at signalized intersections. A vehicle decelerating toward a green light has a different trajectory distribution than the same vehicle decelerating toward a red light.
Interaction Graph Annotations
Real traffic involves interactions with drivers adjusting their behavior in response to other road users. A vehicle that yields to a pedestrian entering its path, a cyclist that swerves around a door opening into the bike lane, a vehicle that slows to create a gap for a merging vehicle these are interaction behaviors whose prediction requires the model to understand both agents’ states and the interaction dynamics between them.
Interaction graph annotations label the relationships between agents that are influencing each other’s behavior: which agent is causing which other agent to adjust its trajectory, and in what way (yielding, following, gap-seeking, avoiding). These annotations are more demanding to produce than trajectory annotations; they require annotators to make behavioral interpretation judgments that go beyond position labeling.
The Rare Interaction Coverage Problem
The distribution of trajectory prediction training data is heavily skewed toward common scenarios: vehicles following other vehicles, pedestrians walking on sidewalks, cyclists riding in bike lanes. The rare interaction scenarios pedestrians making unexpected crossings, vehicles executing unusual maneuvers, cyclists and vehicles in close proximity at intersections are exactly the scenarios where trajectory prediction failure has the highest safety consequence.
Addressing rare interaction coverage requires:
Scenario-type stratification: Classifying each training scenario by its interaction type and monitoring coverage of each type against targets. Common scenarios are well-covered by naturalistic collection; rare interaction types require targeted collection or simulation augmentation.
Simulation-augmented rare scenario data: For the rarest interaction types pedestrian steps from between parked vehicles, vehicle executes emergency lane change simulation generates examples at sufficient density for the model to learn these patterns, while real-world data provides the realism that pure simulation lacks.
Balanced sampling during training: Training with sampling weights that upweight rare scenario types relative to their natural frequency in the dataset, so that the model’s learning is not dominated by common scenarios at the expense of rare ones.
Final Thought
Trajectory prediction training data is the most complex annotation task in AV perception because the ground truth is the future. Programs that collect complete trajectory sequences, annotate future outcomes with enough coverage of rare interaction types, include map context and agent-map relationship labels, and represent the multimodal distribution of possible outcomes produce training data that supports prediction models capable of handling the full behavioral complexity of real traffic.
Programs that annotate only the most common interaction types and collect only the most common scenarios produce prediction models that perform well in test conditions and generate surprises in deployment when the uncommon scenarios arrive.