Predicting Limit Order Fill Probability: A Machine Learning Approach

An analytical deep dive into modeling limit order fill probabilities in high-frequency regimes using feature engineering on queue position, order book imbalances, and Gradient Boosted Trees (XGBoost).

1. Introduction: The Order Routing Dilemma in HFT

When executing orders on fragmented electronic markets, execution algorithms face a fundamental tradeoff: send an aggressive Market Order and guarantee execution while crossing the bid-ask spread and incurring slippage, or submit a passive Limit Order and capture the spread (or earn exchange rebates) while risking non-execution and adverse selection.

Predicting the fill probability of a limit order is key to optimizing execution performance. Quantitative analysis and machine learning allow high-frequency trading (HFT) engines to evaluate the probability that a limit order posted at a specific depth will be filled within a short time window $\Delta t$.

2. Microstructure Features for Limit Order Modeling

To accurately predict order fills, the model feature vector must capture high-frequency dynamics. Algorithms rely on the following primary order book features:

Feature Category	Metric Name	Formula / Representation	Predictive Target
Order Book Imbalance (OBI)	Volume Imbalance	$OBI_t = \frac{V_t^{bid} - V_t^{ask}}{V_t^{bid} + V_t^{ask}}$	Short-term pressure and price direction shifts
Queue Distance	Spread Relative Distance	$D_{queue} = \|P_{order} - P_{mid}\|$	Time-to-fill scaling based on depth placement
Volatility Analytics	Realized Volatility	Rolling standard deviation of mid-price	Probability of execution during wide price swings
Queue Imbalance	Bayesian Queue Position	Estimated priority level in matching engine	Time-to-fill threshold for specific orders

3. Mathematical Formulation of Limit Order Fill

From a mathematical standpoint, the execution of a limit order with limit price $P_{limit}$ and order size $Q_{order}$ submitted at time $t$ can be formulated as a binary random variable $Y \in \{0, 1\}$ over a time horizon $\tau$:

$Y = \begin{cases} 1, & \text{if order is fully filled within } [t, t+\tau] \\ 0, & \text{otherwise} \end{cases}$

Our goal is to build a classifier that estimates the conditional probability of execution:

$p(X_t) = P(Y = 1 \mid X_t)$

Where $X_t$ is the vector of LOB microstructure features at time $t$. To train this model, HFT execution pipelines record tick logs, label orders as filled or cancelled/expired, and minimize a binary cross-entropy loss function.

4. Leveraging XGBoost for Algorithmic Placement

Gradient Boosted Decision Trees (specifically the XGBoost library) deliver state-of-the-art performance for tabular microstructure classification. The main advantages in HFT are:

**Non-linear Relationships**: Decision trees easily handle non-linear interactions between volatility, spread, and queue imbalances.
**Feature Importance**: XGBoost models provide native feature contribution rankings for explainable alpha generation.
**Sub-Microsecond Inference**: Trained trees can be exported into highly optimized C templates, allowing CPU-based inference latency below 10 microseconds.

5. Python Example: Training a Fill Probability Predictor

The following Python script illustrates the process of preparing features and training an XGBoost classifier to estimate limit order fill probabilities:

import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

def train_fill_probability_model(order_events_df):
    """
    Trains an XGBoost classifier to predict limit order fill probability.
    Used in algorithmic execution to optimize order placement.
    """
    # 1. Feature selection
    features = [
        'order_book_imbalance', 
        'queue_distance', 
        'realized_volatility_5s', 
        'bid_ask_spread',
        'depth_volume_ratio'
    ]
    
    X = order_events_df[features]
    y = order_events_df['is_filled'] # 1 - filled, 0 - cancelled/expired
    
    # 2. Train-Test split
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
    # 3. Initialize XGBoost Classifier
    model = xgb.XGBClassifier(
        n_estimators=100,
        max_depth=5,
        learning_rate=0.05,
        objective='binary:logistic',
        eval_metric='logloss',
        random_state=42
    )
    
    # 4. Train Model
    model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
    
    print("XGBoost Model successfully trained.")
    return model

# Example inference
# new_state = pd.DataFrame([[0.45, 1.0, 0.015, 0.05, 0.8]], columns=features)
# prob = model.predict_proba(new_state)[0][1] # Estimated execution probability

6. Conclusion

Utilizing gradient boosted trees to estimate limit order fill probabilities allows execution engines to decrease transaction costs by 15–20%. Systematic queue prediction gives trading algorithms the tools to optimize order routing, manage adverse selection risks, and improve execution quality across global fragmented venues.