Optimizing Airport METAR Forecasts with Machine Learning and LLM Techniques

Introduction

Meteorological reports, especially METARs, deliver critical real-time weather information essential for aviation and various other sectors. Accurate predictions of these conditions can greatly improve safety and operational efficiency. This article explores the development of machine learning models and LLMs specifically designed to forecast the meteorological variables reported in METARs for different airports. The project, along with all relevant resources, can be found in this github repository

Data sources

The meteorological data for this analysis is sourced from the WRF model, provided by METEOGALICIA, the regional meteorological service. Additionally, METAR reports are obtained from Iowa State University. Our study utilizes data from the years 2018 to 2022 to train and test various algorithms. The WRF forecasts are based on the four grid points closest to each airport’s runway, with a model resolution of 4 km between points

Steps to Follow to prepare and train LSTM model:

1. Generate a Template with Model Outputs and METAR Reports:

• Begin by generating a template that is an array of strings. Each string should consist of two parts:
• The first eight words should be the categorical outputs predicted by the machine learning models.
• The rest of the string should be the actual METAR report.
• This can be accomplished using the <ICAO-code>fusion_ml.ipynb notebook. The notebook requires input files named in the format variable_code_<ICAO_code>_d<x>, where:
• variable represents the meteorological variables forecasted by the machine learning models.
• <ICAO_code> is the airport's ICAO code.

Example:

array(['31001KT 9999 WM OVC010 M 12 10 Q1017 VRB01KT 8000 OVC008 11/11 Q1017',
       '16005KT 9999 WM MNClD M 22 17 Q1013 16004KT 110V220 CAVOK 24/18 Q1012 NOSIG'])

2. Prepare Text Data for Training an LSTM Model:

• Use the <ICAO-code>train_ml.ipynb notebook to prepare the text data for training a sequence-based model, such as an LSTM (Long Short-Term Memory) network in TensorFlow/Keras.
• Steps Involved:
• Splitting Data into Training and Testing: Divide the data into training and testing sets.
• Tokenization: Create a tokenizer to convert the text data into sequences of integers based on the tokenizer’s word index
• One-Hot Encoding the Output: transforming the integer labels into a binary matrix

3. Build, Compile Train and save the LSTM Keras model

Example:

# Build the LSTM model
model = Sequential([
    Embedding(input_dim=len(tokenizer.word_index)+1, output_dim=20, input_length=sequence_length),
    Bidirectional(LSTM(130)),

    Dense(len(tokenizer.word_index)+1, activation='softmax')
])

# Compile the model
model.compile(loss='categorical_crossentropy', optimizer=Adam(learning_rate=0.01), metrics=['accuracy'])

# Train the model
model.fit(X, y, epochs=70,batch_size=512)

4. Check the model with <ICAO-code>mlcheck.ipynb.

To forecast METAR, we utilize the Keras algorithm beginning from the specified test date (point 2). We then retrieve both the actual METAR data and the predicted METAR data, which allows us to compare and evaluate the performance of the Machine Learning model against the Language Model (LLM).

In the example below, the first string represents the real METAR data, while the second string shows the forecast generated by the LLM algorithm. The first eight words of the final string represent the input words for the LLM algorithm.

Example:

• Actual METAR: '16005KT 120V210 9999 BKN038 15/15 Q1015 TEMPO 3000 RA'
• LLM Forecast: 'vrb02kt 9999 ra bkn033 16 15 q1015 nosig'
• Input from Machine Learning algorithms (first 8 words): '17009kt 5000 RA BKN040 BKN 16 14 q1015'

We can generate graphical results to compare and evaluate the Machine Learning and LLM forecasts for each meteorological variable in the METAR report. This visual comparison allows us to score the accuracy and performance of both forecasting methods

METAR pressure variable versus metorological model forecast. MAE: Mean absolute error

METAR pressure variable versus LLM model forecast. MAE: Mean absolute error

Additionally, the notebook generates a confusion matrix for present weather conditions, as illustrated in the example below:

This confusion matrix allows us to assess the accuracy of the weather predictions by comparing the actual conditions with the model’s forecasts. Metrics such as precision, recall, and overall accuracy can be derived from this matrix to evaluate and improve the performance of both the Machine Learning model and the LLM.

The files contain the actual present weather data, where “WM” indicates that no present weather was reported. The columns represent the LLM forecasts.

Deployment of the LLM model with mlmetar_forecast.py

The script mlmetar_forecast.py is designed for real-time deployment of an LLM (Language Model) for METAR forecasting. It presents the results through a user-friendly Streamlit interface. The script follows these key steps:

1. Download Meteorological Data: The script first downloads the current meteorological model (WRF) from the METEOGALICIA server. This model provides essential weather data as input for the forecasting process.
2. Load Trained Machine Learning Algorithms: Next, the script loads the pre-trained machine learning models, which are used to forecast various meteorological variables. These algorithms take the meteorological model’s variables as input.
3. Construct Input String: The script then builds a string by concatenating the input meteorological variables. This string serves as the input for the LLM.
4. Forecast Using the Keras Model: Finally, the script applies the Keras-based LLM model to the constructed string. The model predicts the next word (meteorological variable) in the sequence, and this process is repeated iteratively to generate a complete forecast.
5. Display Quality Report: Finally, the script generates and displays a quality report that evaluates the performance of the different machine learning algorithms. This report helps in assessing the accuracy and reliability of the forecasts.

By integrating these steps, the script efficiently forecasts METAR reports and displays the results in real-time through an intuitive Streamlit interface. To deploy the LLM model that generates the complete METAR report, visit this Streamlit app.