AI Challenge

May 5, 2025

Dear Participants,

The Phase 1 scores are posted in the leaderboard as follows:

• Public weighted scores, which are obtained from the public dataset.
• Private weighted scores, which are obtained from the private dataset.
• Preliminary Model scores, which are computed as: M = 0.85 * public_weighted_score + 0.15 * private_weighted_score.

The reason of the preliminary model scores is to allow you to test your own model against the private dataset which is now available on Codabench. In this way, you can verify your own scoring on the model you submitted for the private evaluation before the deadline. After Thursday, May 8th, 11:59:59 PM EDT, the Model scores will be finalized and the invitations to Phase 2 will be sent out.

In the meantime, here you can find instructions and details for the finalists who are admitted to Phase 2, and a LaTex template that you can use to prepare your technical report.

A quick reminder that Phase 2 deadline is Monday, May 26th, 11:59:59 PM EDT.

Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

April 30, 2025

Dear Participants,

The Phase 1 private scores will be released on Monday, May 5th. This extra time allows us to not only fairly evaluate submissions for meaningful approaches, valuable insights, and performance improvements, but also to rigorously verify our own scoring and prevent errors.

In addition, Phase 2 deadline is extended to Monday, May 26th, 11:59:59 PM EDT.

Thanks for your participation and support!

Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

April 24, 2025

Dear Participants,

A few reminders:

• If for some reasons you would like us to score a different submission from the one in the leaderboard, please fill in this form by today Thursday April 24th at 11:59 PM EDT.
• In addition, please fill in this form ASAP to provide a brief explanation of how your model was developed. We want to ensure that the submissions are based on meaningful approaches, contribute valuable insights or performance improvements, and facilitate troubleshooting potential evaluation issues on our end.

We look forward to seeing your submissions. Thanks for your participation and support!

Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

April 23, 2025

Dear Participants,

The end of Phase 1 is today Wednesday, April 23rd at 11:59 PM EDT. After that, we are not going to accept any other submissions.

If for some reasons you would like us to score a different submission from the one in the leaderboard, please fill in this form by tomorrow Thursday April 24th at 11:59 PM EDT.

In addition, please fill in this form ASAP to provide a brief explanation of how your model was developed. We want to ensure that the submissions are based on meaningful approaches, contribute valuable insights or performance improvements, and facilitate troubleshooting potential evaluation issues on our end.

We look forward to seeing your submissions. Thanks for your participation and support!

Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

April 17, 2025

Dear Participants,

The end of Phase 1 is approaching. A few announcements:

• Since our offices are closed on Monday in observance of Patriots’ Day, we will continue to accept submissions for Phase 1 until Wednesday, April 23rd at 11:59 PM EDT, to ensure our capacity to provide submission support for issues encountered over the weekend and leading up to the deadline. This will also allow you to attend Wednesday’s office hours for any last-minute submission support.
• At the end of Phase 1, your models are evaluated on a mix of public and private dataset. We are expecting this process to take up to a week. This will determine your “Private weighted score”, and you will be notified if you are admitted to Phase 2 or not. 
• The finalists who are admitted to Phase 2 will have to prepare and submit a technical report that will be reviewed and evaluated. 
• The technical reports’ scoring scheme assesses the innovation and overall quality of the technical report, including clarity, novelty, technical depth, reproducibility, and insights. The report’s score is part of the competition’s final composite score. More details will be sent out to the finalists who are admitted to Phase 2.
• Three examples of reports from the last year Challenge can be found in Sections 3.1, 3.2, and 3.3 here.

We look forward to seeing your submissions. Thanks for your participation and support!

Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

April 14, 2025

Dear Participants,

This is a reminder that Phase 1 of the Challenge will conclude in 1 week, on April 21st. 

Keep refining, testing, and preparing your submission. This final week is an excellent opportunity to fine-tune your approach and ensure a strong submission. 
We are excited to see the results of your efforts!

Thanks for your participation and support.

Best,
Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

April 7, 2025

Dear Participants,

There are only 2 weeks left before the end of Phase 1. A few clarifications:

• Your model is used to evaluate “both” task 1.1 and task 1.2; the resulting scores are then used in the computation of the “Public weighted score”.
• At the end of Phase 1, your models are evaluated on a mix of public and private dataset (this might take a few days at most). This will determine your “Private weighted score”, and you will be notified if you are admitted to Phase 2.
• Details on the computation of the “Private weighted score” are not public yet, but you should aim to make a model as generalist as possible, since the goal is to apply it to any kind of period and to avoid overfitting, and we will especially reward those models that achieve that adaptability.
• For Phase 2, you have to prepare and submit a report that will be reviewed and evaluated. The report’s score is part of the competition’s final composite score.

We look forward to seeing your submissions. Thanks for your participation and support!

Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

April 3, 2025

Dear AI Challenge participants,

As Codabench’s leaderboard was not properly displaying the results of the challenge, we’ve created a new custom leaderboard hosted on our own webpage.

This new leaderboard updates every 15 minutes, but only if you achieve a new personal best. In that case, your score will appear within 15 to 30 minutes after your submission finishes.

Since this system hasn’t been tested with participants before, please report any issues or bugs you notice. We appreciate your help in making it run smoothly.

Thanks for your participation and support!

Best regards,
AI Challenge Organizers 

MARCH 28, 2025

Dear AI Challenge Participants,

Here are some reminders.

Complete Feedback Form
We’ve noticed that while we have a large and engaged group of participants, only a limited number of teams have submitted their models so far. We understand that there can be many reasons for this, and we want to ensure that everyone has a positive and rewarding experience.

To help us better understand your experience, please complete this short feedback form. (Note: if you prefer to remain anonymous, feel free to not submit name/team/email.)

Submit Your Models!
From now on, please submit your model to the public leaderboard for both Phase 1.1 and 1.2! The challenge is coming to a close, and we’re eager to see the innovative solutions you’ve developed. Moreover, sharing your results is an essential and valuable part of the Challenge since you can learn from the leaderboard and receive feedback.

We look forward to seeing your submissions and hearing your feedback.

Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

MARCH 26, 2025

Dear AI Challenge Participants,

We are now less than a month away from the end of Phase 1. Here are some updates.
 
Complete Feedback Form
 We’ve noticed that while we have a large and engaged group of participants, only a limited number of teams have submitted their models so far. We understand that there can be many reasons for this, and we want to ensure that everyone has a positive and rewarding experience.

To help us better understand your experience, please complete this short feedback form.
 
Submit Your Models!
From now on, please submit your model to the public leaderboard for both Phase 1.1 and 1.2! The challenge is coming to a close, and we’re eager to see the innovative solutions you’ve developed. Moreover, sharing your results is an essential and valuable part of the Challenge since you can learn from the leaderboard and receive feedback.
 
Final Score Computation
As you know, recently we introduced a new task in Phase 1.2 to further challenge your models. To clarify the weighting scheme, this new dataset will account for 80% of the final score, while the original dataset will contribute 20%.
 
Important Corrections
We’ve corrected the altitude values in the Phase 1.2 dataset, they were mistakenly provided in kilometers, unlike Phase 1.1 where they were in meters. Now, all values are consistently in meters. 
Please ignore the column name “Altitude (km)”, the values are in meters despite the label. We’ve kept the name unchanged to avoid breaking existing submissions.
 
Office Hours
Just a reminder that office hours are scheduled every Wednesday at 11:00 AM EDT (UTC−04:00). Don’t miss this opportunity to ask questions and clarify any doubts about the new task. In addition, we understand that this time may not be ideal for everyone in our international community, so please don’t hesitate to let us know if you have scheduling conflicts. We’re happy to explore alternative times, and you can always reach out with questions through the discussion forum or via email.
 
We look forward to seeing your submissions and hearing your feedback.
 
Best regards,
2025 MIT ARCLab Prize for AI Innovation in Space – Organizers

MARCH 18, 2025

Dear AI Challenge Participants,

We have an exciting update! Given the strong performance we’re seeing, we’re introducing a new task in Phase 1.2 to further challenge your models. Think of this as an extension of Phase 1, where your models will now be evaluated on a more difficult dataset. 

Final Score Computation
Your score will be calculated based on the scores from both tasks, with greater weight given to the new dataset due to its higher expected difficulty compared to the original one. Additionally, the Propagation Score (PS) has been renamed to Orbital Density RMSE (OD-RMSE) to avoid confusion caused by the previous name. However, the way it works remains unchanged. 

⚠️ IMPORTANT: Initial States File Location Change
Due to the addition of a new task, initial_states.csv
 is no longer in its previous location. You can now find it at: 📂 /app/input_data/initial_states.csv

To load it in your code, use:
import os
initial_states_file = os.path.join('/app/input_data',"initial_states.csv")

🚨 This is a critical change—make sure your code is updated! 
Also for those that you already have made some submissions, we encourage you to re-submit applying these changes to see your score for this new dataset.

Submit Your Models Early!
If you haven’t submitted yet, now is the time to start! Some participants have already encountered dependency issues, and it’s best to resolve them early rather than at the last minute. We encourage you to begin working on this task soon to ensure strong performance.

Office Hours: Time Change Reminder
Office hours are scheduled every Wednesday at 11:00 AM EDT (UTC−04:00). Be sure to check your local time against Boston time, as it may differ. For those who didn’t adjust their clocks last week, office hours are now one hour earlier due to the U.S. switch. 
Don’t miss this opportunity to ask questions and clarify any doubts about the new task.

We’re excited to see how your models perform under this new challenge. Keep up the great work!

Best regards,
Your AI Challenge Organizers

February 21, 2025

Dear AI Challenge Participants,

We have a few key updates this week regarding your submissions to the Codabench platform and the baseline solution:

1. As some participants have noted, many submissions are failing due to timeout issues. To address this, we have temporarily increased the maximum computing time to 2 hours, allowing more time for computations and model performance insights. This change is temporary, and the limit will likely revert to 1 hour once we find a solution, so please optimize your models.
2. Results are no longer automatically submitted to the leaderboard due to an unresolved Codabench bug. You must now submit your best scores manually, as required by the challenge’s terms and conditions. More details on how to do this can be found on the wiki submission page. Please ensure you complete this step.
3. The baseline propagator settings have been updated to reduce the computational time required for evaluation. To learn more about these changes, please refer to this forum discussion and the propagator tutorial on the STORM-AI wiki.

As always, feel free to reach out through the forums or during office hours (Wednesdays 11:00-11:30 AM EST) if you have any questions. Transcripts of office hours discussions will be posted to the wiki on a weekly basis moving forward. We look forward to seeing what you accomplish next!

Best regards,
Your AI Challenge Organizers

February 18, 2025

Dear AI Challenge Participants,

Thank you for your submissions, questions, and valuable feedback throughout the competition! We appreciate your engagement and the effort you’ve put into improving your models. 
To give you more time to refine your approaches and to compensate for the submission delays, we have extended Phase 1 by five weeks. This means that you will have an additional five weeks to submit your code and compete for the chance to move on to Phase 2, but the length of time allocated to writing and reviewing reports will stay the same. The updated challenge timeline is as follows:

• April 21, 2025: Phase 1 ends. Top 10 finalists are notified of advancement to Phase 2.
• May 19, 2025: Phase 2ends. Technical report deadline.
• June 20, 2025: Winners Announced.

Please continue to reach out through the forums or office hours (Wednesdays 11:00-11:30 AM EST) if you have any questions. We look forward to seeing your continued progress!

Best regards,
Your AI Challenge Organizers

February 7, 2025

Dear AI Challenge Participants,

The STORM-AI wiki now includes a complete tutorial for submitting your code to Codabench! The tutorial contains step-by-step instructions on setting up your conda environment, preparing your submission, and ensuring everything works smoothly on the new platform.

In addition, a video version of the submission tutorial is in the works and will be available soon. We hope this will make the process even clearer and easier for you to follow.

Important Reminder for Team Submissions
If you’re participating as a team, please ensure that all submissions are made under an organization. Here’s how to set one up:

1. Go to your profile (top-right corner of the Codabench interface)
   
2. Scroll to the bottom of the menu and click “Add new organization”
   
3. Once your organization is created, select it on the submission page when uploading your entry

This step is essential to properly manage submissions from multiple team members, so please ensure your team completes it before your next submission.

We appreciate your continued patience and participation as we transition to Codabench. If you encounter any issues or have questions, feel free to ask during our office hours (Wednesdays 11:00-11:30 AM EST) or post in the forums.

Thank you, and we look forward to seeing your submissions!

Best regards,
Your AI Challenge Organizers

February 6, 2025

Dear AI Challenge Participants,

We hope you’re doing well! We wanted to inform you of some important updates regarding the competition:

1. Weekly Office Hours: Starting next week, we will be holding weekly office hours on Zoom every Wednesday from 11:00 AM to 11:30 AM ET. To avoid creating any unfair advantage, we request that discussions about potential approaches to the challenge problem are kept to the forums, but office hours will be a great opportunity to ask clarifying and logistical questions (e.g. about the data or issues with code submission). We encourage you to join if you need assistance!
   
2. Platform Transition to Codabench: As of today, we are transitioning from EvalAI to Codabench for the remainder of the competition. As part of this change, you will now need to submit your code using a conda environment. This new platform will better suit our evaluation needs and will provide a smoother experience moving forward.

With this transition, you will no longer need to update Docker images. Instead, you will manage dependencies using an environment.yml file. We will use this file to install all necessary dependencies for your solution. Specifically, we will create the conda environment using micromamba (due to its high performance) and then execute submission.py, which should remain the entry point/controller of your model.

We strongly encourage you to start adapting your solutions to this new environment and testing them locally to ensure that micromamba can correctly install all dependencies. We will provide a more detailed explanation of the submission process in the coming days as we finish migrating the platform.
We understand that this transition may cause some inconvenience, especially at this stage in the competition, and we apologize for the disruption. Unfortunately, EvalAI was unable to resolve the evaluation issues we encountered, and we believe Codabench will be much more reliable for your submissions.

What’s Next?
We will be adding detailed documentation about this change and the conda environment setup to the STORM-AI wiki later this week. You can now access the challenge on Codabench here. We will send out another email when the wiki is updated with all of the new submission instructions, but in the meantime feel free to submit your code to Codabench for evaluation if you are comfortable doing so.

Additionally, we will do our best to provide strong support to help you adapt to the new requirements and platform as smoothly as possible. We encourage you to participate actively in the forums and ask any questions you may have.

Thank you for your understanding and continued participation. We look forward to seeing your submissions and hope to see you at our weekly office hours!

Best regards,
Your AI Challenge Organizers 

December 17, 2024

Dear challenge participants,

Welcome to Phase 1 of the 2025 MIT ARCLab Prize for AI Innovation in Space! You can now submit your code on a rolling basis for evaluation on the EvalAI platform. You will not be able to see your scores on the leaderboard quite yet, but we should have full functionality on EvalAI within the next few days.

The training dataset, model submission tutorial, development toolkit, and baseline model are available at the following links:

• Dataset: it currently contains initial satellite orbital states between the years 2000 and 2020 along with a 60-day OMNI2 space weather data fragment and 3 days of orbit-averaged thermospheric densities for each initial state. The corresponding 60-day GOES data fragments will be uploaded before the end of the week. We may also add more test cases to increase the size of the training dataset and will keep you posted.
  
• Development toolkit: the Satellite Tracking and Orbit Resilience Modeling with AI (STORM-AI) DevKit accessible on GitHub, reporting tutorials, baseline model, and a high-fidelity orbit propagator.
  
• Baseline model: this script represents the baseline model which appears on the current EvalAI leaderboard.
  
• Model submission tutorial: this section of the wiki discusses Docker containers and example submissions for models coded in Python as well as other helpful information about submitting your model to EvalAI for evaluation.

As a reminder, the STORM-AI Wiki Page reports extensive documentation about the challenge dataset, example code, tutorials, EvalAI submission process, supplemental resources, and FAQs.

If you have any questions about the data or development toolkit, please feel free to reach out via our devkit Q&A. For general inquiries, you can email us at ai_challenge@mit.edu.

We’ll keep you informed via email of any changes to the dataset or EvalAI page over the next few days. Good luck!

Best Regards,
Your AI Challenge Organizers

December 11, 2024

Good evening challenge participants,

We hope that you’re ready for the competition phase because the warm-up period is nearly over! Our baseline model will be available on the devkit this weekend after we make some final adjustments, and you will soon have access to our EvalAI page and the full training dataset. 

In preparation for Phase 1, we have also provided a new version of the warm-up dataset that is organized into I/O pairs. We hope this will help streamline the process of training your algorithms. The new format is consistent with our evaluation pipeline, so the training data that will be released during Phase 1 will follow this convention as well. Changes from the V1 data format include:

• Satellite geodetic position coordinates for each set of initial orbital elements have been added to the initial state file (wu001_to_wu715-initial_states.csv)
  
• GOES & OMNI2 data has been split into 60-day chunks so that each initial state has a corresponding GOES input file and OMNI2 input file with the specified file ID (“wu###” for the warmup dataset)
  
• SWARM density data has been split into 3-day chunks so that each initial state has a corresponding “forecasted” density file for model validation
  
• SWARM density data columns that do not appear in the required output format have been removed, meaning that for each initial state, your model should output a 3-day density forecast in the same format as the files in the Sat_Density folder
  
• The STORM-AI dropbox folder now has the following structure:
  
  📁 STORM-AI
├── 📁 warmup
│   ├── 📁 v1
│   └── 📁 v2
│       ├── 📁 GOES
│       ├── 📁 OMNI2
│       ├── 📁 Sat_Density
│       └── 📝 wu001_to_wu715-initial_states.csv
└── 📁 phase_1

***Please note that the reformatted inputs and outputs still contain gaps due to the nature of the source data***

If you have any questions about the data or development toolkit, please feel free to reach out via our devkit Q&A. For general inquiries, you can email us at ai_challenge@mit.edu.

Best Regards,
Your AI Challenge Organizers

November 15, 2024

Dear challenge participants,

Good evening and welcome to the warm-up phase of the 2025 MIT ARCLab Prize for AI Innovation in Space! Phase 0 launched earlier today, and there are a few key updates and resources that we want to make you aware of.

TL;DR: Competition Overview | DevKit | Dataset | Wiki | Q&A

EvalAI Development Status
As in last year’s competition, we will begin accepting and evaluating your submissions through EvalAI at the launch of Phase 1 in December. We previously announced that the new challenge would be available for you to view on the EvalAI platform during this warm-up period; however, we are still working with the EvalAI team to validate our evaluation pipeline. We’ll give you a heads up when we go live on EvalAI, but in the meantime, we have published a comprehensive challenge overview to the ARCLab website at the link above.

STORM-AI Development Kit
To help you get started on your models, we created the Satellite Tracking and Orbit Resilience Modeling with AI (STORM-AI) DevKit on GitHub. The repo currently includes a transformer-based atmospheric density forecaster based on this paper, which was written by an ARCLab alum and the original inspiration for this year’s challenge problem. In the coming weeks, we will continue to release tutorials and other components of the devkit, including the code for our baseline model that will appear on the Phase 1 leaderboard. We will notify you of new content via email and do our best to respond to any questions or feedback on the devkit Q&A.

Warm-Up Dataset
A portion of the public challenge dataset is currently available for download via Dropbox at the link above. It includes a few years of space weather data and initial satellite states that your models should parse as inputs. There is also a history of orbit-averaged atmospheric density values reported by ESA’s SWARM A satellite, which you may use to validate your model. The challenge data formats should change very little from this preliminary dataset and will be finalized well in advance of the Phase 1 launch date. More detailed information about the data is located in the devkit wiki.

STORM-AI Wiki Page
Extensive documentation about the challenge dataset, example code, EvalAI submission process, and supplemental resources is available on the devkit wiki. The latest long-form tutorials, FAQs, and more will live on the wiki throughout the course of the competition, so refer back to it often!

Thank you for being a part of this year’s AI Challenge! We hope that these resources will provide everything you need to start working with the dataset and developing your prediction models. Feel free to reach out with questions about the data or development toolkit via our devkit Q&A, or for general inquiries, email us at ai_challenge@mit.edu.

Best regards,
Your AI Challenge organizers

November 4, 2024

Hello everyone!

Thank you for your interest in the 2025 MIT ARCLab Prize for AI Innovation in Space! 

We are excited to announce that we are launching Phase 0 of our space weather challenge on Friday November 15th. This is a warm-up phase, during which you will be able to view the competition’s EvalAI page, a preliminary GitHub devkit, and a portion of the dataset you will be provided for training your algorithms. Last year’s webpage, devkit, and warm-up dataset are good references for what you can expect.

Code submissions will not be accepted until the launch of Phase 1 in mid-December, but the warm-up phase will give you the opportunity to familiarize yourself with the data types and formats you’ll be asked to work with during the challenge.

Read about the challenge objective, phases, and prize breakdown in our launch announcement or contact the organizers at ai_challenge@mit.edu for more information.

Best regards,
Your AI Challenge organizers