Leveraging Foundation Models for Earth Observation Applications

In recent years, we’ve witnessed the rise of foundation models 1 for text (e.g. chatGPT, Llama) and images (e.g. Dalle, Stable Diffusion 2), even video (e.g. Stable Video Diffusion).

Foundation models are also being developed for other fields (e.g. Medical science 3), and particularly for Earth Observation4 5 and Climate Science6. The goal of this thesis is to investigate the maturity of these models and evaluating them in real-world downstream applications, like burned area mapping7, flood detection8 9, wildfire forecasting10 11, volcanic unrest detection12 13 or similar applications with impact for social good. Beyond evaluation, this thesis is about devising ways to fine-tune these large models to work better for downstream tasks.

Supervisors: Ioannis Prapas, Ioannis Papoutsis

Relevant skills: Python, Deep Learning basics, Deep Learning library (pytorch, jax, tensorflow), Remote Sensing basics

  1. Bommasani, Rishi, et al. “On the opportunities and risks of foundation models.” arXiv preprint arXiv:2108.07258 (2021). ↩︎

  2. Rombach, Robin, et al. “High-resolution image synthesis with latent diffusion models.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. ↩︎

  3. Moor, Michael, et al. “Foundation models for generalist medical artificial intelligence.” Nature 616.7956 (2023): 259-265. ↩︎

  4. https://www.earthdata.nasa.gov/news/impact-ibm-hls-foundation-model ↩︎

  5. Liu, Fan, et al. “RemoteCLIP: A Vision Language Foundation Model for Remote Sensing.” arXiv preprint arXiv:2306.11029 (2023). ↩︎

  6. Nguyen, Tung, et al. “ClimaX: A foundation model for weather and climate.” arXiv preprint arXiv:2301.10343 (2023). ↩︎

  7. Sdraka, Maria, et al. “FLOGA: A machine learning ready dataset, a benchmark and a novel deep learning model for burnt area mapping with Sentinel-2.” arXiv preprint arXiv:2311.03339 (2023). ↩︎

  8. Bountos, Nikolaos Ioannis, et al. “Kuro Siwo: 12.1 billion $ m^ 2$ under the water. A global multi-temporal satellite dataset for rapid flood mapping.” arXiv preprint arXiv:2311.12056 (2023). ↩︎

  9. Li, Wenwen, et al. “Assessment of a new GeoAI foundation model for flood inundation mapping.” Proceedings of the 6th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. 2023. ↩︎

  10. Kondylatos, Spyros, et al. “Mesogeos: A multi-purpose dataset for data-driven wildfire modeling in the Mediterranean.” arXiv preprint arXiv:2306.05144 (2023). ↩︎

  11. Prapas, Ioannis, et al. “TeleViT: Teleconnection-driven Transformers Improve Subseasonal to Seasonal Wildfire Forecasting.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. ↩︎

  12. Bountos, Nikolaos Ioannis, et al. “Self-supervised contrastive learning for volcanic unrest detection.” IEEE Geoscience and Remote Sensing Letters 19 (2021): 1-5. ↩︎

  13. Bountos, Nikolaos Ioannis, et al. “Hephaestus: A large scale multitask dataset towards InSAR understanding.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. ↩︎

foundation models earth observation EO climate

Related