University of Cambridge > Talks.cam > Hills Coffee Talks > Optimizing Data Delivery and Scalable HI Profile Classification for the SKA Era: Infrastructure and Science Challenges at the Spanish SRC

Optimizing Data Delivery and Scalable HI Profile Classification for the SKA Era: Infrastructure and Science Challenges at the Spanish SRC

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Charles Walker.

This talk presents ongoing work at the Spanish SKA Regional Centre (esSRC) in the context of the SRC Net 0.1. The first part focuses on the development of efficient data delivery techniques from the distributed Rucio-based storage system to the SRC infrastructure and, ultimately, to user workspaces. Several approaches have been evaluated to support science-ready access, yet current solutions often involve unnecessary data duplication in user areas, resulting in increased usage of storage and computational resources. To address this, we have prototyped mechanisms based on file linking, caching, and data reuse, enabling more efficient access paths for users. While these methods show promising improvements in terms of performance and resource usage, challenges remain, particularly in terms of orchestration, scalability, and compatibility with existing workload managers. The second part presents advances in the automated classification of neutral hydrogen (HI) profiles using machine learning methods, building on previous work [Parra et al., 2024, arXiv:2501.11657]. We outline a roadmap for extending these techniques to handle the data volumes expected from the SKA Observatory. This includes developing scalable pipelines capable of ingesting and processing large spectral datasets in a reproducible and efficient manner, and adapting the classification models to cope with the diversity and complexity of the SKA data products.

This talk is part of the Hills Coffee Talks series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity