• Automated validation and improvement of DNN models for computer vision

  • We developed approaches that combine metaheuristic search, simulators, and generative models to automatically generate diverse images as input to a DNN model performing computer vision tasks. The approach automatically generates images leading to "critical" failures. The generated images are diverse and can also be used to retrain the DNN to achieve better performance.

  • We validated our work with case studies from IEE Sensing and ESA, and we are currently leveraging it in a project with Redwire Luxembourg.

  • References:

    • Mohammed Oualid Attaoui, Fabrizio Pastore, and Lionel C. Briand. 2025. Search-based DNN Testing and Retraining with GAN-enhanced Simulations. IEEE Trans. Softw. Eng. 51, 4 (2025), 1086–1103. https://orbilu.uni.lu/handle/10993/63426 

    • Mohammed Attaoui and Fabrizio Pastore. 2025. GAN-enhanced Simulation-driven DNN Testing in Absence of Ground Truth. arXiv preprint arXiv:2503.15953 (2025). https://arxiv.org/abs/2503.15953 

  • Tool:

    • DESIGNATOR https://zenodo.org/records/17865409

• Characterisation of failure-inducing scenarios for DNNs performing computer vision tasks

  • We developed approaches that leverage clustering applied to different feature extraction methods to identify input images that characterise the different scenes in which a DNN may fail. They can be used to further select additional images from a new dataset fo DNN retraining. We validated our work with case studies from IEE Sensing and ESA.

  • References:

    • Mohammed Oualid Attaoui, Hazem Fahmy, Fabrizio Pastore, and Lionel Briand. 2024. Supporting Safety Analysis of Image-processing DNNs through Clustering-based Approaches. ACM Trans. Softw. Eng. Methodol. 33, 5 (2024), 1–48. https://orbilu.uni.lu/handle/10993/58139 

    • Hazem Fahmy, Fabrizio Pastore, Lionel Briand, and Thomas Stifter. 2023. Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-based Safety-critical Systems. ACM Trans. Softw. Eng. Methodol. 32, 4 (2023), 1–47. https://orbilu.uni.lu/handle/10993/52366 

    • Mohammed Oualid Attaoui, Hazem Fahmy, Fabrizio Pastore, and Lionel Briand. 2023. Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering. ACM Trans. Softw. Eng. Methodol. 32, 3 (2023), 1–40. https://orbilu.uni.lu/handle/10993/52372 

    • Hazem Fahmy, Fabrizio Pastore, Mojtaba Bagherzadeh, and Lionel Briand. 2021. Supporting Deep Neural Network Safety Analysis and Retraining Through Heatmap-Based Unsupervised Learning. IEEE Trans. Reliab. 70, 4 (2021), 1641–1657. https://orbilu.uni.lu/handle/10993/46838 

  • Tool:

    • ClusterXplain https://zenodo.org/records/21249809

• Mutation testing of embedded software for cyber-physical systems (space systems)

  • In safety- and mission-critical contexts, test suites must be highly effective. For this reason, ESA commissioned the FAQAS project to support the assessment of test suites and their improvement. Specifically we:

  • Developed an approach for mutation analysis of C/C++ software that leverages statistical methods to determine when to stop assessing mutants.

    • Oscar Cornejo, Fabrizio Pastore, and Lionel C. Briand. 2022. Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain. IEEE Trans. Softw. Eng. 48, 10 (2022), 3913–3939. https://orbilu.uni.lu/handle/10993/47861

    • MASS tool https://github.com/SNTSVV/FAQAS_MASS

  • Developed an approach for the assessment of integration test suites by mutating the data exchanged by components, guided by a data model.

    • Enrico Viganò, Oscar Cornejo, Fabrizio Pastore, and Lionel C. Briand. 2023. Data-driven Mutation Analysis for Cyber-Physical Systems. IEEE Trans. Softw. Eng. 49, 4 (2023), 2182–2201. https://orbilu.uni.lu/handle/10993/52368

    • DAMAT https://github.com/SNTSVV/DAMAT

  • Developed an approach for the automated generation of test cases that leverages fuzzing to identify the inputs that kill C/C++ mutants.

    • Jaekwon Lee, Fabrizio Pastore, and Lionel Briand. 2025. Fuzzing-based Mutation Testing of C/C++ Software in Cyber-Physical Systems. Empir. Softw. Eng. 31, 1 (2025), Article 20. https://orbilu.uni.lu/handle/10993/66936

    • Jaekwon Lee, Enrico Viganò, Oscar Cornejo, Fabrizio Pastore, and Lionel Briand. 2023. Fuzzing for CPS Mutation Testing. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 1377–1389. https://orbilu.uni.lu/handle/10993/55873

    • MOTIF tool  https://github.com/SNTSVV/MOTIF

  • All our approaches were validated with case study subjects provided by leaders in the space sector: GomSpace, Huld, LuxSpace

• Automated Metamorphic Security Testing

  • One of the key difficulties in performing extensive and effective cybersecurity testing is that vulnerabilities can often be discovered only with a few corner-case inputs, which means that, with a large input space,  a wide set of inputs needs to be assessed in order to identify vulnerabilities. However, verifying outputs for many inputs (e.g, specifying test assertions) is extremely expensive, thus making the testing process unaffordable. We developed a metamorphic security testing approach (MST) that relies on a catalogue of metamorphic relations that transform valid input sequences into attacks and determine the presence of vulnerabilities by comparing their outputs, without needing to specify expected outcomes for each test input, making extensive testing reliable. For web systems, we rely on web crawlers to identify valid inputs; for other software, we can rely on existing functional test suites. Currently, in the context of a project with NC3 Luxembourg, we are investigating how LLMs can support Cybersecurity testing by either generating metamorphic relations, identifying valid input sequences, or using MST as an MCP tool.

  • References:

    • Nazanin Bayati Chaleshtari, Yoann Marquer, Fabrizio Pastore, and Lionel C. Briand. 2024. AIM: Automated Input Set Minimization for Metamorphic Security Testing. IEEE Trans. Softw. Eng. 50, 12 (2024), 3403–3434. https://orbilu.uni.lu/handle/10993/62445 

    • Seung Yeob Shin, Fabrizio Pastore, Domenico Bianculli, and Alexandra Baicoianu. 2024. Towards Generating Executable Metamorphic Relations Using Large Language Models. In Quality of Information and Communications Technology (QUATIC 2024). Communications in Computer and Information Science, Springer, 126–141. https://orbilu.uni.lu/handle/10993/61576 

    • Nazanin Bayati Chaleshtari, Fabrizio Pastore, Arda Goknil, and Lionel C. Briand. 2023. Metamorphic Testing for Web System Security. IEEE Trans. Softw. Eng. 49, 6 (2023), 3542–3564. https://orbilu.uni.lu/handle/10993/54576 TSE 2023

§  Phu X. Mai, Fabrizio Pastore, Arda Goknil, and Lionel Briand. 2020. Metamorphic Security Testing for Web Systems. In Proceedings of the 13th IEEE International Conference on Software Testing, Validation and Verification (ICST ’20). IEEE, 186–197. https://hdl.handle.net/10993/41229

• • • The MST toolset https://github.com/MetamorphicSecurityTesting/MST

• Field-based security testing of software-defined networks

  • Software-defined networks (SDNs) enable the prompt reconfiguration of network services, but reconfigurations may introduce mistakes that compromise the dependability of the provided services. Security requirements are particularly hard to verify; for example, although connectivity may function properly, confidentiality may be broken by packets forwarded to the wrong destination. Since such problems depend on the actual network being reconfigured, they cannot be exhaustively tested in-house. We developed FISTS, an approach for field-based security testing of SDN configuration updates: it probes the network before and after a configuration update and relies on unsupervised machine learning to identify the network nodes that likely match across the two configurations and to prioritise the inspection of suspicious node responses. The development of our solution was driven by an empirical study of 147 vulnerabilities affecting four popular Edge frameworks. We assessed our solution in the context of a project with SES.

  • References:

    • Jahanzaib Malik and Fabrizio Pastore. 2025. Field-based Security Testing of SDN Configuration Updates. IEEE Trans. Reliab. 74, 3 (2025), 3469–3483. https://orbilu.uni.lu/handle/10993/63497

    • Jahanzaib Malik and Fabrizio Pastore. 2023. An Empirical Study of Vulnerabilities in Edge Frameworks to Support Security Testing Improvement. Empir. Softw. Eng. 28, 4 (2023), Article 99. https://orbilu.uni.lu/handle/10993/57286

    • FISTS toolset https://doi.org/10.5281/zenodo.18201382

• Automated testing of Android apps

  • App updates are frequent, but automated testing techniques verify entire Apps, thus wasting resources. Supported by a Huawei grant, we developed model-based approaches that focus on the code affected by updates: ATUA synthesizes App models with static analysis and combines complementary testing strategies to generate a small set of inputs exercising the updated code; CALM adapts the App models learned when testing previous App versions, minimising the number of App screens that testers need to inspect visually. Both outperform state-of-the-art approaches.

  • References:

    • Chanh Duc Ngo, Fabrizio Pastore, and Lionel Briand. 2024. Testing Updated Apps by Adapting Learned Models. ACM Trans. Softw. Eng. Methodol. 33, 6 (2024), 1–40. https://orbilu.uni.lu/handle/10993/62617

    • Chanh Duc Ngo, Fabrizio Pastore, and Lionel Briand. 2022. Automated, Cost-effective, and Update-driven App Testing. ACM Trans. Softw. Eng. Methodol. 31, 4 (2022), 1–51. https://orbilu.uni.lu/handle/10993/48809

    • ATUA tool https://github.com/SNTSVV/ATUA

    • CALM tool https://github.com/SNTSVV/CALM

• Automated generation of system-level test cases from use case and misuse case specifications, including product lines

  • Requirements are often specified in natural language (e.g., as use case specifications), and the manual derivation of system-level test cases from them is expensive and error-prone, especially in safety-critical contexts where software standards mandate that each requirement be validated by test cases in a traceable manner. We developed UMTG, an approach that relies on Natural Language Processing (NLP) to automatically identify test scenarios from use case specifications; we extended it to work with product lines, and adapted it to support security testing leveraging misuse cases

  • We validated all approaches with industrial case studies, including an industrial product family provided by IEE Sensing.

  • References:

    • Chunhui Wang, Fabrizio Pastore, Arda Goknil, and Lionel C. Briand. 2022. Automatic Generation of Acceptance Test Cases from Use Case Specifications: An NLP-based Approach. IEEE Trans. Softw. Eng. 48, 2 (2022), 585–616. https://orbilu.uni.lu/handle/10993/43326

    • Ines Hajri, Arda Goknil, Fabrizio Pastore, and Lionel C. Briand. 2020. Automating System Test Case Classification and Prioritization for Use Case-Driven Testing in Product Lines. Empir. Softw. Eng. 25, 5 (2020), 3711–3769. https://orbilu.uni.lu/handle/10993/43531

    • Phu X. Mai, Fabrizio Pastore, Arda Goknil, and Lionel C. Briand. 2019. MCP: A Security Testing Tool Driven by Requirements. In Proceedings of the 41st IEEE/ACM International Conference on Software Engineering: Companion Proceedings (ICSE-Companion ’19). IEEE, 55–58. https://orbilu.uni.lu/handle/10993/39603