FSE 2024 CoLos
32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024)
Powered by
Conference Publishing Consulting

4th International Workshop on Software Engineering and AI for Data Quality in Cyber-Physical Systems/Internet of Things (SEA4DQ 2024), July 15, 2024, Porto de Galinhas, Brazil

SEA4DQ 2024 – Preliminary Table of Contents

Contents - Abstracts - Authors
Twitter: https://twitter.com/esecfse

4th International Workshop on Software Engineering and AI for Data Quality in Cyber-Physical Systems/Internet of Things (SEA4DQ 2024)

Frontmatter

Title Page


Message from the Chairs


Committees


Papers

Evaluating the Quality of Open Source Ansible Playbooks: An Executability Perspective
Pemsith Mendis ORCID logo, Wilson Reeves ORCID logo, Muhammad Ali Babar ORCID logo, Yue Zhang ORCID logo, and Akond Rahman ORCID logo
(Auburn University, USA; University of Adelaide, Adelaide, Australia)
Infrastructure as code (IaC) is the practice of automatically managing computing platforms, such as Internet of Things (IoT) platforms. IaC has gained popularity in recent years, yielding a plethora of software artifacts, such as Ansible playbooks that are available on social coding platforms. Despite the availability of open source software (OSS) Ansible playbooks, there is a lack of empirical research on the quality of these playbooks, which can hinder the progress of IaC-related research. To that end, we conduct an empirical study with 2,952 OSS Ansible playbooks where we evaluate the quality of OSS playbooks from the perspective of executability, i.e., if publicly available OSS Ansible playbooks can be executed without failures. From our empirical study, we observe 71.5% of the mined 2,952 Ansible playbooks cannot be executed as is because of four categories of failures.

Article Search
A Pilot Study in Surveying Data Challenges of Automatic Software Engineering Tasks
Liming Dong ORCID logo, Qinghua Lu ORCID logo, and Liming Zhu ORCID logo
(CSIRO’s Data61, Australia)
The surge in automatic SE research aims to boost development efficiency and quality while reducing costs. However, challenges such as limited real-world project data and inadequate data conditions constrain the effectiveness of these methods. To systematically understand these challenges, our pilot study reviews prevalent data challenges across various SE tasks. Despite these challenges, thanks to the advances of large language model offers promising performance on SE tasks. Overall, this pilot survey focused on provide a quick retrospective review on SE data challenges and introduce practical LLM solutions from the SE community to mitigate these challenges.

Article Search
A Hitchhiker’s Guide to Jailbreaking ChatGPT via Prompt Engineering
Yi Liu ORCID logo, Gelei Deng ORCID logo, Zhengzi Xu ORCID logo, Yuekang Li ORCID logo, Yaowen Zheng ORCID logo, Ying Zhang ORCID logo, Lida Zhao ORCID logo, Tianwei Zhang ORCID logo, and Kailong Wang ORCID logo
(Nanyang Technological University, Singapore; UNSW, Australia; Institute of Information Engineering at Chinese Academy of Sciences, China; Virginia Tech, USA; Huazhong University of Science and Technology, China)
Natural language prompts serve as an essential interface between users and Large Language Models (LLMs) like GPT-3.5 and GPT-4, which are employed by ChatGPT to produce outputs across various tasks. However, prompts crafted with malicious intent, known as jailbreak prompts, can circumvent the restrictions of LLMs, posing a significant threat to systems integrated with these models. Despite their critical importance, there is a lack of systematic analysis and comprehensive understanding of jailbreak prompts. Our paper aims to address this gap by exploring key research questions to enhance the robustness of LLM systems: 1) What common patterns are present in jailbreak prompts? 2) How effectively can these prompts bypass the restrictions of LLMs? 3) With the evolution of LLMs, how does the effectiveness of jailbreak prompts change? To address our research questions, we embarked on an empirical study targeting the LLMs underpinning ChatGPT, one of today's most advanced chatbots. Our methodology involved categorizing 78 jailbreak prompts into 10 distinct patterns, further organized into three jailbreak strategy types, and examining their distribution. We assessed the effectiveness of these prompts on GPT-3.5 and GPT-4, using a set of 3,120 questions across 8 scenarios deemed prohibited by OpenAI. Additionally, our study tracked the performance of these prompts over a 3-month period, observing the evolutionary response of ChatGPT to such inputs. Our findings offer a comprehensive view of jailbreak prompts, elucidating their taxonomy, effectiveness, and temporal dynamics. Notably, we discovered that GPT-3.5 and GPT-4 could still generate inappropriate content in response to malicious prompts without the need for jailbreaking. This underscores the critical need for effective prompt management within LLM systems and provides valuable insights and data to spur further research in LLM testing and jailbreak prevention.

Article Search

proc time: 1.82