Generating Collaborative Problem Solving Environments for Diverse Communication Tasks (2024)

1 Introduction

Modern life requires teamwork to solve problems(Marks etal., 2001), but what makes a team work well together? This area of study, known as collaborative problem solving (CPS), is active across many disciplines, e.g., psychologists study the construction of team mental models in team discussionsLee (2015), business management sciences investigate how communication style affects performance evaluation(Proell etal., 2022), and educators develop tools to teach team communication strategies(Stewart etal., 2023), emphasizing the research direction of discovering how team members talk to one another.Conducting empirical work in CPS faces many challenges, in large part because of a large CPS task design space (e.g., what is the problem, who makes up the team, and who knows what information when).As a result, despite extensive interdisciplinary work in CPS, task designs in empirical studies have often focused on teams of two collaborating to solve problems such as selecting a designated object, modeling search and rescue, and making decisions.

AI agents have the potential to increase team effectiveness, and developing ways to integrate AI into teams is an active area of research in communities such as HCI(Cai etal., 2019), NLPBansal etal. (2019); Vats etal. (2024), and AI fairness(Lai etal., 2021).Example integrations include AI-assisted decision making with one human and one AI (e.g., cancer diagnosis, Chen etal., 2021) and AI-assisted creative tooling (e.g.,Tsiros and Palladini, 2020; Lu etal., 2024a).Developing these collaborative tools is made possible through open datasets. For example, various Amazon reviews datasets (e.g., Fornaciari and Poesio, 2014 and Ni etal., 2019) have been used to develop sentiment classifiers and deception detectors that can be used as AI-assisted decision makers,and the Reddit WritingPrompts datasetFan etal. (2018) has been valuable in developing co-writing AI systems.Unfortunately, a paucity of open datasets with more than two parties leads to challenges in integrating AI with larger human teams, as we lack understanding of team dynamics when an AI communicates to a team, rather than an individual.

To support CPS study across different designs (e.g., adding a third AI teammate to a two-human team or using voice instead of text communication), we introduce a CPS task environment generator, CPS-TaskForge.CPS-TaskForge instantiates a resource management activity through a tower defense game and supports adjusting a range of CPS design parameters such as team composition, communication method, and how stressful the task is.In a tower defense game, players must defend their base by using limited resources to construct towers that can defeat enemies before the enemies destroy the base.We provide a CPS task design checklist, CPS-✓, adapted from the PISA 2015 theoretical CPS framework (PISA2015) developed by the OECD(OECD, 2017), to support generating the desired task environment with CPS-TaskForge.

We illustrate CPS-TaskForge capabilities by presenting several CPS task designs and conducting a case study that can collect human communication data exhibiting a range of CPS skills, including social skills such as maintaining group communication and cognitive skills such as developing strategic plans. Our study has small groups of 3–4 participants complete a task multiple times with increasing difficulty.We observe many different successful strategies and a wide range in CPS skill usage across teams, demonstrating the versatility of collecting data through CPS-TaskForge.

To summarize our contributions:

2 Collaboration and Problem Solving

Collaborative problem solving (CPS) processes are well-studied for human teams, but when human-AI teams are considered, downstream task performance has been prioritized, leaving human-AI CPS processes understudied.For example, Proell etal. (2022) found human team communication more effective when the appropriate style was used in conjunction with the delivery of relevant information.Humans have different expectations towards AI teammates (Zhang etal., 2023, 2021; Grimes etal., 2021), so human-AI teams may value communication style differently.Studying human-AI CPS processes requires developing the appropriate datasets, but resources for creating such data is deficient.

Understanding how effective and efficient communication can predict successful teamwork requires collecting data in a variety of CPS settings. The tasks used to elicit relevant data often model real-world activities, e.g., rescuing humans from a burning building (ASIST; Corral etal., 2021; Freeman etal., 2021), instruction following through selecting designated objects (e.g., PentoRef,Zarrieß etal., 2016; KTH Tangrams,Shore etal., 2018; PhotoBook,Takmaz etal., 2020; Doll Dialogue, Tenbrink etal., 2017; Paxton etal., 2021), and navigating environments (e.g., HCRC Map Task, Anderson etal., 1991; Effenberger etal., 2021), and use human participants.The resulting datasets have been used to study a wide variety of communication and linguistic phenomena, including language entrainment (i.e., when communicative behavior becomes similar among interlocutors, including lexical choice and rhythm) and common ground building (i.e., when interlocutors develop their own code).To the best of our knowledge, analogous settings incorporating an AI team member in a CPS task have not explored similar communication and linguistic phenomena because only recently has AI-generated natural language become indistinguishable from humans(Clark etal., 2021; Dugan etal., 2022), enabling exploration of AI teammates as peers.Unfortunately, expanding pre-existing datasets to other CPS settings, such as involving an AI agent or a third human team member, is challenging because the tasks were designed to study a specific team composition; for example, what role would a third participant play in a navigation task originally designed for one human to tell another human where to go?

Despite the extensive body of literature studying CPS, publicly available resources remain scarce, particularly when more than two agents are involved.We summarize a sample of CPS task activities in the literature in Table1 to illustrate gaps in task type and team size between studies with or without data release to the research community.

3 CPS-TaskForge and Tower Defense

To advance CPS research, we need ways to systematically study CPS when varying factors, allowing comparison of CPS results across settings.We therefore develop a CPS task environment generator, CPS-TaskForge, which can generate CPS environments with different design factors.We also release a CPS task design checklist, CPS-✓, that describes how varying design factors produces different environments. We defer discussion of CPS-✓to Section 4; here we give a concrete description of the task environments our work targets.

We start with several requirements:(R1) CPS-TaskForge should be built on an activity that can support the different values in CPS-✓;(R2) the activity should be fun, to motivate participant signups, because CPS studies require multiple participants, making schedulinga logistical barrier to conducting CPS research;(R3) the activity should be easy to learn for both participants and researchers, in order to minimize time spent in tutorials and allow researchers to quickly design different CPS studies;and (R4) the activity should easily scale in difficulty to enable CPS research studying effects of expertise on collaboration.

We meet our design requirements by using the Tower Defense (TD) game genre as our CPS-TaskForge activity. The premise of a TD game is to defend a base from enemies by placing towers on the map, which can destroy the enemies. TD games require strategy and resource management—a vital aspect of CPS tasksCare etal. (2015)—and games have been successfully used by the research community to study communication (e.g., Codenames;Shaikh etal., 2023) and collect data (e.g., Verbosity; von Ahn etal., 2006, Duolingo(von Ahn, 2013), SearchWar (Law etal., 2009), and MatchIn(Hacker and von Ahn, 2009).

TD games are known for having a gentle learning curve, short levels (R3), and ease in scaling difficulty through simple designs (R1, R4;Avery etal., 2011). The 2021 mobile market value for TD games was estimated at 940 million USDAnalytica (2022); this popularity suggests the potential for participants to play the game of their own volition (R2).It is also known to support 1–4 players in cooperative play,¹¹1Bloons TD 6™ is a commercial game with a 4-player cooperative mode. natively supporting studying human-AI teams involving as few as one human.

We briefly describe what a TD game involves, referencing an in-game screenshot (Figure1) of an environment produced by CPS-TaskForge.In a TD game, the player needs to defend their base (7) from enemies by placing towers on the map whose inhabitants can attack the oncoming enemies. The enemies will appear at designated spawn points (1) and traverse the map along specific paths known to the player, allowing the player to strategize where to place towers effectively.Players must manage their resources (3) (e.g., gold and map real estate) when developing their defense strategy.Levels differ in the enemy spawning behavior (e.g., enemies can spawn without a break, or there is time in between groups of enemies), enemy variants (e.g., a faster or slower enemy), map terrain (e.g., obstacles can prevent tower placement), and player resources (e.g., types of towers, amount of starting gold).The standard TD game has two phases: planning, a static phase where players can place towers on the map, and attack, a dynamic phase during which enemies spawn, and players can react to the changing situation by adjusting their towers.

CPS-TaskForge is built on the open-source Godot²²2https://godotengine.org game engine, and further details of implementation and the tower defense games it produces are deferred to the documentation of our open-source release, as they are not essential to understanding the research contributions.

4 CPS-✓: A CPS Task Design Checklist

The PISA 2015 CPS Framework (PISA2015) (OECD, 2017) describes CPS tasks through a set of 15 design factors, showing how different CPS settings can be studied by manipulating different combinations of factors (e.g., team size and composition).To operationalize CPS research goals as design parameters that CPS-TaskForge can use to generate the environment, we define CPS-✓, a design checklist adapted from PISA2015 (Table2). We provide default values for CPS-✓ items in the event that some items are unnecessary to adjust for a particular study.We next explore how different hypothetical research goals can be targeted with different TD games generated by CPS-TaskForge and designed with the help of completing CPS-✓.

5 Case Study: Communication ofSmall Groups as Task Difficulty Increases

To validate its flexibility, we want to explore whether CPS-TaskForge is capable of producing an environment that elicits diverse collaborative problem solving behavior.Prior work in CPS primarily used tasks with dyads or task reptitions at the same difficulty level, so we design a CPS task where teams of 3–4 people complete a task, aiming to minimize expenditure of gold, at multiple difficulty levels.

We design our CPS-TaskForge environment as follows.Task success is evaluated by the amount of money left unused, enemies destroyed, and health of the base (CPS-✓, Q1).A single level takes 5–8 minutes to complete, depending on level difficulty, and we design 3 levels with increasing difficulty (Appendix3(a); Q2–3).All players are human (Q4), and each player is given 2–4 unique towers from a pool of 12 towers with different properties (AppendixA) so that players have different roles, encouraging all players to engage and suggest usage of their own towers (Q5–6).Players are provided a surplus of gold, and costs are balanced to slightly favor upgrading over placing more towers, giving teams the opportunity to find many successful strategies (Q7).All new information is distributed to players simultaneously (e.g., how much damage an enemy receives from a tower) (Q8).Players are under moderate time stress because each level is calibrated to give ample time to discuss strategy and place towers, and we disabled interaction during the attack phase (Q9). We designated level-specific planning time to ensure the study is completed in a reasonable amount of time.Players can only communicate through text chat (Q10).These design decisions showcase the simplicity with which the TD genre affords the ability to create different CPS task environments.

5.1 Data Collection

12 teams of 3–4 people (total 42 individuals) were recruited to participate in a 1.5-hour study³³3Our local IRB approved our study. and compensated with a gift card at a rate of 20 USD/hour.The study was conducted both in-person and remotely, and all studies were moderated. Recruitment occurred through school email listings and paper flyers posted around town. Participants were aged 18–24 (72%), 25-31 (18%), and 32+ (10%); 55% of participants were current undergraduates and 36% were in a graduate degree program; a third of participants rated their tower defense game familiarity below 3 on a 5-point Likert scale.Familiarity between teammates was not controlled, allowing some team compositions to contain strangers and others a subset of friends.

The study began with individual pre-surveys collecting basic demographic information, then participants watched a tutorial video explaining how to play the game and played a simple tutorial level together to become familiar with the interface. After the tutorial, they were given time to ask any questions about how to play the game. They then played 3 different levels 3 times each for a total of 9 games. Levels increased in difficulty, but the three rounds were the same for each level. Finally, they completed individual post-surveys containing questions about teamwork quality, team role identity, and team communication.

Dimension	CPS Skill	Example	Count	Avg. Tokens
Social	Maintaining communication	“haha okay”	222	2.3
	Sharing information	“I have a tower damange all enemies”	114	7.0
	Establishing shared understanding	“what does the diamond tower do?”	67	5.4
	Negotiating	“do we want to risk getting rid of anything else?”	38	5.4
Cognitive	Representing and formulating	“fires in multiple directions”	105	9.3
	Planning	“ok we can chokepoint the corners”	227	7.2
	Executing Actions	“k i maxed [upgrades]”	42	5.9
	Monitoring	“50 seconds D:”	86	5.0

The first 4 teams were used to calibrate game difficulty and level designs,and the final dataset contains 7 teams producing 4k utterances with a vocabulary size of 1.2k (Appendix Table4).The average win rate in the final round for levels 1–3 was 100%, 71%, and 71%.

6 Related Work

Prior work in CPS has studied a range of factors to understand effective teams, from identifying how the effects of team member personalities to how teamwork processes can be evaluated. When an AI teammate is involved, an important research direction investigates how and why humans choose to rely on AI. Findings from CPS human team processes can lead to improvements in AI agents and discovering how to better integrate AI into human teams to solve more complex problems.

Researchers have investigated how team composition affects human team outcomes (e.g., Ruch etal., 2018; Mathieu etal., 2014; Bell etal., 2018; Hollenbeck etal., 2004, inter alia), discovering predictors of team outcomes through team roles, individual expertise, demographics, and team knowledge.Lykourentzou etal. (2016) found five-person teams with balanced personalities outperformed those with an imbalance in personalities on collaborative tasks.Analogously, Wang etal. (2023) and Fan etal. (2024) were able to improve LM performance on downstream tasks by instructing the LM to simulate teams of domain-specific personas to collaborate internally.Priming an LM agent with a persona enables the simulation of inherited knowledge and linguistic patternsMasumura etal. (2018); Wei etal. (2023); Park etal. (2023), and searching for optimal personas in human-AI teams could lead to improvements in human-AI team performance.

CPS tasks can be evaluated for overall task success, but improving teamwork requires evaluating intermediate processes.Pavez etal. (2022) analyzed over a hundred studies on team performance measurement to propose a framework for evaluating teamwork along 4 dimensions: project team processes, project team emergent states, project team tangible outcomes, and project team perceptual benefits.Educators have classified CPS communication for CPS skill usage to provide feedback to students on how to improve their group communication(Andrews etal., 2019; Graesser etal., 2018; Flor etal., 2016; Stewart etal., 2023).Despite extensive work in evaluating CPS teams, there is little data released to the research community.

The increased deployment of AI in high-stakes collaborative environments, e.g., the medical domain, has opened questions about how an AI collaborator affects human behavior, leading to work in trust and reliability of AI. Humans are known to overrely on AI, following AI suggestions even when they are wrong(Lai and Tan, 2019; Jacobs etal., 2021; Bussone etal., 2015). Gazit etal. (2023), Mesbah etal. (2021), and Lu etal. (2024b) designed studies to understand human (over)reliance on AI using “judge-advisor system” tasks where a human or AI advisor provides advice to a human judge, and the judge is responsible for making the final decision.However, decisions in these tasks are independent, and the judges are not able to explain their reasoning to the advisor in a bid to adjust the advisor’s position, preventing the study of longer-term effects of human-AI interactions and human-AI communication.

7 Conclusion

Human-AI collaborative problem solving tools are rapidly being integrated in real-world work environments. The modern workforce uses teams with more than two parties, but empirical research with larger teams lags behind. The task design space for conducting CPS research is large, and the tooling to systematically explore CPS designs is lacking. Our CPS task environment generator, CPS-TaskForge, enables diverse, systematic CPS research through a tower defense game environment that appeals to human subjects and is grounded in theory.

We will release all code for CPS-TaskForge and communication data collected in our case study to encourage studying multi-human and multi-AI collaborative problem solving.

8 Limitations

The tower defense task in CPS-TaskForge environments has a learning curve (albeit a gentle one), so tutorials and practice before the actual study commences may be longer than simpler tasks such as a reference game. This complexity is necessary to support a broad range of complex tasks. CPS-TaskForge environments currently only support a top-down perspective of the world, so supporting first-person settings (e.g., simulating a Minecraft search and rescue task) is infeasible. We believe these design limitations can encourage the development of other similarly specialized CPS environment generators.

Our initial release of CPS-TaskForge implements many common attributes of tower defense games. There are many more attributes available for implemention that have been successfully deployed in commercial tower defense games that may be beneficial for future CPS studies, such as increasing the task difficulty by giving enemies resistance to certain towers. We hope to see CPS-TaskForge evolve in its feature set through usage.

Although CPS-TaskForge was developed in English, and our case study used English, usage of CPS-TaskForge does not require English. CPS-TaskForge was built in the open-source game engine Godot which natively supports other languages and localization.

CPS-✓is adapted from PISA2015, but the CPS researcher may find other CPS frameworks (e.g., ATSC21 Hesse etal. (2015), and the generalized competancy model by Sun etal., 2020) more appropriate as a checklist. We expect adapting other frameworks into a checklist that can be used to generate CPS-TaskForge environments should not be a major challenge, as other frameworks are describing CPS tasks using different attributes, and the TD game used in CPS-TaskForge is fundamentally a CPS task.

9 Ethical Considerations

The flexibility in designing CPS task environments through CPS-TaskForge necessarily places a large responsibility on the designer to design studies appropriate for their target audience or research goal. For example, the imagery used in-game for enemies and towers could be offensive to certain audiences and should be adapted as needed. As with any study in communication, appropriate content filter measures should be in place as required.

The development of generative AI agents as peers that can communicate with humans comes with the risks of the AI agents generating inappropriate content and the concerns of AI replacing humans. Our intentions are that the AI agents can augment human capabilities in more complex problem solving situations, boosting CPS abilities; however, we acknowledge that some problem solving tasks can be simulated and solved through internal or multi-agent collaboration.

Our study was approved by our institution’s IRB, and participants were fairly compensated and consented to data sharing with the research community.

References

• Analytica (2022)Astute Analytica. 2022.Mobilte tower defense games market - industry dynamics, market size, and opportunity forecast to 2030.https://www.astuteanalytica.com/industry-report/mobile-tower-defense-games-market.Accessed: 2 June 2024.
• Anderson etal. (1991)AnneH Anderson, Miles Bader, EllenGurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, etal. 1991.The hcrc map task corpus.Language and speech, 34(4):351–366.
• Andrews etal. (2019)JessicaJ. Andrews, Tanner Jackson, and Christopher Kurzum. 2019.Collaborative problem solving assessment in an online mathematics task.ETS Research Report Series, pages 1–7.
• Avery etal. (2011)Phillipa Avery, Julian Togelius, Elvis Alistar, and RobertPieter VanLeeuwen. 2011.Computational intelligence and tower defence games.In 2011 IEEE Congress of Evolutionary Computation (CEC), pages 1084–1091. IEEE.
• Bansal etal. (2019)Gagan Bansal, Besmira Nushi, Ece Kamar, WalterS. Lasecki, DanielS. Weld, and Eric Horvitz. 2019.Beyond accuracy: The role of mental models in human-ai team performance.In AAAI Conference on Human Computation & Crowdsourcing.
• Bell etal. (2018)SuzanneT Bell, ShaniqueG Brown, Anthony Colaneri, and Neal Outland. 2018.Team composition and the abcs of teamwork.American psychologist, 73(4):349.
• Bussone etal. (2015)Adrian Bussone, Simone Stumpf, and Dympna O’Sullivan. 2015.The role of explanations on trust and reliance in clinical decision support systems.In 2015 International Conference on Healthcare Informatics, pages 160–169.
• Butchibabu etal. (2016)Abhizna Butchibabu, Christopher Sparano-Huiban, Liz Sonenberg, and Julie Shah. 2016.Implicit coordination strategies for effective team communication.Human Factors, 58(4):595–610.PMID: 27113991.
• Cai etal. (2019)CarrieJ. Cai, Samantha Winter, DavidF. Steiner, Lauren Wilcox, and Michael Terry. 2019."hello ai": Uncovering the onboarding needs of medical practitioners for human-ai collaborative decision-making.Proceedings of the ACM on Human-Computer Interaction, 3:1 – 24.
• Care etal. (2015)Esther Care, Patrick Griffin, Claire Scoular, Nafisa Awwal, and Nathan Zoanetti. 2015.Collaborative problem solving tasks.Assessment and teaching of 21st century skills: Methods and approach, pages 85–104.
• Chen etal. (2021)Zi-Hang Chen, LiLin, Chen-Fei Wu, Chao-Feng Li, Rui-Hua Xu, and Ying Sun. 2021.Artificial intelligence for assisting cancer diagnosis and treatment in the era of precision medicine.Cancer Communications, 41(11):1100–1115.
• Clark etal. (2021)Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and NoahA. Smith. 2021.All that’s ‘human’ is not gold: Evaluating human evaluation of generated text.In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282–7296, Online. Association for Computational Linguistics.
• Corral etal. (2021)ChristopherC Corral, KeerthiShrikar Tatapudi, Verica Buchanan, Lixiao Huang, and NancyJ Cooke. 2021.Building a synthetic task environment to support artificial social intelligence research.In Proceedings of the human factors and ergonomics society annual meeting, volume65, pages 660–664. SAGE Publications Sage CA: Los Angeles, CA.
• Dugan etal. (2022)Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2022.Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text.In AAAI Conference on Artificial Intelligence.
• Effenberger etal. (2021)Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi. 2021.Analysis of language change in collaborative instruction following.In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2803–2811, Punta Cana, Dominican Republic. Association for Computational Linguistics.
• Fan etal. (2018)Angela Fan, Mike Lewis, and Yann Dauphin. 2018.Hierarchical neural story generation.In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
• Fan etal. (2024)Zhihao Fan, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, Jun Xi, Fei Huang, and Jingren Zhou. 2024.Ai hospital: Interactive evaluation and collaboration of llms as intern doctors for clinical diagnosis.arXiv preprint arXiv:2402.09742.
• Flor etal. (2016)Michael Flor, Su-Youn Yoon, Jiangang Hao, Lei Liu, and Alina von Davier. 2016.Automated classification of collaborative problem solving interactions in simulated science tasks.In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 31–41, San Diego, CA. Association for Computational Linguistics.
• Fornaciari and Poesio (2014)Tommaso Fornaciari and Massimo Poesio. 2014.Identifying fake Amazon reviews as learning from crowds.In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 279–287, Gothenburg, Sweden. Association for Computational Linguistics.
• Freeman etal. (2021)Jared Freeman, Lixiao Huang, Matt Wood, and StephenJ Cauffman. 2021.Evaluating artificial social intelligence in an urban search and rescue task environment.In Aaai fall symposium, pages 72–84. Springer.
• Gazit etal. (2023)Lior Gazit, Ofer Arazy, and Uri Hertz. 2023.Choosing between human and algorithmic advisors: The role of responsibility sharing.Computers in Human Behavior: Artificial Humans, 1(2):100009.
• Graesser etal. (2018)ArthurC. Graesser, StephenM. Fiore, Samuel Greiff, Jessica Andrews-Todd, PeterW. Foltz, and FriedrichW. Hesse. 2018.Advancing the science of collaborative problem solving.Psychological Science in the Public Interest, 19(2):59–92.PMID: 30497346.
• Grimes etal. (2021)G.Mark Grimes, RyanM. Schuetzler, and JustinScott Giboney. 2021.Mental models and expectation violations in conversational ai interactions.Decision Support Systems, 144:113515.
• Hacker and von Ahn (2009)Severin Hacker and Luis von Ahn. 2009.Matchin: eliciting user preferences with an online game.In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09, page 1207–1216, New York, NY, USA. Association for Computing Machinery.
• Hesse etal. (2015)Friedrich Hesse, Esther Care, Juergen Buder, Kai Sassenberg, and Patrick Griffin. 2015.A framework for teachable collaborative problem solving skills.Assessment and teaching of 21st century skills: Methods and approach, pages 37–56.
• Hill etal. (2003)RandallW. Hill, J.Gratch, Stacy Marsella, Jeff Rickel, W.Swartout, and DavidR. Traum. 2003.Virtual humans in the mission rehearsal exercise system.Künstliche Intell., 17:5–.
• Hoegl and Gemuenden (2001)Martin Hoegl and HansGeorg Gemuenden. 2001.Teamwork quality and the success of innovative projects: A theoretical concept and empirical evidence.Organization science, 12(4):435–449.
• Hollenbeck etal. (2004)JohnR Hollenbeck, DScott DeRue, and Rick Guzzo. 2004.Bridging the gap between i/o research and hr practice: Improving team composition, team training, and team task design.Human Resource Management: Published in Cooperation with the School of Business Administration, The University of Michigan and in alliance with the Society of Human Resources Management, 43(4):353–366.
• Huang etal. (2022)Lixiao Huang, Jared Freeman, Nancy Cooke, Samantha Dubrow, John“JCR” Colonna-Romano, Matt Wood, Verica Buchanan, Stephen Caufman, and Xiaoyun Yin. 2022.Artificial Social Intelligence for Successful Teams (ASIST) Study 2.
• Hutchins etal. (2008)SusanG Hutchins, Anthony Kendall, and Alex Bordetsky. 2008.Understanding patterns of team collaboration employed to solve unique problems.In Proceedings of the 13 th International Command and Control Research & Technology Symposium, pages 17–19.
• Jacobs etal. (2021)M.Jacobs, M.Pradier, T.McCoy, P.Roy, F.Doshi-Velez, and G.Krzysztof. 2021.How machine learning recommendations influence clinician treatment selections: example of antidepressant selection.Translational Psychiatry, 1:1–9.
• Kokel etal. (2022)Harsha Kokel, M.Das, Rakibul Islam, Julia Bonn, JonZ. Cai, Soham Dan, Anjali Narayan-Chen, Prashant Jayannavar, JanardhanRao Doppa, J.Hockenmaier, Sriraam Natarajan, Martha Palmer, and Dan Roth. 2022.Human-guided collaborative problem solving: A natural language based framework.ArXiv, abs/2207.09566.
• Kontogiannis and Kossiavelou (1999)Tom Kontogiannis and Zoe Kossiavelou. 1999.Stress and team performance: principles and challenges for intelligent decision aids.Safety science, 33(3):103–128.
• Lai etal. (2021)Vivian Lai, Chacha Chen, QingziVera Liao, Alison Smith-Renner, and Chenhao Tan. 2021.Towards a science of human-ai decision making: A survey of empirical studies.ArXiv, abs/2112.11471.
• Lai and Tan (2019)Vivian Lai and Chenhao Tan. 2019.On human predictions with explanations and predictions of machine learning models: A case study on deception detection.In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 29–38, New York, NY, USA. Association for Computing Machinery.
• Law etal. (2009)Edith Law, Luis von Ahn, and Tom Mitchell. 2009.Search war: a game for improving web search.In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’09, page31, New York, NY, USA. Association for Computing Machinery.
• Lee (2015)Jiwon Lee. 2015.Analysis of the refinement of shared mental model in science-gifted students’ collaborative problem solving process.Journal of the Korean Association for Research in Science Education, 35:1049–1062.
• Lu etal. (2024a)Bo-Ru Lu, Nikita Haduong, Chia-Hsuan Lee, Zeqiu Wu, Hao Cheng, Paul Koester, Jean Utke, Tao Yu, NoahA. Smith, and Mari Ostendorf. 2024a.Does collaborative human-lm dialogue generation help information extraction from human dialogues?Preprint, arXiv:2307.07047.
• Lu etal. (2024b)Zhuoran Lu, Dakuo Wang, and Ming Yin. 2024b.Does more advice help? the effects of second opinions in ai-assisted decision making.Proc. ACM Hum.-Comput. Interact., 8(CSCW1).
• Lykourentzou etal. (2016)Ioanna Lykourentzou, Angeliki Antoniou, Yannick Naudet, and StevenP. Dow. 2016.Personality matters: Balancing for personality types leads to better outcomes for crowd teams.In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW ’16, page 260–273, New York, NY, USA. Association for Computing Machinery.
• Ma etal. (2023)Yingbo Ma, Gloria AshiyaKatuka, Mehmet Celepkolu, and Kristy ElizabethBoyer. 2023.Automatically Predicting Peer Satisfaction During Collaborative Learning with Linguistic, Acoustic, and Visual Features.Journal of Educational Data Mining, 15(2).
• Marks etal. (2001)MichelleA. Marks, JohnE. Mathieu, and StephenJ. Zaccaro. 2001.A temporally based framework and taxonomy of team processes.The Academy of Management Review, 26(3):356–376.
• Masumura etal. (2018)Ryo Masumura, Tomohiro Tanaka, Atsushi Ando, Hirokazu Masataki, and Yushi Aono. 2018.Role play dialogue aware language models based on conditional hierarchical recurrent encoder-decoder.In Interspeech.
• Mathieu etal. (2014)JohnE Mathieu, ScottI Tannenbaum, JamieS Donsbach, and GeorgeM Alliger. 2014.A review and integration of team composition models: Moving toward a dynamic and temporal framework.Journal of management, 40(1):130–160.
• Mesbah etal. (2021)Neda Mesbah, Christoph Tauchert, and Peter Buxmann. 2021.Whose advice counts more - man or machine? an experimental investigation of ai-based advice utilization.In Hawaii International Conference on System Sciences.
• Ni etal. (2019)Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.Justifying recommendations using distantly-labeled reviews and fine-grained aspects.In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 188–197, Hong Kong, China. Association for Computational Linguistics.
• OECD (2017)OECD. 2017.PISA 2015 collaborative problem-solving framework.OECD.
• Orasanu etal. (2004)Judith Orasanu, Ute Fischer, Yuri Tada, and Norbert Kraft. 2004.Team stress and performance: Implications for long-duration space missions.In Proceedings of the human factors and ergonomics society annual meeting, volume48, pages 552–556. SAGE Publications Sage CA: Los Angeles, CA.
• Park etal. (2023)JoonSung Park, JosephC. O’Brien, CarrieJ. Cai, MeredithRingel Morris, Percy Liang, and MichaelS. Bernstein. 2023.Generative agents: Interactive simulacra of human behavior.In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), UIST ’23, New York, NY, USA. Association for Computing Machinery.
• Pavez etal. (2022)Ignacio Pavez, Hugo Gómez, Canlong Liu, and VicenteA. González. 2022.Measuring project team performance: A review and conceptualization.International Journal of Project Management, 40(8):951–971.
• Paxton etal. (2021)Alexandra Paxton, JenniferM. Roche, Alyssa Ibarra, and MichaelK. Tanenhaus. 2021.Predictions of miscommunication in verbal communication during collaborative joint action.Journal of Speech, Language, and Hearing Research, 64(2):613–627.
• Pfaff (2012)MarkS Pfaff. 2012.Negative affect reduces team awareness: The effects of mood and stress on computer-mediated team communication.Human Factors, 54(4):560–571.
• Potts (2012)Christopher Potts. 2012.Goal-driven answers in the Cards dialogue corpus.In Proceedings of the 30th West Coast Conference on Formal Linguistics, Somerville, MA. Cascadilla Press.
• Proell etal. (2022)ChadA. Proell, Yuepin(Daniel) Zhou, and MarkW. Nelson. 2022.It’s Not Only What You Say … How Communication Style and Team Culture Affect Audit Issue Follow-Up and Auditor Performance Evaluations.The Accounting Review, 97(2):373–395.
• Rockenbach etal. (2007)Bettina Rockenbach, Abdolkarim Sadrieh, and Barbara Mathauschek. 2007.Teams take the better risks.Journal of Economic Behavior & Organization, 63(3):412–422.
• Rodrigues etal. (2021)MichelleA. Rodrigues, SiOn Yoon, Kathryn B.H. Clancy, and Elizabeth A.L. Stine-Morrow. 2021.What are friends for? the impact of friendship on communicative efficiency and cortisol response during collaborative problem solving among younger and older women.Journal of Women & Aging, 33(4):411–427.PMID: 34038325.
• Ruch etal. (2018)Willibald Ruch, Fabian Gander, Tracey Platt, and Jennifer Hofmann. 2018.Team roles: Their relationships to character strengths and job satisfaction.The Journal of Positive Psychology, 13(2):190–199.
• Savelsbergh etal. (2012)Chantal Savelsbergh, JosetteMP Gevers, BeatriceIJM Vander Heijden, and RobF Poell. 2012.Team role stress: Relationships with team learning and performance in project teams.Group & organization management, 37(1):67–100.
• Schelble etal. (2022)BeauG. Schelble, Christopher Flathmann, NathanJ. McNeese, Guo Freeman, and Rohit Mallick. 2022.Let’s think together! assessing shared mental models, performance, and trust in human-agent teams.Proc. ACM Hum.-Comput. Interact., 6(GROUP).
• Shaikh etal. (2023)Omar Shaikh, Caleb Ziems, William Held, AryanJ Pariani, Fred Morstatter, and Diyi Yang. 2023.Modeling cross-cultural pragmatic inference with codenames duet.arXiv preprint arXiv:2306.02475.
• Shore etal. (2018)Todd Shore, Theofronia Androulakaki, and Gabriel Skantze. 2018.KTH tangrams: A dataset for research on alignment and conceptual pacts in task-oriented dialogue.In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
• Stewart etal. (2023)AngelaE.B. Stewart, Arjun Rao, Amanda Michaels, Chen Sun, NicholasD. Duran, ValerieJ. Shute, and SidneyK. D’Mello. 2023.Cpscoach: The design and implementation of intelligent collaborative problem solving feedback.In Artificial Intelligence in Education - 24th International Conference, AIED 2023, Proceedings, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pages 695–700, Germany. Springer Science and Business Media Deutschland GmbH.Publisher Copyright: © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 24th International Conference on Artificial Intelligence in Education, AIED 2023 ; Conference date: 03-07-2023 Through 07-07-2023.
• Suhr etal. (2019)Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019.Executing instructions in situated collaborative interactions.In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2119–2130, Hong Kong, China. Association for Computational Linguistics.
• Sun etal. (2020)Chen Sun, ValerieJ. Shute, Angela Stewart, Jade Yonehiro, Nicholas Duran, and Sidney D’Mello. 2020.Towards a generalized competency model of collaborative problem solving.Computers and Education, 143.
• Takmaz etal. (2020)Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, and Raquel Fernández. 2020.Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts.In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4350–4368, Online. Association for Computational Linguistics.
• Tenbrink etal. (2017)Thora Tenbrink, Elena Andonova, Gesa Schole, and KennyR. Coventry. 2017.Communicative success in spatial dialogue: The impact of functional features and dialogue strategies.Language and Speech, 60(2):318–329.PMID: 28697700.
• Tsiros and Palladini (2020)Augoustinos Tsiros and Alessandro Palladini. 2020.Towards a human-centric design framework for ai assisted music production.In New Interfaces for Musical Expression.
• Vats etal. (2024)Vanshika Vats, MarziaBinta Nizam, Minghao Liu, Ziyuan Wang, Richard Ho, MohnishSai Prasad, Vincent Titterton, SaiVenkat Malreddy, Riya Aggarwal, Yanwen Xu, etal. 2024.A survey on human-ai teaming with large pre-trained models.arXiv preprint arXiv:2403.04931.
• von Ahn (2013)Luis von Ahn. 2013.Duolingo: learn a language for free while helping to translate the web.In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, page 1–2, New York, NY, USA. Association for Computing Machinery.
• von Ahn etal. (2006)Luis von Ahn, Mihir Kedia, and Manuel Blum. 2006.Verbosity: a game for collecting common-sense facts.In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’06, page 75–78, New York, NY, USA. Association for Computing Machinery.
• Wang etal. (2023)Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023.Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration.arXiv preprint arXiv:2307.05300.
• Wei etal. (2023)Jimmy Wei, Kurt Shuster, Arthur Szlam, Jason Weston, Jack Urbanek, and Mojtaba Komeili. 2023.Multi-party chat: Conversational agents in group settings with humans and models.ArXiv, abs/2304.13835.
• Wiltshire etal. (2018)TravisJ Wiltshire, JonathanE Butner, and StephenM Fiore. 2018.Problem-solving phase transitions during team collaboration.Cognitive science, 42(1):129–167.
• Zarrieß etal. (2016)Sina Zarrieß, Julian Hough, Casey Kennington, Ramesh Manuvinakurike, David DeVault, Raquel Fernández, and David Schlangen. 2016.PentoRef: A corpus of spoken references in task-oriented dialogues.In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 125–131, Portorož, Slovenia. European Language Resources Association (ELRA).
• Zhang etal. (2023)Guanglu Zhang, Leah Chong, Kenneth Kotovsky, and Jonathan Cagan. 2023.Trust in an ai versus a human teammate: The effects of teammate identity and performance on human-ai cooperation.Computers in Human Behavior, 139:107536.
• Zhang etal. (2021)Rui Zhang, NathanJ McNeese, Guo Freeman, and Geoff Musick. 2021." an ideal human" expectations of ai teammates in human-ai teaming.Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3):1–25.

Appendix A Tower Defense Designs

Currently implemented tower defense designs that can be adjusted to suit the specified CPS task are as follows.

We expect to implement other common game design paradigms such as segmenting the map so players can only place towers on their designated section as the platform matures.

Appendix B Case study results

Table4 describes our case study in the context of other tasks with open data.

Sample conversations are in Table5.

Appendix C Survey questions

The pre-survey collected basic demographic information.

The post-survey contained the Teamwork Quality questionnaire(Hoegl and Gemuenden, 2001), VIA Team roles inventory(Ruch etal., 2018), and an open-ended task-specific questionnaire. Both TWQ and VIA used a 7-point Likert scale with options: Strongly Disagree, Disagree, Somewhat Disagree, Neutral, Somewhat Agree, Agree, and Strongly Agree.

Appendix D CPS classification

Appendix E Potential CPS-TaskForge Tasks

We decided to use the tower defense game genre as the task for CPS-TaskForge after considering several other games.

Appendix F License

The Godot game engine has an MIT license.The terms for use of our artifactswill be included in our released package.