Andrew Kuznetsov, NRP
New: on the job market for industry research positions!
I'm a final year PhD student and CMLH Digital Health Innovation fellow in the School of Computer Science at Carnegie Mellon University working with Dr. Niki Kittur (Human Computer Interaction Institute) and Dr. Anita Woolley (Organization Behavior Theory, Tepper School of Business).
My PhD thesis work investigated how humans use information when performing complex tasks, such as conducting multi-step searches or implementing a care plan. This involved two synergistic approaches: (1) prototyping new systems to help people save and use task-related 'provenance' information, (2) developing new methods for collecting human task data at scale, which can then be used for modeling.
By collecting this data, we establish a foundation for augmenting human performance in everyday tasks rarely captured at scale, such as longer tasks involving many iterations of sensemaking, social knowledge acquisition, and social coordination. This data also enables us to study relevant theoretical constructs (e.g. sensemaking theory 'loops' and transactive memory 'systems').
I'm particularly proud of my work with the NSF AI-CARING AI Institute. In this segment of my thesis, I designed and oversaw a multi-year research program to create a novel methodology for collecting and modeling information retrieval behavior in small non-expert healthcare teams working in shifts, at scale.
Outside of the PhD, I am a nationally registered paramedic and conduct research on trauma triage systems.
My PhD research aimed to enable "distributed sensemaking" for everyday tasks. Traditional sensemaking research focuses on individual users performing tasks with hidden cognitive processes (e.g., comparing items, making trust judgments). By studying sensemaking in a distributed context, we can uncover and augment these hidden processes. In my thesis, I introduced "AI-Mediated Distributed Sensemaking" (illustrated above), a bottom-up approach that integrates rich provenance data into the traditional sensemaking cycle. This method not only enhances hidden steps of sensemaking but also re-imagines the orchestration of complex tasks in multi-agent or large human teams, building on organizational and cooperative work research that highlights the value of organic high-level structuring such as coordination, meta-reasoning, and standardization.
Publications
Tasks, Time, and Tools: Quantifying Online Sensemaking Through a Survey based Study
Arxiv, 2024.
In this work, we use a survey-based approach with aided recall focused on segmenting and contextualizing individual exploratory browsing sessions to conduct a mixed method analysis of everyday sensemaking sessions in the traditional desktop browser setting while preserving user privacy. We report data from our survey (n=111) collected in September, 2022, and use these results to update and deepen the rich literature on information seeking behavior and exploratory search, contributing new empirical insights into the time spent per week and distribution of that time across tasks, and the lack of externalization and tool-use despite widespread desire for support.
Transactive Memory in Caregiver Networks Using Artificial Intelligence
AAAI Fall Symposium Series (FSS), 2024.
[Summary]
In this work, we explore how artificial intelligence (AI) may be used to develop tools to help loosely connected care networks develop better collective cognition. Specifically, we focus on helping members of care networks develop a transactive memory system, or a shared system for storing and retrieving knowledge that expands the capacity of a group to effectively use information. In this paper, we describe the motivation for our study, and our planned research program based on the use of an online experimental platform facilitating human-AI collaboration to develop and test tools to enhance collective cognition in care networks.
The Collaborative Caring Virtual
Testbed: A software platform for prototyping collective intelligence interventions for asynchronous care-teams.
ACM Collective Intelligence (CI), 2024.
[Summary]
In this work, we demonstrate the Collaborative Caring Virtual Testbed, a software platform and framework designed to support online experimentation regarding sensemaking and collective intelligence tools for careteams. Each platform session features a caregiving simulation within a modular interface. Combined, the two components enable the evaluation of many different interventions within a variety of collaborative caring situations. In each session, the platform uses interface modules to implement interventions that users may interact with as they complete their tasks, such as the introduction of a software system (e.g. decision-making support, sensemaking-support, and team communication tools), AI-assistance (e.g. AI-mediated peer communication, interactive AI agents), and less technical interventions such as cognitive priming. In addition, the caregiving simulation can be configured to modify the care network available to each caregiver, and the number and health conditions of care recipients. The platform supports both individual user studies, as well as asynchronous team studies by connecting shift sessions together. In the future, we plan to integrate this platform with a physical testbed platform to enable the full-cycle simulation and validation of interventions for distributed teams. In this demonstration, we will be showing the platform as configured for the study of informal caregivers of individuals with mild-cognitive impairment (MCI).
Beyond efficiency: Commitment issues: Feedback, commitment, and performance in algorithmically managed contexts.
ACM Collective Intelligence (CI), 2024.
Fuse: In-Situ Sensemaking Support in the Browser
ACM Symposium on User Interface Software and Technology (UIST), 2022.
A browser extension that externalizes users’ working memory by combining low-cost collection with lightweight organization of content in a compact card-based sidebar that is always available. Fuse helps users simultaneously extract key web content and structure it in a lightweight and visual way. We discuss how these affordances help users externalize more of their mental model into the system (e.g., saving, annotating, and structuring items) and support fast reviewing and resumption of task contexts. Our 22-month public deployment (N=89) and follow-up interviews provide longitudinal insights into the structuring behaviors of real-world users conducting information foraging tasks.
Wigglite: Low-cost Information Collection and Triage
ACM Symposium on User Interface Software and Technology (UIST), 2022.
In this work, we explore a new interaction technique called “wiggling,” which can be used to fluidly collect, organize, and rate information during early sensemaking stages with a single gesture. Wiggling involves rapid back-and-forth movements of a pointer or up-and-down scrolling on a smartphone, which can indicate the information to be collected and its valence, using a single, light-weight gesture that does not interfere with other interactions that are already available. Through implementation and user evaluation, we found that wiggling helped participants accurately collect information and encode their mental context with a 58% reduction in operational cost while being 24% faster compared to a common baseline.
Templates and Trust-o-meters: Towards a widely deployable indicator of trust in Wikipedia.
ACM CHI Conference on Human Factors in Computing Systems (CHI), 2022.
In this work we identify and address three key challenges: empirically determining which metrics from prior and existing community approaches most impact reader trust; 2) validating indicator placements and designs that are both compact yet noticed by readers; and 3) demonstrating that such indicators can not only lower trust but also increase perceived trust in the system when appropriate. By addressing these, we aim to provide a foundation for future tools that can practically increase trust in user generated content and the sociotechnical systems that generate and maintain them.
An open repository of real-time COVID-19 indicators
Proceedings of the National Academy of Sciences (PNAS), 2021.
Operational since April 2020, the COVIDcast API provides open access to both traditional public health surveillance signals (cases, deaths, and hospitalizations) and many auxiliary indicators of COVID-19 activity, such as signals extracted from deidentified medical claims data, massive online surveys, cell phone mobility data, and internet search trends. These are available at a fine geographic resolution (mostly at the county level) and are updated daily. The COVIDcast API also tracks all revisions to historical data, allowing modelers to account for the frequent revisions and backfill that are common for many public health data sources. All of the data are available in a common format through the API and accompanying R and Python software packages. This paper describes the data sources and signals, and provides examples demonstrating that the auxiliary signals in the COVIDcast API present information relevant to tracking COVID activity, augmenting traditional public health reporting and empowering research and decision-making.
LIFT: Integrating Stakeholder Voices into Algorithmic Team Formation
ACM CHI Conference on Human Factors in Computing Systems (CHI), 2020.
A novel learner-centered workflow where students propose, vote for, and weigh the criteria used as inputs to the team formation algorithm. We conducted an experiment (N=289) comparing LIFT to the usual instructor-led process, and interviewed participants to evaluate their perceptions of LIFT and its outcomes. Learners proposed novel criteria not included in existing algorithmic tools, such as organizational style. They avoided criteria like gender and GPA that instructors frequently select, and preferred those promoting efficient collaboration. LIFT led to team outcomes comparable to those achieved by the instructor-led approach, and teams valued having control of the team formation process. We provide instructors and designers with a workflow and evidence supporting giving learners control of the algorithmic process used for grouping them into teams.
It's just a matter of perspective (s): Crowd-Powered Consensus Organization of Corpora
arXiv, 2016.
[Summary]
[arxiv]
We develop cost-efficient, accurate algorithms for identifying the consensus organization (i.e., the organizing perspective most workers prefer to employ), and incorporate these algorithms into a cost-effective workflow for organizing a collection of objects, termed ORCHESTRA. We compare our algorithms with other algorithms for clustering, on a variety of real-world datasets, and demonstrate that ORCHESTRA organizes items better and at significantly lower costs.
Other Work & Demos
Outside of full-stack web development, I maintain a wide range of prototyping experiences, including mobile development, AR/VR , hardware and IoT devices, as well as some more esoteric stuff like Solidity (Ethereum).
Left 4 Virtual Reality (2015)
Re-purposing Consumer Toys as VR Input Devices
Nerf toy // Wii controller // Microsoft Kinect // Hardware flex sensors // Particle, Arduino micro-controller board.
[Github]
QuickMed (2015)
Prototype Mobile Platform for Field Delivery of Medication
Map web app // iOS Mobile application
[Github]
StreamPoint (2016)
Prototype Meeting Software to Generate Real-time Slides During Presentation
Presentation web app // Bing API // iOS Mobile application // NLP // Voice-to-Text.
[Github]
Search3 (2018)
Prototype Data Network for Search and Rescue Robotics
Ethereum smart contract // Computer vision embeddings // Camera-equipped drone // iOS mobile application.
[Github]
Exponent Network (2018)
Prototype Data Marketplace for Crowdsourcing Search Leads
Ethereum smart contact // Modified query incentive network mechanism.
[Github]
PhD Positions Dashboard (2023)
Deployed System for Collecting Open CS/HCI PhD Positions, ∼15,000 yearly users.
Multi-agent LLM orchestration // Google sheets API // Image-to-text
[Website]