Publications

The boardgame Diplomacy is a challenging setting for communicative and cooperative artificial intelligence. The most prominent communicative Diplomacy AI, Cicero, has excellent strategic abilities, exceeding human players. However, the best Diplomacy players master communication, not just tactics, which is why the game has received attention as an AI challenge. This work seeks to understand the degree to which Cicero succeeds at communication. First, we annotate in-game communication with abstract meaning representation to separate in-game tactics from general language. Second, we run two dozen games with humans and Cicero, totaling over 200 human-player hours of competition. While AI can consistently outplay human players, AI-Human communication is still limited because of AI's difficulty with deception and persuasion. This shows that Cicero relies on strategy and has not yet reached the full promise of communicative and cooperative AI.

        @InProceedings{acl24diplomacy,
  title     = {More Victories, Less Cooperation: Assessing Cicero's Diplomacy Play}, 
  author    = {Wichayaporn Wongkamjan and Feng Gu and Yanze Wang and Ulf Hermjakob and Jonathan May and Brandon M. Stewart and Jonathan K. Kummerfeld and Denis Peskoff and Jordan Lee Boyd-Graber},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  month     = {August},
  year      = {2024},
  address   = {Bangkok, Thailand},
  url       = {https://arxiv.org/abs/2406.04643}, 
  pages     = {},
}
        

Readers find text difficult to consume for many reasons. Summarization can address some of these difficulties, but introduce others, such as omitting, misrepresenting, or hallucinating information, which can be hard for a reader to notice. One approach to addressing this problem is to instead modify how the original text is rendered to make important information more salient. We introduce Grammar-Preserving Text Saliency Modulation (GP-TSM), a text rendering method with a novel means of identifying what to de-emphasize. Specifically, GP-TSM uses a recursive sentence compression method to identify successive levels of detail beyond the core meaning of a passage, which are de-emphasized by rendering words in successively lighter but still legible gray text. In a lab study (n=18), participants preferred GP-TSM over pre-existing word-level text rendering methods and were able to answer GRE reading comprehension questions more efficiently.

        @InProceedings{ch124gptsm,
  author    = {Gu, Ziwei and Arawjo, Ian and Li, Kenneth and Kummerfeld, Jonathan K. and Glassman, Elena L.},
  title     = {An AI-Resilient Text Rendering Technique for Reading and Skimming Documents},
  booktitle = {Proceedings of the CHI Conference on Human Factors in Computing Systems},
  location  = {Honolulu, HI, USA},
  month     = {May},
  year      = {2024},
  doi       = {10.1145/3613904.3642699},
  pages     = {},
  url       = {https://doi.org/10.1145/3613904.3642699},
  isbn      = {9798400703300},
  software  = {https://github.com/ZiweiGu/GP-TSM},
  arxiv     = {https://arxiv.org/abs/2401.10873},
}
        

While alignment algorithms are now commonly used to tune pre-trained language models towards a user's preferences, we lack explanations for the underlying mechanisms in which models become ``aligned'', thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in a pre-trained language model, GPT2-medium. We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting model averts toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the model, reverting it back to its toxic behavior.

        @InProceedings{icml24mechanistic,
  author    = {Andrew Lee and Xiaoyan Bai and Itamar Pres and Martin Wattenberg and Jonathan K. Kummerfeld and Rada Mihalcea},
  title     = {A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity},
  booktitle = {Proceedings of the 41st International Conference on Machine Learning},
  month     = {July},
  year      = {2024},
  arxiv     = {https://arxiv.org/abs/2401.01967},
  data      = {},
  awards    = {Oral Presentation},
}
        

Large language models (LLMs) are capable of generating multiple responses to a single prompt, yet little effort has been expended to help end-users or system designers make use of this capability. In this paper, we explore how to present many LLM responses at once. We design five features, which include both pre-existing and novel methods for computing similarities and differences across textual documents, as well as how to render their outputs. We report on a controlled user study (n=24) and eight case studies evaluating these features and how they support users in different tasks. We find that the features support a wide variety of sensemaking tasks and even make tasks tractable that our participants previously considered to be too difficult to attempt. Finally, we present design guidelines to inform future explorations of new LLM interfaces.

        @InProceedings{ch124sense,
  author    = {Gero, Katy Ilonka and Swoopes, Chelse and Gu, Ziwei and Kummerfeld, Jonathan K. and Glassman, Elena L.},
  title     = {Supporting Sensemaking of Large Language Model Outputs at Scale},
  booktitle = {Proceedings of the CHI Conference on Human Factors in Computing Systems},
  location  = {Honolulu, HI, USA},
  month     = {May},
  year      = {2024},
  doi       = {10.1145/3613904.3642139},
  pages     = {},
  url       = {https://doi.org/10.1145/3613904.3642139},
  arxiv     = {https://arxiv.org/abs/2401.13726},
  isbn      = {9798400703300},
}
        

Recently, empathetic dialogue systems have received significant attention.While some researchers have noted limitations, e.g., that these systems tend to generate generic utterances, no study has systematically verified these issues. We survey 21 systems, asking what progress has been made on the task. We observe multiple limitations of current evaluation procedures. Most critically, studies tend to rely on a single non-reproducible empathy score, which inadequately reflects the multidimensional nature of empathy. To better understand the differences between systems, we comprehensively analyze each system with automated methods that are grounded in a variety of aspects of empathy. We find that recent systems lack three important aspects of empathy: specificity, reflection levels, and diversity. Based on our results, we discuss problematic behaviors that may have gone undetected in prior evaluations, and offer guidance for developing future systems.

        @InProceedings{eacl24empathy,
  title     = {A Comparative Multidimensional Analysis of Empathetic Systems},
  author    = {Lee, Andrew and Kummerfeld, Jonathan K. and Ann, Larry  and Mihalcea, Rada},
  year      = {2024},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  month     = {March},
  url       = {https://aclanthology.org/2024.eacl-long.11},
  location  = {St. Julian{'}s, Malta},
  pages     = {179--189},
}
        

        @InProceedings{uist24sql,
  title     = {SQLucid: Grounding Natural Language Database Queries with Interactive Explanations},
  author    = {Yuan Tian and Jonathan K. Kummerfeld and Toby Jia-Jun Li and Tianyi Zhang},
  year      = {2024},
  booktitle = {ACM Symposium on User Interface Software and Technology},
  month     = {October},
  url       = {},
  location  = {Pittsburgh, PA, USA},
  pages     = {},
}
        

Relational databases play an important role in business, science, and more. However, many users cannot fully unleash the analytical power of relational databases, because they are not familiar with database languages such as SQL. Many techniques have been proposed to automatically generate SQL from natural language, but they suffer from two issues: (1) they still make many mistakes, particularly for complex queries, and (2) they do not provide a flexible way for non-expert users to validate and refine incorrect queries. To address these issues, we introduce a new interaction mechanism that allows users to directly edit a step-by-step explanation of a query to fix errors. Our experiments on multiple datasets, as well as a user study with 24 participants, demonstrate that our approach can achieve better performance than multiple SOTA approaches. Our code and datasets are available at https://github.com/magic-YuanTian/STEPS.

        @inproceedings{emnlp23sql,
  title     = {Interactive Text-to-{SQL} Generation via Editable Step-by-Step Explanations},
  author    = {Yuan Tian and Zheng Zhang and Zheng Ning and Toby Li and Jonathan K. Kummerfeld and Tianyi Zhang},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  month     = {December},
  year      = {2023},
  address   = {Singapore},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2023.emnlp-main.1004},
  doi       = {10.18653/v1/2023.emnlp-main.1004},
  pages     = {16149--16166},
  software  = {https://github.com/magic-YuanTian/STEPS},
  poster    = {https://www.jkk.name/pub/emnlp23sql_poster.pdf},
}
        

Understanding empathy in text dialogue data is a difficult, yet critical, skill for effective human-machine interaction. In this work, we ask whether systems are making meaningful progress on this challenge. We consider a simple model that checks if an input utterance is similar to a small set of empathetic examples. Crucially, the model does not look at what the utterance is a response to, i.e., the dialogue context. This model performs comparably to other work on standard benchmarks and even outperforms state-of-the-art models for empathetic rationale extraction by 16.7 points on T-F1 and 4.3 on IOU-F1. This indicates that current systems rely on the surface form of the response, rather than whether it is suitable in context. To confirm this, we create examples with dialogue contexts that change the interpretation of the response and show that current systems continue to label utterances as empathetic. We discuss the implications of our findings, including improvements for empathetic benchmarks and how our model can be an informative baseline.

        @InProceedings{eacl23empathy,
  title     = {Empathy Identification Systems are not Accurately Accounting for Context},
  author    = {Lee, Andrew and Kummerfeld, Jonathan K. and An, Lawrence C. and Mihalcea, Rada},
  year      = {2023},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  month     = {May},
  url       = {https://aclanthology.org/2023.eacl-main.123/},
  location  = {Dubrovnik, Croatia},
  pages     = {1686--1695},
}
        

Conversation disentanglement is the task of taking a log of intertwined conversations from a shared channel and breaking the log into individual conversations. The standard datasets for disentanglement are in a single domain and were annotated by linguistics experts with careful training for the task. In this paper, we introduce the first multi-domain dataset and a study of annotation by people without linguistics expertise or extensive training. We experiment with several variations in interfaces, conducting user studies with domain experts and crowd workers. We also test a hypothesis from prior work that link-based annotation is more accurate, finding that it actually has comparable accuracy to set-based annotation. Our new dataset will support the development of more useful systems for this task, and our experimental findings suggest that users are capable of improving the usefulness of these systems by accurately annotating their own data.

        @InProceedings{alta23disentangle,
  author    = {Sai R. Gouravajhala and Andrew M. Vernier and Yiming Shi and Zihan Li and Mark Ackerman and Jonathan K. Kummerfeld},
  title     = {Chat Disentanglement: Data for New Domains and Methods for More Accurate Annotation},
  booktitle = {Proceedings of the The 21st Annual Workshop of the Australasian Language Technology Association},
  location  = {Melbourne, Australia},
  month     = {November},
  year      = {2023},
  pages     = {112--117},
  url       = {https://aclanthology.org/2023.alta-1.12/},
  data      = {https://www.jkk.name/irc-disentanglement},
}
        

Personalized language models are designed and trained to capture language patterns specific to individual users. This makes them more accurate at predicting what a user will write. However, when a new user joins a platform and not enough text is available, it is harder to build effective personalized language models. We propose a solution for this problem, using a model trained on users that are similar to a new user. In this paper, we explore strategies for finding the similarity between new users and existing ones and methods for using the data from existing users who are a good match. We further explore the trade-off between available data for new users and how well their language can be modeled.

        @InProceedings{acl22personal,
  title     = {Leveraging Similar Users for Personalized Language Modeling with Limited Data},
  author    = {Welch, Charles and Gu, Chenxi and Kummerfeld, Jonathan K. and Perez-Rosas, Veronica and Mihalcea, Rada},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  month     = {May},
  year      = {2022},
  address   = {Dublin, Ireland},
  url       = {https://aclanthology.org/2022.acl-long.122},
  doi       = {10.18653/v1/2022.acl-long.122},
  pages     = {1742--1752},
}
        

We use paraphrases as a unique source of data to analyze contextualized embeddings, with a particular focus on BERT. Because paraphrases naturally encode consistent word and phrase semantics, they provide a unique lens for investigating properties of embeddings. Using the Paraphrase Database{'}s alignments, we study words within paraphrases as well as phrase representations. We find that contextual embeddings effectively handle polysemous words, but give synonyms surprisingly different representations in many cases. We confirm previous findings that BERT is sensitive to word order, but find slightly different patterns than prior work in terms of the level of contextualization across BERT{'}s layers.

        @InProceedings{naacl22embeddings,
  title     = {Using Paraphrases to Study Properties of Contextual Embeddings},
  author    = {Burdick, Laura and Kummerfeld, Jonathan K. and Mihalcea, Rada},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  location  = {Seattle, United States},
  pages     = {4558--4568},
  month     = {July},
  year      = {2022},
  url       = {https://aclanthology.org/2022.naacl-main.338},
}
        

The standard task-oriented dialogue pipeline uses intent classification and slot-filling to interpret user utterances. While this approach can handle a wide range of queries, it does not extract the information needed to handle more complex queries that contain relationships between slots. We propose integration of relation extraction into this pipeline as an effective way to expand the capabilities of dialogue systems. We evaluate our approach by using an internal dataset with slot and relation annotations spanning three domains. Finally, we show how slot-filling annotation schemes can be simplified once the expressive power of relation annotations is available, reducing the number of slots while still capturing the user's intended meaning.

        @InProceedings{ws-aaai-dstc22re,
  title     = {Augmenting Task-Oriented Dialogue Systems with Relation Extraction},
  author    = {Lee, Andrew and Chen, Zhenguo and Leach, Kevin and Kummerfeld, Jonathan K.},
  year      = {2022},
  booktitle = {10th Edition of the Dialog System Technology Challenges at AAAI 2022},
  month     = {March},
  url       = {https://arxiv.org/pdf/2210.13344.pdf},
}
        

Word embeddings are powerful representations that form the foundation of many natural language processing architectures, both in English and in other languages. To gain further insight into word embeddings, we explore their stability (e.g., overlap between the nearest neighbors of a word in different embedding spaces) in diverse languages. We discuss linguistic properties that are related to stability, drawing out insights about correlations with affixing, language gender systems, and other features. This has implications for embedding use, particularly in research that uses them to study language trends.

        @InProceedings{emnlp21stability,
  title     = {Analyzing the Surprising Variability in Word Embedding Stability Across Languages},
  author    = {Burdick, Laura and Kummerfeld, Jonathan K. and Mihalcea, Rada},
  doi       = {10.18653/v1/2021.emnlp-main.476},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  month     = {November},
  year      = {2021},
  location  = {Online and Punta Cana, Dominican Republic},
  pages     = {5891--5901},
  url       = {https://aclanthology.org/2021.emnlp-main.476},
  arxiv     = {https://arxiv.org/pdf/2004.14876.pdf},
  video     = {https://aclanthology.org/2021.emnlp-main.476.mp4},
  software  = {https://github.com/laura-burdick/multilingual-stability},
}
        

Many statistical models have high accuracy on test benchmarks, but are not explainable, struggle in low-resource scenarios, cannot be reused for multiple tasks, and cannot easily integrate domain expertise. These factors limit their use, particularly in settings such as mental health, where it is difficult to annotate datasets and model outputs have significant impact. We introduce a micromodel architecture to address these challenges. Our approach allows researchers to build interpretable representations that embed domain knowledge and provide explanations throughout the model's decision process. We demonstrate the idea on multiple mental health tasks: depression classification, PTSD classification, and suicidal risk assessment. Our systems consistently produce strong results, even in low-resource scenarios, and are more interpretable than alternative methods.

        @InProceedings{emnlp-findings21micromodels,
  title     = {Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Health},
  author    = {Lee, Andrew and Kummerfeld, Jonathan K. and An, Lawrence C. and Mihalcea, Rada},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Findings},
  month     = {November},
  year      = {2021},
  location  = {Punta Cana, Dominican Republic},
  pages     = {4257--4272},
  url       = {https://aclanthology.org/2021.findings-emnlp.360},
  doi       = {10.18653/v1/2021.findings-emnlp.360},
  arxiv     = {https://arxiv.org/abs/2109.13770},
  software  = {https://github.com/MichiganNLP/micromodels},
}
        

Extensive work has argued in favour of paying crowd workers a wage that is at least equivalent to the U.S. federal minimum wage. Meanwhile, research on collecting high quality annotations suggests using a qualification that requires workers to have previously completed a certain number of tasks. If most requesters who pay fairly require workers to have completed a large number of tasks already then workers need to complete a substantial amount of poorly paid work before they can earn a fair wage. Through analysis of worker discussions and guidance for researchers, we estimate that workers spend approximately 2.25 months of full time effort on poorly paid tasks in order to get the qualifications needed for better paid tasks. We discuss alternatives to this qualification and conduct a study of the correlation between qualifications and work quality on two NLP tasks. We find that it is possible to reduce the burden on workers while still collecting high quality data.

        @InProceedings{acl21fair-work,
  author    = {Kummerfeld, Jonathan K.},
  title     = {Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing},
  location  = {Online},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  month     = {August},
  year      = {2021},
  url       = {https://aclanthology.org/2021.acl-short.44/},
  pages     = {343--349},
  doi       = {10.18653/v1/2021.acl-short.44},
  arxiv     = {https://arxiv.org/abs/2105.12762},
  supplementary = {https://aclanthology.org/attachments/2021.acl-short.44.OptionalSupplementaryMaterial.zip},
  video     = {https://aclanthology.org/2021.acl-short.44.mp4},
}
        

A growing number of people engage in online health forums, making it important to understand the quality of the advice they receive. In this paper, we explore the role of expertise in responses provided to help-seeking posts regarding mental health. We study the differences between (1) interactions with peers; and (2) interactions with self-identified mental health professionals. First, we show that a classifier can distinguish between these two groups, indicating that their language use does in fact differ. To understand this difference, we perform several analyses addressing engagement aspects, including whether their comments engage the support-seeker further as well as linguistic aspects, such as dominant language and linguistic style matching. Our work contributes toward the developing efforts of understanding how health experts engage with health information- and support-seekers in social networks. More broadly, it is a step toward a deeper understanding of the styles of interactions that cultivate supportive engagement in online communities.

        @InProceedings{acl21counseling,
  author    = {Lahnala, Allison and Zhao, Yuntian and Welch, Charles and Kummerfeld, Jonathan K. and An, Lawrence C. and Resnicow, Kenneth and Mihalcea, Rada and P{\\'e}rez-Rosas, Ver{\\'o}nica},
  title     = {Exploring Self-Identified Counseling Expertise in Online Support Forums},
  location  = {Online},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  month     = {August},
  year      = {2021},
  url       = {https://aclanthology.org/2021.findings-acl.392/},
  pages     = {4467--4480},
  doi       = {10.18653/v1/2021.findings-acl.392},
  software  = {https://github.com/MichiganNLP/MHP-and-Peers-Reddit},
}
        

Natural language processing methods have been applied in a variety of music studies, drawing the connection between music and language. In this paper, we expand those approaches by investigating chord embeddings, which we apply in two case studies to address two key questions: (1) what musical information do chord embeddings capture?; and (2) how might musical applications benefit from them? In our analysis, we show that they capture similarities between chords that adhere to important relationships described in music theory. In the first case study, we demonstrate that using chord embeddings in a next chord prediction task yields predictions that more closely match those by experienced musicians. In the second case study, we show the potential benefits of using the representations in tasks related to musical stylometrics.

        @InProceedings{evomusart21,
  title     = {Chord Embeddings: Analyzing What They Capture and Their Role for Next Chord Prediction and Artist Attribute Prediction},
  author    = {Lahnala, Allison and Kambhatla, Gauri and Peng, Jiajun and Whitehead, Matthew and Minnehan, Gillian and Guldan, Eric and Kummerfeld, Jonathan K. and Çamcı, Anıl and Mihalcea, Rada},
  booktitle = {Proceedings of the 10th International Conference on Artificial Intelligence in Music, Sound, Art and Design},
  year      = {2021},
  month     = {April},
  url       = {https://arxiv.org/pdf/2102.02917.pdf},
  pages     = {171--186},
}
        

This paper introduces the Eighth Dialog System Technology Challenge. In line with recent challenges, the eighth edition focuses on applying end-to-end dialog technologies in a pragmatic way for multi-domain task-completion, noetic response selection, audio visual scene-aware dialog, and schema-guided dialog state tracking tasks. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.

        @Article{dstc8-ieee,
  title     = {Overview of the Eighth Dialog System Technology Challenge: DSTC8},
  author    = {Kim, Seokhwan and Galley, Michel and Gunasekara, Chulaka and Lee, Sungjin and Atkinson, Adam and Baolin, Peng and Schulz, Hannes and Gao, Jianfeng and Li, Jinchao and Adada, Mahmoud and Huang, Minlie and Lastras, Luis and Kummerfeld, Jonathan K. and Lasecki, Walter S. and Hori, Chiori and Cherian, Anoop and Marks, Tim and Rastogi, Abhinav and Zang, Xiaoxue and Sunkara, Srinivas and Gupta, Raghav},
  journal   = {IEEE/ACM Transactions on Audio, Speech, and Language Processing},
  publisher = {IEEE},
  doi       = {10.1109/TASLP.2021.3078368},
  year      = {2021},
  url       = {https://ieeexplore.ieee.org/document/9426430},
  volume    = {29},
  number    = {1},
  pages     = {2529--2540},
}
        

Many natural language processing architectures are greatly affected by seemingly small design decisions, such as batching and curriculum learning (how the training data are ordered during training). In order to better understand the impact of these decisions, we present a systematic analysis of different curriculum learning strategies and different batching strategies. We consider multiple datasets for three tasks: text classification, sentence and phrase similarity, and part-of-speech tagging. Our experiments demonstrate that certain curriculum learning and batching decisions do increase performance substantially for some tasks.

        @Article{math21batch,
  author    = {Burdick, Laura and Kummerfeld, Jonathan K. and Mihalcea, Rada},
  title     = {To Batch or Not to Batch? Comparing Batching and Curriculum Learning Strategies across Tasks and Datasets},
  journal   = {Mathematics},
  volume    = {9},
  year      = {2021},
  number    = {18},
  article-number = {2234},
  url       = {https://www.jkk.name/pub/math21batch.pdf},
  issn      = {2227-7390},
  doi      = {10.3390/math9182234}
}
        

For each goal-oriented dialog task of interest, large amounts of data need to be collected for end-to-end learning of a neural dialog system. Collecting that data is a costly and time-consuming process. Instead, we show that we can use only a small amount of data, supplemented with data from a related dialog task. Naively learning from related data fails to improve performance as the related data can be inconsistent with the target task. We describe a meta-learning based method that selectively learns from the related dialog task data. Our approach leads to significant accuracy improvements in an example dialog task.

        @InProceedings{ws-emnlp-convai21learn,
  author    = {Rajendran, Janarthanan and Kummerfeld, Jonathan K. and Singh, Satinder},
  title     = {Learning to Learn End-to-End Goal-Oriented Dialog From Related Dialog Tasks},
  location  = {Online},
  booktitle = {3rd Workshop on NLP for ConvAI},
  month     = {November},
  year      = {2021},
  url       = {https://aclanthology.org/2021.nlp4convai-1.16},
  pages     = {163--178},
  doi       = {10.18653/v1/2021.nlp4convai-1.16},
  arxiv     = {https://arxiv.org/abs/2110.15724},
}
        

Word embeddings are usually derived from corpora containing text from many individuals, thus leading to general purpose representations rather than individually personalized representations. While personalized embeddings can be useful to improve language model performance and other language processing tasks, they can only be computed for people with a large amount of longitudinal data, which is not the case for new users. We propose a new form of personalized word embeddings that use demographic-specific word representations derived compositionally from full or partial demographic information for a user (i.e., gender, age, location, religion). We show that the resulting demographic-aware word representations outperform generic word representations on two tasks for English: language modeling and word associations. We further explore the trade-off between the number of available attributes and their relative effectiveness and discuss the ethical implications of using them.

        @InProceedings{emnlp20demographics,
  title     = {Compositional Demographic Word Embeddings},
  author    = {Welch, Charles and Kummerfeld, Jonathan K. and P{\\'e}rez-Rosas, Ver{\\'o}nica and Mihalcea, Rada},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  month     = {November},
  year      = {2020},
  location  = {Online},
  pages     = {4076--4089},
  url       = {https://www.aclweb.org/anthology/2020.emnlp-main.334},
  doi       = {10.18653/v1/2020.emnlp-main.334},
  arxiv     = {https://arxiv.org/abs/2010.02986},
  blog_post = {/post/2020-10-10_demographicembeddings/},
  video     = {https://slideslive.com/38939153},
}
        

Many NLP applications, such as biomedical data and technical support, have 10-100 million tokens of in-domain data and limited computational resources for learning from it. How should we train a language model in this scenario? Most language modeling research considers either a small dataset with a closed vocabulary (like the standard 1 million token Penn Treebank), or the whole web with byte-pair encoding. We show that for our target setting in English, initialising and freezing input embeddings using in-domain data can improve language model performance by providing a useful representation of rare words, and this pattern holds across several different domains. In the process, we show that the standard convention of tying input and output embeddings does not improve perplexity when initializing with embeddings trained on in-domain data.

        @InProceedings{emnlp20lm,
  title     = {Improving Low Compute Language Modeling with In-Domain Embedding Initialisation},
  author    = {Welch, Charles and Mihalcea, Rada and Kummerfeld, Jonathan K.},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing},
  month     = {November},
  year      = {2020},
  location  = {Online},
  pages     = {8625--8634},
  url       = {https://www.aclweb.org/anthology/2020.emnlp-main.696},
  supplementary = {https://www.aclweb.org/anthology/attachments/2020.emnlp-main.696.OptionalSupplementaryMaterial.zip},
  doi       = {10.18653/v1/2020.emnlp-main.696},
  software  = {https://github.com/jkkummerfeld/emnlp20lm},
  arxiv     = {https://arxiv.org/abs/2009.14109},
  blog_post = {/post/2020-09-29_PretrainingLM/},
  video     = {https://slideslive.com/38938903},
}
        

Diverse data is crucial for training robust models, but crowdsourced text often lacks diversity as workers tend to write simple variations from prompts. We propose a general approach for guiding workers to write more diverse text by iteratively constraining their writing. We show how prior workflows are special cases of our approach, and present a way to apply the approach to dialog tasks such as intent classification and slot-filling. Using our method, we create more challenging versions of test sets from prior dialog datasets and find dramatic performance drops for standard models. Finally, we show that our approach is complementary to recent work on improving data diversity, and training on data collected with our approach leads to more robust models.

        @InProceedings{emnlp20taboo,
  title     = {Iterative Feature Mining for Constraint-Based Data Collection to Increase Data Diversity and Model Robustness},
  author    = {Larson, Stefan and Zheng, Anthony and Mahendran, Anish and Tekriwal, Rishi and Cheung, Adrian and Guldan, Eric and Leach, Kevin and Kummerfeld, Jonathan K.},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  month     = {November},
  year      = {2020},
  location  = {Online},
  pages     = {8097--8106},
  url       = {https://www.aclweb.org/anthology/2020.emnlp-main.650},
  doi       = {10.18653/v1/2020.emnlp-main.650},
  blog_post = {/post/2020-10-10_taboo/},
  video     = {https://slideslive.com/38938895},
}
        

Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy. In particular, we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels. Our three-stage workflow produces annotations with 95% accuracy for predicate labels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we decrease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable annotation of SRL, and could enable annotation of NLP tasks that have previously been considered too complex to effectively crowdsource.

        @InProceedings{emnlp-findings20srl,
  title     = {A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels},
  author    = {Jiang, Youxuan and Zhu, Huaiyu and Kummerfeld, Jonathan K. and Li, Yunyao and Lasecki, Walter},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings},
  month     = {November},
  year      = {2020},
  location  = {Online},
  pages     = {415--421},
  url       = {https://www.aclweb.org/anthology/2020.findings-emnlp.38},
  doi       = {10.18653/v1/2020.findings-emnlp.38},
  blog_post = {/post/2020-10-04_CrowdSRL/},
}
        

Detecting rhetoric that manipulates readers’ emotions requires distinguishing intrinsically emotional content (IEC; e.g., a parent losing a child) from emotionally manipulative language (EML; e.g., using fear-inducing language to spread anti-vaccine propaganda). However, this remains an open classifcation challenge for both automatic and crowdsourcing approaches. Machine Learning approaches only work in narrow domains where labeled training data is available, and non-expert annotators tend to confate IEC with EML. We introduce an approach, anchor comparison, that leverages workers’ ability to identify and remove instances of EML in text to create a paraphrased 'anchor text', which is then used as a comparison point to classify EML in the original content. We evaluate our approach with a dataset of news-style text snippets and show that precision and recall can be tuned for system builders’ needs. Our contribution is a crowdsourcing approach that enables non-expert disentanglement of social references from content.

        @InProceedings{chi20anchor,
  author    = {Huffaker, Jordan S. and Kummerfeld, Jonathan K. and Lasecki, Walter S. and Ackerman, Mark S.},
  title     = {Crowdsourced Detection of Emotionally Manipulative Language},
  year      = {2020},
  publisher = {Association for Computing Machinery},
  booktitle = {Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems},
  isbn      = {9781450367080},
  address   = {New York, NY, USA},
  doi       = {10.1145/3313831.3376375},
  pages     = {1--14},
  location  = {Honolulu, HI, USA},
  url       = {https://www.jkk.name/pub/chi20anchor.pdf},
}
        

This paper provides detailed information about the seventh Dialog System Technology Challenge (DSTC7) and its three tracks aimed to explore the problem of building robust and accurate end-to-end dialog systems. In more detail, DSTC7 focuses on developing and exploring end-to-end technologies for the following three pragmatic challenges: (1) sentence selection for multiple domains, (2) generation of informational responses grounded in external knowledge, and (3) audio visual scene-aware dialog to allow conversations with users about objects and events around them. This paper summarizes the overall setup and results of DSTC7, including detailed descriptions of the different tracks, provided datasets and annotations, overview of the submitted systems and their final results. For Track 1, LSTM-based models performed best across both datasets, allowing teams to effectively handle task variants where no correct answer was present or when multiple paraphrases were included. For Track 2, RNN-based architectures augmented to incorporate facts by using two types of encoders: a dialog encoder and a fact encoder plus using attention mechanisms and a pointer-generator approach provided the best results. Finally, for Track 3, the best model used Hierarchical Attention mechanisms to combine the text and vision information obtaining a 22% better result than the baseline LSTM system for the human rating score. More than 220 participants were registered and about 40 teams participated in the final challenge. 32 scientific papers reporting the systems submitted to DSTC7, and 3 general technical papers for dialog technologies, were presented during the one-day wrap-up workshop at AAAI-19. During the workshop, we reviewed the state-of-the-art systems, shared novel approaches to the DSTC7 tasks, and discussed the future directions for the challenge (DSTC8).

        @article{csl20dstc,
  title     = {Overview of the seventh Dialog System Technology Challenge: DSTC7},
  journal   = {Computer Speech & Language},
  pages     = {101068},
  year      = {2020},
  issn      = {0885-2308},
  doi       = {https://doi.org/10.1016/j.csl.2020.101068},
  url       = {https://www.jkk.name/pub/csl20dstc.pdf},
  alt-url   = {http://www.sciencedirect.com/science/article/pii/S0885230820300012},
  data      = {https://ibm.github.io/dstc-noesis/public/index.html},
  author    = {D'Haro, Luis Fernando and Yoshino, Koichiro and Hori, Chiori and Marks, Tim K. and Polymenakos, Lazaros and Kummerfeld, Jonathan K. and Galley, Michel and Gao, Xiang},
}
        

Slot-filling models in task-driven dialog systems rely on carefully annotated training data. However, annotations by crowd workers are often inconsistent or contain errors. Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct. If we can identify inconsistencies, we can focus effort where it is needed. Toward this end, we define six inconsistency types in slot-filling annotations. Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed. We then introduce automatic methods of identifying inconsistencies. Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement.

        @InProceedings{coling20svp,
  title     = {Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods},
  author    = {Larson, Stefan and Cheung, Adrian and Mahendran, Anish and Leach, Kevin and Kummerfeld, Jonathan K.},
  year      = {2020},
  month     = {December},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  url       = {https://www.aclweb.org/anthology/2020.coling-main.442},
  pages     = {5035--5046},
  location  = {Barcelona, Spain (Online)},
}
        

In this paper, we introduce personalized word embeddings, and examine their value for language modeling. We compare the performance of our proposed prediction model when using personalized versus generic word representations, and study how these representations can be leveraged for improved performance. We provide insight into what types of words can be more accurately predicted when building personalized models. Our results show that a subset of words belonging to specific psycholinguistic categories tend to vary more in their representations across users and that combining generic and personalized word embeddings yields the best performance, with a 4.7{%} relative reduction in perplexity. Additionally, we show that a language model using personalized word embeddings can be effectively used for authorship attribution.

        @InProceedings{coling20personal,
  title     = {Exploring the Value of Personalized Word Embeddings},
  author    = {Welch, Charles and Kummerfeld, Jonathan K. and P{\\'e}rez-Rosas, Ver{\\'o}nica and Mihalcea, Rada},
  year      = {2020},
  month     = {December},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  pages     = {6856--6862},
  location  = {Barcelona, Spain (Online)},
  url       = {https://www.aclweb.org/anthology/2020.coling-main.604},
}
        

Diplomacy is a seven-player non-stochastic, non-cooperative game, where agents acquire resources through a mix of teamwork and betrayal. Reliance on trust and coordination makes Diplomacy the first non-cooperative multi-agent benchmark for complex sequential social dilemmas in a rich environment. In this work, we focus on training an agent that learns to play the No Press version of Diplomacy where there is no dedicated communication channel between players. We present DipNet, a neural-network-based policy model for No Press Diplomacy. The model was trained on a new dataset of more than 150,000 human games. Our model is trained by supervised learning (SL) from expert trajectories, which is then used to initialize a reinforcement learning (RL) agent trained through self-play. Both the SL and RL agents demonstrate state-of-the-art No Press performance by beating popular rule-based bots.

        @InProceedings{neurips19diplomacy,
  author    = {Paquette, Philip and Lu, Yuchen and Bocco, Steven and Smith, Max O. and Ortiz-Gagn{\\'e}, Satya and Kummerfeld, Jonathan K. and Pineau, Joelle and Singh, Satinder and Courville, Aaron},
  title     = {No-Press Diplomacy: Modeling Multi-Agent Gameplay},
  booktitle = {Advances in Neural Information Processing Systems 32},
  year      = {2019},
  month     = {December},
  pages     = {4476--4487},
  url       = {https://papers.nips.cc/paper/8697-no-press-diplomacy-modeling-multi-agent-gameplay},
  supplementary = {https://papers.nips.cc/paper/8697-no-press-diplomacy-modeling-multi-agent-gameplay-supplemental.zip},
  arxiv     = {https://arxiv.org/abs/1909.02128},
  blog_post = {/post/2019-09-13_diplomacynopress/},
}
        

Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.

        @InProceedings{emnlp19data,
  title     = {An Evaluation for Intent Classification and Out-of-Scope Prediction},
  author    = {Larson, Stefan and Mahendran, Anish and Peper, Joseph J. and Clarke, Christopher and Lee, Andrew and Hill, Parker and Kummerfeld, Jonathan K. and Leach, Kevin and Laurenzano, Michael A. and Tang, Lingjia and Mars, Jason},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
  month     = {November},
  year      = {2019},
  location  = {Hong Kong, China},
  pages     = {1311--1316},
  url       = {https://www.aclweb.org/anthology/D19-1131/},
  doi       = {10.18653/v1/D19-1131},
  data      = {https://github.com/clinc/oos-eval},
  arxiv     = {https://arxiv.org/abs/1909.02027},
}
        

Disentangling conversations mixed together in a single stream of messages is a difficult task, made harder by the lack of large manually annotated datasets. We created a new dataset of 77,563 messages manually annotated with reply-structure graphs that both disentangle conversations and define internal conversation structure. Our data is 16 times larger than all previously released datasets combined, the first to include adjudication of annotation disagreements, and the first to include context. We use our data to re-examine prior work, in particular, finding that 89% of conversations in a widely used dialogue corpus are either missing messages or contain extra messages. Our manually-annotated data presents an opportunity to develop robust data-driven methods for conversation disentanglement, which will help advance dialogue research.

        @InProceedings{acl19disentangle,
  author    = {Kummerfeld, Jonathan K. and Gouravajhala, Sai R. and Peper, Joseph J. and Athreya, Vignesh and Gunasekara, Chulaka and Ganhotra, Jatin and Patel, Siva Sankalp and Polymenakos, Lazaros and Lasecki, Walter S.},
  title     = {A Large-Scale Corpus for Conversation Disentanglement},
  location  = {Florence, Italy},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  month     = {July},
  year      = {2019},
  url       = {https://www.aclweb.org/anthology/P19-1374.pdf},
  pages     = {3846--3856},
  doi       = {10.18653/v1/P19-1374},
  arxiv     = {https://arxiv.org/abs/1810.11118},
  software  = {https://www.jkk.name/irc-disentanglement},
  data      = {https://www.jkk.name/irc-disentanglement},
  blog_post = {/post/2019-07-10_disentanglement/},
  supplementary = {https://www.aclweb.org/anthology/attachments/P19-1374.Supplementary.pdf},
  poster    = {https://www.jkk.name/pub/acl19disentangle_poster.pdf},
}
        

Many annotation tools have been developed, covering a wide variety of tasks and providing features like user management, pre-processing, and automatic labeling. However, all of these tools use a Graphical User Interface, and often require substantial effort for installation and configuration. This paper presents a new annotation tool that is designed to fill the niche of a lightweight interface for users with a terminal-based workflow. Slate supports annotation at different scales (spans of characters, tokens, and lines, or a document) and of different types (free text, labels, and links), with easily customisable keybindings, and unicode support. In a user study comparing with other tools it was consistently the easiest to install and use. Slate fills a need not met by existing systems, and has already been used to annotate two corpora, one of which involved over 250 hours of annotation effort.

        @InProceedings{acl19slate,
  title     = {SLATE: A Super-Lightweight Annotation Tool for Experts},
  author    = {Kummerfeld, Jonathan K.},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  location  = {Florence, Italy},
  month     = {July},
  year      = {2019},
  pages     = {7--12},
  url       = {https://www.aclweb.org/anthology/P19-3002.pdf},
  doi       = {10.18653/v1/P19-3002},
  software  = {https://www.jkk.name/slate},
  poster    = {https://www.jkk.name/pub/acl19slate_poster.pdf},
}
        

In a corpus of data, outliers are either errors: mistakes in the data that are counterproductive, or are unique: informative samples that improve model robustness. Identifying outliers can lead to better datasets by (1) removing noise in datasets and (2) guiding collection of additional data to fill gaps. However, the problem of detecting both outlier types has received relatively little attention in NLP, particularly for dialog systems. We introduce a simple and effective technique for detecting both erroneous and unique samples in a corpus of short texts using neural sentence embeddings combined with distance-based outlier detection. We also present a novel data collection pipeline built atop our detection technique to automatically and iteratively mine unique data samples while discarding erroneous samples. Experiments show that our outlier detection technique is effective at finding errors while our data collection pipeline yields highly diverse corpora that in turn produce more robust intent classification and slot-filling models.

        @InProceedings{naacl19outliers,
  title     = {Outlier Detection for Improved Data Quality and Diversity in Dialog Systems},
  author    = {Larson, Stefan and Mahendran, Anish and Lee, Andrew and Kummerfeld, Jonathan K. and Hill, Parker and Laurenzano, Michael and Hauswald, Johann and Tang, Lingjia and Mars, Jason},
  year      = {2019},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  pages     = {517--527},
  data      = {https://github.com/clinc/uniqueness},
  month     = {June},
  arxiv     = {https://arxiv.org/abs/1904.03122},
  url       = {https://www.aclweb.org/anthology/N19-1051.pdf},
  doi       = {10.18653/v1/N19-1051},
  location  = {Minneapolis, Minnesota},
}
        

We examine a large dialog corpus obtained from the conversation history of a single individual with 104 conversation partners. The corpus consists of half a million instant messages, across several messaging platforms. We focus our analyses on seven speaker attributes, each of which partitions the set of speakers, namely: gender; relative age; family member; romantic partner; classmate; co-worker; and native to the same country. In addition to the content of the messages, we examine conversational aspects such as the time messages are sent, messaging frequency, psycholinguistic word categories, linguistic mirroring, and graph-based features reflecting how people in the corpus mention each other. We present two sets of experiments predicting each attribute using (1) short context windows; and (2) a larger set of messages. We find that using all features leads to gains of 9-14% over using message text only.

        @InProceedings{cicling19personal,
  title     = {Look Who's Talking: Inferring Speaker Attributes from Personal Longitudinal Dialog},
  year      = {2019},
  author    = {Welch, Charles and P{\\'e}rez-Rosas, Ver{\\'o}nica and Kummerfeld, Jonathan K. and Mihalcea, Rada},
  address   = {La Rochelle, France},
  booktitle = {Proceedings of the 20th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing)},
  publisher = {Springer},
  month     = {April},
  awards    = {Best Student Paper Award},
  url       = {https://www.jkk.name/pub/cicling19personal.pdf},
  arxiv     = {https://arxiv.org/abs/1904.11610},
}
        

We explore the use of longitudinal dialog data for two dialog prediction tasks: next message prediction and response time prediction. We show that a neural model using personal data that leverages a combination of message content, style matching, time features, and speaker attributes leads to the best results for both tasks, with error rate reductions of up to 15% compared to a classifier that relies exclusively on message content and to a classifier that does not use personal data.

        @Article{ieee19personal,
  author    = {Welch, Charles and P{\\'e}rez-Rosas, Ver{\\'o}nica and Kummerfeld, Jonathan K. and Mihalcea, Rada},
  editor    = {Erik Cambria},
  title     = {Learning from Personal Longitudinal Dialog Data},
  year      = {2019},
  journal   = {IEEE Intelligent Systems},
  volume    = {34},
  number    = {4},
  pages     = {16--23},
  url       = {https://www.jkk.name/pub/ieee19personal.pdf},
}
        

To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.

        @InProceedings{acl18sql,
  author    = {Finegan-Dollak\\*, Catherine and Kummerfeld\\*, Jonathan K. and Zhang, Li and Ramanathan, Karthik and Sadasivam, Sesh and Zhang, Rui and Radev, Dragomir},
  title     = {Improving Text-to-SQL Evaluation Methodology},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  month     = {July},
  year      = {2018},
  address   = {Melbourne, Victoria, Australia},
  pages     = {351--360},
  doi       = {10.18653/v1/P18-1033},
  url       = {https://aclanthology.org/P18-1033.pdf},
  arxiv     = {https://arxiv.org/abs/1806.09029},
  software  = {https://www.jkk.name/text2sql-data},
  data      = {https://www.jkk.name/text2sql-data},
  poster    = {https://www.aclweb.org/anthology/attachments/P18-1033.Poster.pdf},
}
        

Despite the recent popularity of word embedding methods, there is only a small body of work exploring the limitations of these representations. In this paper, we consider one aspect of embedding spaces, namely their stability. We show that even relatively high frequency words (100-200 occurrences) are often unstable. We provide empirical evidence for how various factors contribute to the stability of word embeddings, and we analyze the effects of stability on downstream tasks.

        @InProceedings{naacl18embeddings,
  author    = {Burdick, Laura and Kummerfeld, Jonathan K. and Mihalcea, Rada},
  title     = {Factors Influencing the Surprising Instability of Word Embeddings},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year      = {2018},
  month     = {June},
  location  = {New Orleans, Louisiana, USA},
  url       = {https://aclanthology.org/N18-1190.pdf},
  pages     = {2092--2102},
  arxiv     = {https://arxiv.org/abs/1804.09692},
  doi       = {10.18653/v1/N18-1190},
}
        

Most summarization research focuses on summarizing the entire given text, but in practice readers are often interested in only one aspect of the document or conversation. We propose 'targeted summarization' as an umbrella category for summarization tasks that intentionally consider only parts of the input data. This covers query-based summarization, update summarization, and a new task we propose where the goal is to summarize a particular aspect of a document. However, collecting data for this new task is hard because directly asking annotators (e.g., crowd workers) to write summaries leads to data with low accuracy when there are a large number of facts to include. We introduce a novel crowdsourcing workflow, Pin-Refine, that allows us to collect highquality summaries for our task, a necessary step for the development of automatic systems.

        @InProceedings{naacl18summary,
  author    = {Jiang, Youxuan and Finegan-Dollak, Catherine and Kummerfeld, Jonathan K. and Lasecki, Walter},
  title     = {Effective Crowdsourcing for a New Type of Summarization Task},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  pages     = {628--633},
  year      = {2018},
  month     = {June},
  location  = {New Orleans, Louisiana, USA},
  url       = {https://aclanthology.org/N18-2099.pdf},
  doi       = {10.18653/v1/N18-2099},
}
        

Industrial dialogue systems such as Apple Siri and Google Assistant require large scale diverse training data to enable their sophisticated conversation capabilities. Crowdsourcing is a scalable and inexpensive data collection method, but collecting high quality data efficiently requires thoughtful orchestration of crowdsourcing jobs. Prior study of data collection process has focused on tasks with limited scope and performed intrinsic data analysis, which may not be indicative of impact on trained model performance. In this paper, we present a study of crowdsourcing methods for a user intent classification task in one of our deployed dialogue systems. Our task requires classification over 47 possible user intents and contains many intent pairs with subtle differences. We consider different crowdsourcing job types and job prompts, quantitatively analyzing the quality of collected data and downstream model performance on a test set of real user queries from production logs. Our observations provide insight into how design decisions impact crowdsourced data quality, with clear recommendations for future data collection for dialogue systems.

        @InProceedings{naacl18data,
  author    = {Kang, Yiping and Zhang, Yunqi and Kummerfeld, Jonathan K. and Hill, Parker and Hauswald, Johann and Laurenzano, Michael A. and Tang, Lingjia and Mars, Jason},
  doi       = {10.18653/v1/N18-3005},
  title     = {Data Collection for a Production Dialogue System: A Startup Perspective},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)},
  pages     = {33--40},
  year      = {2018},
  month     = {June},
  location  = {New Orleans, Louisiana, USA},
  url       = {https://aclanthology.org/N18-3005.pdf},
  video     = {https://vimeo.com/277631102},
}
        

In this paper we explore the role played by world knowledge in semantic parsing. We look at the types of errors that currently exist in a state-of-the-art Abstract Meaning Representation (AMR) parser, and explore the problem of how to integrate world knowledge to reduce these errors. We look at three types of knowledge from (1) WordNet hypernyms and super senses, (2) Wikipedia entity links, and (3) retraining a named entity recognizer to identify concepts in AMR. The retrained entity recognizer is not perfect and cannot recognize all concepts in AMR and we examine the limitations of the named entity features using a set of oracles. The oracles show how performance increases if it can recognize different subsets of AMR concepts. These results show improvement on multiple fine-grained metrics, including a 6% increase in named entity F-score, and provide insight into the potential of world knowledge for future work in Abstract Meaning Representation parsing.

        @InProceedings{lrec18amr,
  author    = {Welch, Charles and Kummerfeld, Jonathan K. and Feng, Song and Mihalcea, Rada},
  title     = {World Knowledge for Abstract Meaning Representation Parsing},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year      = {2018},
  month     = {May},
  location  = {Miyazaki, Japan},
  url       = {http://www.lrec-conf.org/proceedings/lrec2018/pdf/1085.pdf},
}
        

General treebank analyses are graph structured, but parsers are typically restricted to tree structures for efficiency and modeling reasons. We propose a new representation and algorithm for a class of graph structures that is flexible enough to cover almost all treebank structures, while still admitting efficient learning and inference. In particular, we consider directed, acyclic, one-endpoint-crossing graph structures, which cover most long-distance dislocation, shared argumentation, and similar tree-violating linguistic phenomena. We describe how to convert phrase structure parses, including traces, to our new representation in a reversible manner. Our dynamic program uniquely decomposes structures, is sound and complete, and covers 97.3% of the Penn English Treebank. We also implement a proofof-concept parser that recovers a range of null elements and trace types.

        @InProceedings{tacl17parsing,
  author    = {Kummerfeld, Jonathan K. and Klein, Dan},
  title     = {Parsing with Traces: An O($n^4$) Algorithm and a Structural Representation},
  booktitle = {Transactions of the Association for Computational Linguistics},
  doi       = {10.1162/tacl_a_00072},
  volume    = {5},
  issn      = {2307-387X},
  pages     = {441--454},
  year      = {2017},
  url       = {https://aclanthology.org/Q17-1031.pdf},
  arxiv     = {https://arxiv.org/abs/1707.04221},
  software  = {https://www.jkk.name/1ec-graph-parser/},
  interview = {https://soundcloud.com/nlp-highlights/46-parsing-with-traces-with-jonathan-kummerfeld},
  video     = {https://vimeo.com/238235203},
}
        

One weakness of machine-learned NLP models is that they typically perform poorly on out-of-domain data. In this work, we study the task of identifying products being bought and sold in online cybercrime forums, which exhibits particularly challenging cross-domain effects. We formulate a task that represents a hybrid of slot-filling information extraction and named entity recognition and annotate data from four different forums. Each of these forums constitutes its own 'fine-grained domain' in that the forums cover different market sectors with different properties, even though all forums are in the broad domain of cybercrime. We characterize these domain differences in the context of a learning-based system: supervised models see decreased accuracy when applied to new forums, and standard techniques for semi-supervised learning and domain adaptation have limited effectiveness on this data, which suggests the need to improve these techniques. We release a dataset of 1,938 annotated posts from across the four forums.

        @InProceedings{emnlp17forums,
  author    = {Durrett, Greg and Kummerfeld, Jonathan K. and Berg-Kirkpatrick, Taylor and Portnoff, Rebecca S. and Afroz, Sadia and McCoy, Damon and Levchenko, Kirill and Paxson, Vern},
  title     = {Identifying Products in Online Cybercrime Marketplaces: A Dataset for Fine-grained Domain Adaptation},
  arxiv     = {https://arxiv.org/abs/1708.09609},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  doi       = {10.18653/v1/D17-1275},
  year      = {2017},
  month     = {September},
  location  = {Copenhagen, Denmark},
  url       = {https://www.aclweb.org/anthology/D17-1275.pdf},
  pages     = {2588--2597},
  software  = {https://evidencebasedsecurity.org/forums/},
  supplementary = {https://www.aclweb.org/anthology/attachments/D17-1275.Attachment.zip},
}
        

Linguistically diverse datasets are critical for training and evaluating robust machine learning systems, but data collection is a costly process that often requires experts. Crowdsourcing the process of paraphrase generation is an effective means of expanding natural language datasets, but there has been limited analysis of the trade-offs that arise when designing tasks. In this paper, we present the first systematic study of the key factors in crowdsourcing paraphrase collection. We consider variations in instructions, incentives, data domains, and workflows. We manually analyzed paraphrases for correctness, grammaticality, and linguistic diversity. Our observations provide new insight into the trade-offs between accuracy and diversity in crowd responses that arise as a result of task design, providing guidance for future paraphrase generation procedures.

        @InProceedings{acl17paraphrase,
  author    = {Jiang, Youxuan and Kummerfeld, Jonathan K. and Lasecki, Walter S.},
  doi       = {10.18653/v1/P17-2017},
  title     = {Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  month     = {July},
  year      = {2017},
  location  = {Vancouver, Canada},
  pages     = {103--109},
  url       = {https://aclanthology.org/P17-2017.pdf},
  data      = {https://www.aclweb.org/anthology/attachments/P17-2017.Datasets.zip},
  slidespdf = {https://www.aclweb.org/anthology/attachments/P17-2017.Presentation.pdf},
  video     = {https://vimeo.com/234958413},
  arxiv     = {https://arxiv.org/abs/1704.05753},
}
        

Underground forums are widely used by criminals to buy and sell a host of stolen items, datasets, resources, and criminal services. These forums contain important resources for understanding cybercrime. However, the number of forums, their size, and the domain expertise required to understand the markets makes manual exploration of these forums unscalable. In this work, we propose an automated, top-down approach for analyzing underground forums. Our approach uses natural language processing and machine learning to automatically generate high-level information about underground forums, first identifying posts related to transactions, and then extracting products and prices. We also demonstrate, via a pair of case studies, how an analyst can use these automated approaches to investigate other categories of products and transactions. We use eight distinct forums to assess our tools: Antichat, Blackhat World, Carders, Darkode, Hack Forums, Hell, L33tCrew and Nulled. Our automated approach is fast and accurate, achieving over 80% accuracy in detecting post category, product, and prices.

        @InProceedings{www17forums,
  title     = {Tools for Automated Analysis of Cybercriminal Markets},
  author    = {Portnoff, Rebecca S. and Afroz, Sadia and Durrett, Greg and Kummerfeld, Jonathan K. and Berg-Kirkpatrick, Taylor and McCoy, Damon and Levchenko, Kirill and Paxson, Vern},
  booktitle = {Proceedings of 26th International World Wide Web conference},
  pages     = {657--666},
  month     = {April},
  year      = {2017},
  location  = {Perth, Australia},
  url       = {https://www.jkk.name/pub/www17forums.pdf},
  software  = {http://evidencebasedsecurity.org/forums/},
}
        

Representation of syntactic structure is a core area of research in Computational Linguistics, disambiguating distinctions in meaning that are crucial for correct interpretation of language. Development of algorithms and statistical models over the past three decades has led to systems that are accurate enough to be deployed in industry, playing a key role in products such as Google Search and Apple Siri. However, syntactic parsers today are usually constrained to tree representations of language, and performance is interpreted through a single metric that conveys no linguistic information regarding remaining errors.\n\nIn this dissertation, we present new algorithms for error analysis and parsing. The heart of our approach to error analysis is the use of structural transformations to identify more meaningful classes of errors, and to enable comparisons across formalisms. For parsing, we combine a novel dynamic program with careful choices in syntactic representation to create an efficient parser that produces graph structured output. Together, these developments allowed us to evaluate the outstanding challenges in parsing and to address a key weakness in current work.\n\nFirst, we present a search algorithm that, given two structures, finds a sequence of modifications leading from one structure to the other. We applied this algorithm to syntactic error analysis, where one structure is the output of a parser, the other is the correct parse, and each modification corresponds to fixing one error. We constructed a tool based on the algorithm and analyzed variations in behavior between parsers, types of text, and languages. Our observations shine light on several assumptions about syntactic errors, showing some to be true and others to be false. For example, prepositional phrase attachment errors are indeed a major issue, while coordination scope errors do not hurt performance as much as expected.\n\nNext, we describe an algorithm that builds a parse in one syntactic representation to match a parse in another representation. Specifically, we build phrase structure parses from Combinatory Categorial Grammar derivations. Our approach follows the philosophy of CCG, defining specific phrase structures for each lexical category and generic rules for combinatory steps. The new parse is built by following the CCG derivation bottom-up, gradually building the corresponding phrase structure parse. This produced significantly more accurate parses than past work, and enabled us to compare performance of several parsers across formalisms.\n\nFinally, we address a weakness we observed in phrase structure parsers: the exclusion of syntactic trace structures for computational convenience. We present an efficient dynamic programming algorithm that constructs the graph structure that has the highest score under an edge-factored scoring function. We define a parse representation compatible with the algorithm, and show how certain linguistic distinctions dramatically impact coverage. We also show various ways to modify the algorithm to improve performance by exploiting properties of observed linguistic structure. This approach to syntactic parsing is the first to cover virtually all structure encoded in the Penn Treebank.

        @PhDThesis{thesis16parsing,
  title     = {Algorithms for Identifying Syntactic Errors and Parsing with Graph Structured Output},
  author    = {Kummerfeld, Jonathan K.},
  year      = {2016},
  month     = {Aug},
  location  = {Berkeley, CA, USA},
  school    = {EECS Department, University of California, Berkeley},
  number    = {UCB/EECS-2016-138},
  url       = {https://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-138.html},

  In this dissertation, we present new algorithms for error analysis and parsing. The heart of our approach to error analysis is the use of structural transformations to identify more meaningful classes of errors, and to enable comparisons across formalisms. For parsing, we combine a novel dynamic program with careful choices in syntactic representation to create an efficient parser that produces graph structured output. Together, these developments allowed us to evaluate the outstanding challenges in parsing and to address a key weakness in current work.

  First, we present a search algorithm that, given two structures, finds a sequence of modifications leading from one structure to the other. We applied this algorithm to syntactic error analysis, where one structure is the output of a parser, the other is the correct parse, and each modification corresponds to fixing one error. We constructed a tool based on the algorithm and analyzed variations in behavior between parsers, types of text, and languages. Our observations shine light on several assumptions about syntactic errors, showing some to be true and others to be false. For example, prepositional phrase attachment errors are indeed a major issue, while coordination scope errors do not hurt performance as much as expected.

  Next, we describe an algorithm that builds a parse in one syntactic representation to match a parse in another representation. Specifically, we build phrase structure parses from Combinatory Categorial Grammar derivations. Our approach follows the philosophy of CCG, defining specific phrase structures for each lexical category and generic rules for combinatory steps. The new parse is built by following the CCG derivation bottom-up, gradually building the corresponding phrase structure parse. This produced significantly more accurate parses than past work, and enabled us to compare performance of several parsers across formalisms.

  Finally, we address a weakness we observed in phrase structure parsers: the exclusion of syntactic trace structures for computational convenience. We present an efficient dynamic programming algorithm that constructs the graph structure that has the highest score under an edge-factored scoring function. We define a parse representation compatible with the algorithm, and show how certain linguistic distinctions dramatically impact coverage. We also show various ways to modify the algorithm to improve performance by exploiting properties of observed linguistic structure. This approach to syntactic parsing is the first to cover virtually all structure encoded in the Penn Treebank.},
}
        

Despite the convexity of structured max-margin objectives (Taskar et al., 2004; Tsochantaridis et al., 2004), the many ways to optimize them are not equally effective in practice. We compare a range of online optimization methods over a variety of structured NLP tasks (coreference, summarization, parsing, etc) and find several broad trends. First, margin methods do tend to outperform both likelihood and the perceptron. Second, for max-margin objectives, primal optimization methods are often more robust and progress faster than dual methods. This advantage is most pronounced for tasks with dense or continuous-valued features. Overall, we argue for a particularly simple online primal subgradient descent method that, despite being rarely mentioned in the literature, is surprisingly effective in relation to its alternatives.

        @InProceedings{emnlp15learn,
  title     = {An Empirical Analysis of Optimization for Max-Margin NLP},
  author    = {Kummerfeld, Jonathan K. and Berg-Kirkpatrick, Taylor and Klein, Dan},
  doi       = {10.18653/v1/D15-1032},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  month     = {September},
  year      = {2015},
  location  = {Lisbon, Portugal},
  pages     = {273--279},
  url       = {https://www.jkk.name/pub/emnlp15learn.pdf},
  poster    = {https://www.jkk.name/pub/emnlp15learn_poster.png},
  software  = {https://github.com/tberg12/murphy},
}
        

Coreference resolution metrics quantify errors but do not analyze them. Here, we consider an automated method of categorizing errors in the output of a coreference system into intuitive underlying error types. Using this tool, we first compare the error distributions across a large set of systems, then analyze common errors across the top ten systems, empirically characterizing the major unsolved challenges of the coreference resolution task.

        @InProceedings{emnlp13analysis,
  title     = {Error-Driven Analysis of Challenges in Coreference Resolution},
  author    = {Kummerfeld, Jonathan K. and Klein, Dan},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  month     = {October},
  year      = {2013},
  location  = {Seattle, Washington, USA},
  pages     = {265--277},
  software  = {https://www.jkk.name/berkeley-coreference-analyser/},
  url       = {https://aclanthology.org/D13-1027.pdf},
  slides    = {https://www.jkk.name/pub/emnlp13analysis_keynote.key},
  slidespdf = {https://www.jkk.name/pub/emnlp13analysis_slides.pdf},
}
        

Aspects of Chinese syntax result in a distinctive mix of parsing challenges. However, the contribution of individual sources of error to overall difficulty is not well understood. We conduct a comprehensive automatic analysis of error types made by Chinese parsers, covering a broad range of error types for large sets of sentences, enabling the first empirical ranking of Chinese error types by their performance impact. We also investigate which error types are resolved by using gold part-of-speech tags, showing that improving Chinese tagging only addresses certain error types, leaving substantial outstanding challenges.

        @InProceedings{acl13analysis,
  title     = {An Empirical Examination of Challenges in Chinese Parsing},
  author    = {Kummerfeld, Jonathan K. and Tse, Daniel and Curran, James R. and Klein, Dan},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  location  = {Sofia, Bulgaria},
  pages     = {98--103},
  month     = {August},
  year      = {2013},
  software  = {https://www.jkk.name/berkeley-parser-analyser/},
  url       = {https://aclanthology.org/P13-2018.pdf},
  slides    = {https://www.jkk.name/pub/acl13analysis_keynote.key},
  slidespdf = {https://www.jkk.name/pub/acl13analysis_slides.pdf},
}
        

We present a catalogue of high-velocity clouds (HVCs) from the Galactic All Sky Survey (GASS) of southern-sky neutral hydrogen, which has 57 mK sensitivity and 1 km/s velocity resolution and was obtained with the Parkes Telescope. Our catalogue has been derived from the stray-radiation corrected second release of GASS. We describe the data and our method of identifying HVCs and analyse the overall properties of the GASS population. We catalogue a total of 1693 HVCs at declinations < 0 deg, including 1111 positive velocity HVCs and 582 negative velocity HVCs. Our catalogue also includes 295 anomalous velocity clouds (AVCs). The cloud line-widths of our HVC population have a median FWHM of ~19 km/s, which is lower than found in previous surveys. The completeness of our catalogue is above 95% based on comparison with the HIPASS catalogue of HVCs, upon which we improve with an order of magnitude in spectral resolution. We find 758 new HVCs and AVCs with no HIPASS counterpart. The GASS catalogue will shed an unprecedented light on the distribution and kinematic structure of southern-sky HVCs, as well as delve further into the cloud populations that make up the anomalous velocity gas of the Milky Way.

        @Article{astro13clouds,
  title     = {High-velocity Clouds in the Galactic All Sky Survey. I. Catalog},
  author    = {Moss, Vanessa A. and McClure-Griffiths, Naomi M. and Murphy, Tara and Pisano, D. J. and Kummerfeld, Jonathan K. and Curran, James R.},
  volume    = {209},
  number    = {1},
  pages     = {12},
  publisher = {IOP Publishing},
  journal   = {The Astrophysical Journal Supplement Series},
  year      = {2013},
  url       = {http://iopscience.iop.org/0067-0049/209/1/12},
  arxiv     = {https://arxiv.org/abs/1309.4104},
}
        

Constituency parser performance is primarily interpreted through a single metric, F-score on WSJ section 23, that conveys no linguistic information regarding the remaining errors. We classify errors within a set of linguistically meaningful types using tree transformations that repair groups of errors together. We use this analysis to answer a range of questions about parser behaviour, including what linguistic constructions are difficult for state-of-the-art parsers, what types of errors are being resolved by rerankers, and what types are introduced when parsing out-of-domain text.

        @InProceedings{emnlp12analysis,
  title     = {Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output},
  author    = {Kummerfeld, Jonathan K. and Hall, David and Curran, James R. and Klein, Dan},
  booktitle = {Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning},
  year      = {2012},
  pages     = {1048--1059},
  month     = {July},
  location  = {Jeju Island, South Korea},
  software  = {https://www.jkk.name/berkeley-parser-analyser/},
  url       = {https://aclanthology.org/D12-1096.pdf},
  slides    = {https://www.jkk.name/pub/emnlp12analysis_keynote.key},
  slidespdf = {https://www.jkk.name/pub/emnlp12analysis_slides.pdf},
}
        

We propose an improved, bottom-up method for converting CCG derivations into PTB-style phrase structure trees. In contrast with past work (Clark and Curran, 2009), which used simple transductions on category pairs, our approach uses richer transductions attached to single categories. Our conversion preserves more sentences under round-trip conversion (51.1% vs. 39.6%) and is more robust. In particular, unlike past methods, ours does not require ad-hoc rules over non-local features, and so can be easily integrated into a parser.

        @InProceedings{acl12conversion,
  title     = {Robust Conversion of CCG Derivations to Phrase Structure Trees},
  author    = {Kummerfeld, Jonathan K. and Klein, Dan and Curran, James R.},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year      = {2012},
  pages     = {105--109},
  month     = {July},
  location  = {Jeju Island, South Korea},
  software  = {https://www.jkk.name/berkeley-ccg2pst/},
  url       = {https://aclanthology.org/P12-2021.pdf},
  slides    = {https://www.jkk.name/pub/acl12conversion_keynote.key},
  slidespdf = {https://www.jkk.name/pub/acl12conversion_slides.pdf},
}
        

Our submission was a reduced version of the system described in Haghighi and Klein (2010), with extensions to improve mention detection to suit the OntoNotes annotation scheme. Including exact matching mention detection in this shared task added a new and challenging dimension to the problem, particularly for our system, which previously used a very permissive detection method. We improved this aspect of the system by adding filters based on the annotation scheme for OntoNotes and analysis of system behavior on the development set. These changes led to improvements in coreference F-score of 10.06, 5.71, 6.78, 6.63 and 3.09 on the MUC, B3 , Ceaf-e, Ceaf-m and Blanc, metrics, respectively, and a final task score of 47.10.

        @InProceedings{conll11coreference,
  title     = {Mention Detection: Heuristics for the OntoNotes annotations},
  author    = {Kummerfeld, Jonathan K. and Bansal, Mohit and Burkett, David and Klein, Dan},
  booktitle = {Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task},
  year      = {2011},
  pages     = {102--106},
  month     = {June},
  location  = {Portland, Oregon, USA},
  url       = {https://aclanthology.org/W11-1916.pdf},
  poster    = {https://www.jkk.name/pub/conll11coreference_poster.pdf},
}
        

We propose a novel self-training method for a parser which uses a lexicalised grammar and supertagger, focusing on increasing the speed of the parser rather than its accuracy. The idea is to train the supertagger on large amounts of parser output, so that the supertagger can learn to supply the supertags that the parser will eventually choose as part of the highest scoring derivation. Since the supertagger supplies fewer supertags overall, the parsing speed is increased. We demonstrate the effectiveness of the method using a CCG supertagger and parser, obtaining significant speed increases on newspaper text with no loss in accuracy. We also show that the method can be used to adapt the CCG parser to new domains, obtaining accuracy and speed improvements for Wikipedia and biomedical text.

        @InProceedings{acl10adapt,
  title     = {Faster Parsing by Supertagger Adaptation},
  author    = {Kummerfeld, Jonathan K. and Roesner, Jessika and Dawborn, Tim and Haggerty, James and Curran, James R. and Clark, Stephen},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  month     = {July},
  year      = {2010},
  location  = {Uppsala, Sweden},
  pages     = {345--355},
  software  = {http://downloads.schwa.org/acl10adapt_fast_news_model.tar.bz2},
  slidespdf = {https://www.jkk.name/pub/acl10adapt_slides.pdf},
  url       = {https://aclanthology.org/P10-1036.pdf},
}
        

We identify the pattern of microscopic dynamical relaxation for a two-dimensional glass-forming liquid. On short time scales, bursts of irreversible particle motion, called cage jumps, aggregate into clusters. On larger time scales, clusters aggregate both spatially and temporally into avalanches. This propagation of mobility takes place along the soft regions of the systems, which have been identified by computing isoconfigurational Debye-Waller maps. Our results characterize the way in which dynamical heterogeneity evolves in moderately supercooled liquids and reveal that it is astonishingly similar to the one found for dense glassy granular media.

        @Article{prl10chemistry,
  title     = {Spatiotemporal Hierarchy of Relaxation Events, Dynamical Heterogeneities, and Structural Reorganization in a Supercooled Liquid},
  author    = {Candelier, Raphael and Widmer-Cooper, Asaph and Kummerfeld, Jonathan K. and Dauchot, Olivier and Biroli, Giulio and Harrowell, Peter and Reichman, David R.},
  journal   = {Physical Review Letters},
  volume    = {105},
  number    = {13},
  pages     = {135702},
  doi       = {10.1103/PhysRevLett.105.135702},
  numpages  = {4},
  year      = {2010},
  month     = {September},
  publisher = {American Physical Society},
  url       = {http://prl.aps.org/abstract/PRL/v105/i13/e135702},
  arxiv     = {https://arxiv.org/abs/0912.0193},
}
        

Because English is a low morphology language, current statistical parsers tend to ignore morphology and accept some level of redundancy. This paper investigates how costly such redundancy is for a lexicalised grammar such as CCG.\n\nWe use morphological analysis to split verb inflectional suffixes into separate tokens, so that they can receive their own lexical categories. We find that this improves accuracy when the splits are based on correct POS tags, but that errors in gold standard or automatically assigned POS tags are costly for the system. This shows that the parser can benefit from morphological analysis, so long as the analysis is correct.

        @InProceedings{coling10morph,
  title     = {Morphological Analysis Can Improve a CCG Parser for English},
  author    = {Honnibal, Matthew and Kummerfeld, Jonathan K. and Curran, James R.},
  booktitle = {Proceedings of the 23rd International Conference on Computational Linguistics},
  year      = {2010},
  pages     = {445--453},
  location  = {Beijing, China},
  month     = {August},
  url       = {https://aclanthology.org/C10-2051.pdf},

We use morphological analysis to split verb inflectional suffixes into separate tokens, so that they can receive their own lexical categories. We find that this improves accuracy when the splits are based on correct POS tags, but that errors in gold standard or automatically assigned POS tags are costly for the system. This shows that the parser can benefit from morphological analysis, so long as the analysis is correct.},
}
        

Statistical parsers are crucial for tackling the grand challenges of Natural Language Processing. The most effective approaches to these tasks are data driven, but parsers are too slow to be effectively used on large data sets. State-of-the-art parsers generally cannot process more than one sentence a second, and the fastest cannot process more than fifty sentences a second. The situation is even worse when they are applied outside of the domain of their training data. The fastest systems have two components, a parser, which has time complexity O(n3) and a supertagger, which has linear time complexity. By shifting work from the parser to the supertagger we dramatically improve speed.\n\nThis work demonstrates several major novel ideas that improve parsing efficiency. The core idea is that the tags chosen by the parser are gold standard data for its supertagger. This leads to the second surprising conceptual development, that decreasing tagging accuracy can improve parsing performance. To demonstrate these ideas required extensive development of the C&C supertagger, including imple- mentation of more efficient estimation algorithms and parallelisation of the training process. This was particularly challenging as the C&C supertagger is a state-of-the-art high performance system designed with a focus on speed rather than flexibility.\n\nI was able to significantly improve performance on the standard evaluation corpus by using the parser to generate extremely large new resources for supertagger training. I have also shown that these methods provide significant benefits on another domain, Wikipedia text, without the cost of generating human annotated data sets. These parsing performance gains occur while supertagging accuracy decreases.\n\nDespite extensive use of supertaggers to improve parsing efficiency there has been no comprehensive study of the interaction between a supertagger and a parser. I present the first systematic exploration of the relationship, show the potential benefits of understanding it, and demonstrate a novel algorithm for optimising the parameters that define it.\n\nI have constructed models that process newspaper text 86% faster than previously, and Wikipedia text 30% faster, without any loss in accuracy and without the aid of extra gold standard resources in either domain. This work will lead directly to improvements in a range of Natural Language Processing tasks by enabling the use of far more parsed data.

        @PhDThesis{thesis09adapt,
  title     = {Adaptive Supertagging for Faster Parsing},
  author    = {Kummerfeld, Jonathan K.},
  school    = {The University of Sydney},
  year      = {2009},
  location  = {Sydney, Australia},
  url       = {https://www.jkk.name/pub/thesis09adapt_thesis.pdf},
  poster    = {https://www.jkk.name/pub/thesis09adapt_poster.pdf},
  slidespdf = {https://www.jkk.name/pub/thesis09adapt_slides.pdf},

  This work demonstrates several major novel ideas that improve parsing efficiency. The core idea is that the tags chosen by the parser are gold standard data for its supertagger. This leads to the second surprising conceptual development, that decreasing tagging accuracy can improve parsing performance. To demonstrate these ideas required extensive development of the C&C supertagger, including imple- mentation of more efficient estimation algorithms and parallelisation of the training process. This was particularly challenging as the C&C supertagger is a state-of-the-art high performance system designed with a focus on speed rather than flexibility.

  I was able to significantly improve performance on the standard evaluation corpus by using the parser to generate extremely large new resources for supertagger training. I have also shown that these methods provide significant benefits on another domain, Wikipedia text, without the cost of generating human annotated data sets. These parsing performance gains occur while supertagging accuracy decreases.

  Despite extensive use of supertaggers to improve parsing efficiency there has been no comprehensive study of the interaction between a supertagger and a parser. I present the first systematic exploration of the relationship, show the potential benefits of understanding it, and demonstrate a novel algorithm for optimising the parameters that define it.

  I have constructed models that process newspaper text 86% faster than previously, and Wikipedia text 30% faster, without any loss in accuracy and without the aid of extra gold standard resources in either domain. This work will lead directly to improvements in a range of Natural Language Processing tasks by enabling the use of far more parsed data.},
}
        

Parsers are often the bottleneck for data acquisition, processing text too slowly to be widely applied. One way to improve the efficiency of parsers is to construct more confident statistical models. More training data would enable the use of more sophisticated features and also provide more evidence for current features, but gold standard annotated data is limited and expensive to produce.\n\nWe demonstrate faster methods for training a supertagger using hundreds of millions of automatically annotated words, constructing statistical models that further constrain the number of derivations the parser must consider. By introducing new features and using an automatically annotated corpus we are able to double parsing speed on Wikipedia and the Wall Street Journal, and gain accuracy slightly when parsing Section 00 of the Wall Street Journal.

        @InProceedings{alta09tagging,
  title     = {Faster parsing and supertagging model estimation},
  author    = {Kummerfeld, Jonathan K. and Roesner, Jessika and Curran, James R.},
  booktitle = {Proceedings of the Australasian Language Technology Association Workshop 2009},
  year      = {2009},
  pages     = {62--70},
  location  = {Sydney, Australia},
  month     = {December},
  url       = {https://aclanthology.org/U09-1009.pdf},
  slidespdf = {https://www.jkk.name/pub/alta09tagging_slides.pdf},

We demonstrate faster methods for training a supertagger using hundreds of millions of automatically annotated words, constructing statistical models that further constrain the number of derivations the parser must consider. By introducing new features and using an automatically annotated corpus we are able to double parsing speed on Wikipedia and the Wall Street Journal, and gain accuracy slightly when parsing Section 00 of the Wall Street Journal.},
}
        

This paper considers the homogeneous packing of binary hard spheres in an equimolar stoichiometry, and postulates the densest packing at each sphere size ratio. Monte Carlo simulated annealing optimizations are seeded with all known atomic inorganic crystal structures, and the search is performed within the degrees of freedom associated with each homogeneous AB structure type. Structures isopointal to the FeB structure type are found to have the highest packing fraction at all sphere size ratios. The optimized structures match or improve on the best previously demonstrated packings of this type, and show that compound structures can pack more densely than segregated close-packed structures at all radius ratios less than 0.62.

        @Article{chem08packing,
  title     = {The densest packing of AB binary hard-sphere homogeneous compounds across all size ratios},
  author    = {Kummerfeld, Jonathan K. and Hudson, Toby S and Harrowell, Peter},
  journal   = {The Journal of Physical Chemistry B},
  month     = {August},
  year      = {2008},
  volume    = {112},
  issue     = {35},
  pages     = {10773--10776},
  url       = {http://pubs.acs.org/doi/abs/10.1021/jp804953r},
}
        

Manually maintaining comprehensive databases of multi-word expressions, for example Verb-Particle Constructions (VPCs), is infeasible. We describe a new classifier for potential VPCs, which uses information in the Google Web1T corpus to perform a simple linguistic constituency test. Specifically, we consider the fronting test, comparing the frequencies of the two possible orderings of the given verb and particle. Using only a small set of queries for each verb-particle pair, the system was able to achieve an F-score of 78.4% in our evaluation while processing thousands of queries a second.

        @InProceedings{alta08vpc,
  title     = {Classification of Verb Particle Constructions with the Google Web1T Corpus},
  author    = {Kummerfeld, Jonathan K. and Curran, James R.},
  booktitle = {Proceedings of the Australasian Language Technology Association Workshop 2008},
  month     = {December},
  year      = {2008},
  location  = {Hobart, Australia},
  pages     = {55--63},
  volume    = {6},
  url       = {https://aclanthology.org/U08-1008.pdf},
  poster    = {https://www.jkk.name/pub/alta08vpc_poster.eps},
}
        

AI is powerful, but it can make choices that result in objective errors, contextually inappropriate outputs, and disliked options. We need AI-resilient interfaces that help people be resilient to the AI choices that are not right, or not right for them. To support this goal, interfaces need to help users notice and have the context to appropriately judge those AI choices. Existing human-AI interaction guidelines recommend efficient user dismissal, modification, or otherwise efficient recovery from AI choices that a user does not like. However, in order to recover from AI choices, the user must notice them first. This can be difficult. For example, when generating summaries of long documents, a system's exclusion of a detail that is critically important to the user is hard for the user to notice. That detail can be hiding in a wall of text in the original document, and the existence of a summary may tempt the user not to read the original document as carefully. Once noticed, judging AI choices well can also be challenging. The interface may provide very little information that contextualizes the choices, and the user may fall back on assumptions when deciding whether to dismiss, modify, or otherwise recover from an AI choice. Building on prior work, this paper defines key aspects of AI-resilient interfaces, illustrated with examples. Designing interfaces for increased AI-resilience of users will improve AI safety, usability, and utility. This is especially critical where AI-powered systems are used for context- and preference-dominated open-ended AI-assisted tasks, like ideating, summarizing, searching, sensemaking, and the reading and writing of text or code.

        @misc{position24airesilientinterfaces,
  title     = {AI-Resilient Interfaces}, 
  author    = {Elena L. Glassman and Ziwei Gu and Jonathan K. Kummerfeld},
  year      = {2024},
  eprint    = {2405.08447},
  archivePrefix = {arXiv},
  primaryClass = {cs.HC},
  url       = {https://arxiv.org/abs/2405.08447}, 
}
        

The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.

        @misc{dagstuhl24,
  title     = {What Can Natural Language Processing Do for Peer Review?},
  author    = {Ilia Kuznetsov and Osama Mohammed Afzal and Koen Dercksen and Nils Dycke and Alexander Goldberg and Tom Hope and Dirk Hovy and Jonathan K. Kummerfeld and Anne Lauscher and Kevin Leyton-Brown and Sheng Lu and Mausam and Margot Mieskes and Aurélie Névéol and Danish Pruthi and Lizhen Qu and Roy Schwartz and Noah A. Smith and Thamar Solorio and Jingyan Wang and Xiaodan Zhu and Anna Rogers and Nihar B. Shah and Iryna Gurevych},
  year      = {2024},
  eprint    = {2405.06563},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url       = {https://arxiv.org/abs/2405.06563},
}
        

Ensuring a language model doesn't generate problematic text is difficult. Traditional evaluation methods, like automatic measures or human annotation, can fail to detect all problems, whether because system designers were not aware of a kind of problem they should attempt to detect, or because an automatic measure fails to reliably detect certain kinds of problems. In this paper we propose sensemaking tools as a robust and open-ended method to evaluate the large number of linguistic outputs produced by a language model. We demonstrate one potential sensemaking interface based on concordance tables, showing that we are able to detect problematic outputs and distributional shifts in minutes, despite not knowing exactly what kind of problems to look for.

        @InProceedings{ws-neurips-hegm22lm,
  title     = {Sensemaking Interfaces for Human Evaluation of Language Model Outputs},
  author    = {Katy Gero and Jonathan K. Kummerfeld and Elena Glassman},
  year      = {2022},
  booktitle = {NeurIPS Workshop: Human Evaluation of Generative Models},
  month     = {December},
  url       = {https://www.jkk.name/pub/ws23sensemaking.pdf},
}
        

Real-world conversation often involves more than two participants and complex conversation structures, but most datasets for dialogue research simplify the task to make it more tractable. This shared task built on prior tasks for goal-oriented dialogue, moving towards more realistic settings. Seventeen teams participated in the primary task, predicting the next utterance in a multi-party conversation, and several teams participated in supplementary tasks. All of the datasets have been publicly released, providing a standard benchmark for future work in this space.

        @InProceedings{ws-aaai-dstc20task2,
  title     = {NOESIS II: Predicting Responses, Identifying Success, and Managing Complexity in Task-Oriented Dialogue},
  author    = {Gunasekara, Chulaka and Kummerfeld, Jonathan K. and Lastras, Luis and Lasecki, Walter S.},
  year      = {2020},
  booktitle = {8th Edition of the Dialog System Technology Challenges at AAAI 2019},
  month     = {January},
  url       = {https://www.jkk.name/pub/dstc20task2.pdf},
  data      = {https://github.com/dstc8-track2/NOESIS-II/},
}
        

Extensive work has argued in favour of paying crowd workers a wage that is at least equivalent to the U.S.~federal minimum wage. Meanwhile, research on collecting high quality annotations (e.g.~for Natural Language Processing) suggests using qualifications such as a minimum number of previously completed tasks. If most requesters who pay fairly use this kind of minimum qualification, then workers may be forced to complete a substantial amount of poorly paid work for other requesters before they can earn a fair wage. This paper (1) explores current conventions for the threshold, (2) discusses possible alternatives, and (3) presents a study of correlation between approved work and work quality.

        @InProceedings{hcomp20fair,
  title     = {Qualification Labour: A Fair Wage Isn't Enough if Workers Need to Do 5,000 Low Paid Tasks to Qualify for Your Task},
  author    = {Kummerfeld, Jonathan K.},
  year      = {2020},
  month     = {October},
  booktitle = {The eighth AAAI Conference on Human Computation and Crowdsourcing},
  url       = {https://www.jkk.name/pub/hcomp20fair.pdf},
}
        

Goal-oriented dialogue in complex domains is an extremely challenging problem and there are relatively few datasets. This task provided two new resources that presented different challenges: one was focused but small, while the other was large but diverse. We also considered several new variations on the next utterance selection problem: (1) increasing the number of candidates, (2) including paraphrases, and (3) not including a correct option in the candidate set. Twenty teams participated, developing a range of neural network models, including some that successfully incorporated external data to boost performance. Both datasets have been publicly released, enabling future work to build on these results, working towards robust goal-oriented dialogue systems.

        @InProceedings{ws-aaai-dstc19task1,
  title     = {DSTC7 Task 1: Noetic End-to-End Response Selection},
  author    = {Gunasekara, Chulaka and Kummerfeld, Jonathan K. and Polymenakos, Lazaros and Lasecki, Walter S.},
  year      = {2019},
  booktitle = {7th Edition of the Dialog System Technology Challenges at AAAI 2019},
  month     = {January},
  url       = {https://www.jkk.name/pub/ws18dstc_task1.pdf},
  alt-url   = {http://workshop.colips.org/dstc7/papers/dstc7_task1_final_report.pdf},
  data      = {https://ibm.github.io/dstc-noesis/public/index.html},
}
        

This paper introduces the Eighth Dialog System Technology Challenge. In line with recent challenges, the eighth edition focuses on applying end-to-end dialog technologies in a pragmatic way for multi-domain task-completion, noetic response selection, audio visual scene-aware dialog, and schema-guided dialog state tracking tasks. This paper describes the task definition, provided datasets, and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.

        @InProceedings{ws-neurips-convai19dstc,
  title     = {The Eighth Dialog System Technology Challenge},
  author    = {Kim, Seokhwan and Galley, Michel and Gunasekara, Chulaka and Lee, Sungjin and Atkinson, Adam and Peng, Baolin and Schulz, Hannes and Gao, Jianfeng and Li, Jinchao and Adada, Mahmoud and Huang, Minlie and Lastras, Luis and Kummerfeld, Jonathan K. and Lasecki, Walter S. and Hori, Chiori and Cherian, Anoop and Marks, Tim K. and Rastogi, Abhinav and Zang, Xiaoxue and Sunkara, Srinivas and Gupta, Raghav},
  year      = {2019},
  booktitle = {NeurIPS Workshop: Conversational AI: Today's Practice and Tomorrow's Potential},
  url       = {https://arxiv.org/abs/1911.06394},
  location  = {Vancouver, Canada},
  month     = {December},
  arxiv     = {https://arxiv.org/abs/1911.06394},
  data      = {https://github.com/dstc8-track2/NOESIS-II/},
}
        

        @InProceedings{ws-cscw19voids,
  title     = {Training Data Voids: Novel Attacks Against NLP Content Moderation},
  author    = {Huffaker, Jordan S. and Kummerfeld, Jonathan K. and Lasecki, Walter S. and Ackerman, Mark S.},
  year      = {2019},
  booktitle = {CSCW Workshop: Volunteer Work: Mapping the Future of Moderation Research},
  url       = {https://www.jkk.name/pub/ws19voids.pdf},
  location  = {Austin, TX},
  month     = {November},
}
        

        @InProceedings{ws-acl-convai19dstc7,
  title     = {DSTC7 Task 1: Noetic End-to-End Response Selection},
  author    = {Gunasekara, Chulaka and Kummerfeld, Jonathan K. and Polymenakos, Lazaros and Lasecki, Walter S.},
  year      = {2019},
  booktitle = {Proceedings of the First Workshop on NLP for Conversational AI},
  pages     = {60--67},
  url       = {https://www.aclweb.org/anthology/W19-4107},
  location  = {Florence, Italy},
  month     = {August},
  doi       = {10.18653/v1/W19-4107},
  data      = {https://ibm.github.io/dstc-noesis/public/index.html},
}
        

This paper introduces the Seventh Dialog System Technology Challenges (DSTC), which use shared datasets to explore the problem of building dialog systems. Recently, end-to-end dialog modeling approaches have been applied to various dialog tasks. The seventh DSTC (DSTC7) focuses on developing technologies related to end-to-end dialog systems for (1) sentence selection, (2) sentence generation and (3) audio visual scene aware dialog. This paper summarizes the overall setup and results of DSTC7, including detailed descriptions of the different tracks and provided datasets. We also describe overall trends in the submitted systems and the key results. Each track introduced new datasets and participants achieved impressive results using state-of-the-art end-to-end technologies.

        @InProceedings{ws-neurips-convai18dstc,
  title     = {Dialog System Technology Challenge 7},
  author    = {Yoshino, Koichiro and Hori, Chiori and Perez, Julien and D'Haro, Luis Fernando and Polymenakos, Lazaros and Gunasekara, Chulaka and Lasecki, Walter S. and Kummerfeld, Jonathan K. and Galley, Michel and Brockett, Chris and Gao, Jianfeng and Dolan, Bill and Gao, Xiang and Alamari, Huda and Marks, Tim K. and Parikh, Devi and Batra, Dhruv},
  year      = {2018},
  booktitle = {NeurIPS Workshop: Conversational AI: Today's Practice and Tomorrow's Potential},
  url       = {https://arxiv.org/abs/1901.03461},
  location  = {Montreal, Quebec, Canada},
  month     = {December},
  arxiv     = {https://arxiv.org/abs/1901.03461},
  data      = {https://ibm.github.io/dstc-noesis/public/index.html},
}
        

Scalable syntactic processing will underpin the sophisticated language technology needed for next generation information access. Companies are already using nlp tools to create web-scale question answering and 'semantic search' engines. Massive amounts of parsed web data will also allow the automatic creation of semantic knowledge resources on an unprecedented scale. The web is a challenging arena for syntactic parsing, because of its scale and variety of styles, genres, and domains.\n\nThe goals of our workshop were to scale and adapt an existing wide-coverage parser to Wikipedia text; improve the efficiency of the parser through various methods of chart pruning; use self-training to improve the efficiency and accuracy of the parser; use the parsed wiki data for an innovative form of bootstrapping to make the parser both more efficient and more accurate; and finally use the parsed web data for improved disambiguation of coordination structures, using a variety of syntactic and semantic knowledge sources.\n\nThe focus of the research was the C&C parser (Clark and Curran, 2007c), a state-of-the-art statistical parser based on Combinatory Categorial Grammar (ccg). The parser has been evaluated on a number of standard test sets achieving state-of-the-art accuracies. It has also recently been adapted successfully to the biomedical domain (Rimell and Clark, 2009). The parser is surprisingly efficient, given its detailed output, processing tens of sentences per second. For web-scale text processing, we aimed to make the parser an order of magnitude faster still. The C&C parser is one of only very few parsers currently available which has the potential to produce detailed, accurate analyses at the scale we were considering.

        @TechReport{report09jhu,
  title     = {Large-Scale Syntactic Processing: Parsing the Web},
  author    = {Clark, Stephen and Copestake, Ann and Curran, James R. and Zhang, Yue and Herbelot, Aurelie and Haggerty, James and Ahn, Byung-Gyu and Wyk, Curt Van and Roesner, Jessika and Kummerfeld, Jonathan K. and Dawborn, Tim},
  year      = {2009},
  institution = {Johns Hopkins University},
  url       = {https://www.jkk.name/pub/report09jhu.pdf},

The goals of our workshop were to scale and adapt an existing wide-coverage parser to Wikipedia text; improve the efficiency of the parser through various methods of chart pruning; use self-training to improve the efficiency and accuracy of the parser; use the parsed wiki data for an innovative form of bootstrapping to make the parser both more efficient and more accurate; and finally use the parsed web data for improved disambiguation of coordination structures, using a variety of syntactic and semantic knowledge sources.

The focus of the research was the C&C parser (Clark and Curran, 2007c), a state-of-the-art statistical parser based on Combinatory Categorial Grammar (ccg). The parser has been evaluated on a number of standard test sets achieving state-of-the-art accuracies. It has also recently been adapted successfully to the biomedical domain (Rimell and Clark, 2009). The parser is surprisingly efficient, given its detailed output, processing tens of sentences per second. For web-scale text processing, we aimed to make the parser an order of magnitude faster still. The C&C parser is one of only very few parsers currently available which has the potential to produce detailed, accurate analyses at the scale we were considering.},
}