Transactions of the Association for Computational Linguisticshttps://transacl.org/index.php/tacl
<div id="journalDescription"><p>Welcome to the TACL submission site!</p><p>Transactions of the Association for Computational Linguistics (TACL) is an <a href="http://www.aclweb.org/">ACL</a>-sponsored journal <a href="https://www.mitpressjournals.org/loi/tacl">published by MIT Press</a> that publishes papers in all areas of computational linguistics and natural language processing. TACL has the following features:</p><ul><li>TACL publishes conference-length papers, but has a journal-style reviewing process (for example, the option for an action editor to recommend the “revise and resubmit” category for a paper).</li><li>Papers appearing at TACL are eligible for a presentation at certain ACL-sponsored conferences. Thus the model combines the benefits of a journal, with the benefits of being able to present the work at a major conference. (Presentation is optional; authors do not have to present their papers at the conference).</li><li>TACL accepts submissions all year (the 1st day of each month is a submission deadline).</li><li class="hover">TACL is committed to fast-turnaround reviewing.</li></ul>Links: <a href="/ojs/index.php/tacl/about/submissions" target="_blank">Information for authors, including submission instructions</a>; <a href="/ojs/index.php/tacl/about/editorialPolicies#peerReviewProcess" target="_blank">review process</a>; <a href="/ojs/index.php/tacl/about/history">annual reports</a> (includes journal statistics presented to the ACL); <a href="/ojs/index.php/tacl/about/editorialPolicies#custom-0">publication ethics statement</a>.</div><div> </div><div id="announcementsHome"> </div>The MIT Pressen-USTransactions of the Association for Computational Linguistics2307-387X<p>Copyright for TACL papers is held by the Association for Computational Linguistics, and articles are distributed under Creative Commons License CC-BY.</p><p>However, for TACL papers published in volumes 1 or 2 or in volume 3 up to and including page 403, the original published pdfs may not include a note about licensing or may mention Creative Commons License CC-BY-NC-SA <span class="__postbox-detected-content __postbox-detected-date">4.0 instead.</span></p><p> </p><p> </p>

Phonotactic Complexity and its Trade-offshttps://transacl.org/index.php/tacl/article/view/1759
<p>We present methods for calculating a measure of phonotactic complexity---bits per phoneme---that permits a straightforward cross-linguistic comparison. When given a word, represented as a sequence of phonemic segments such as symbols in the international phonetic alphabet, and a statistical model trained on a sample of word types from the language, we can approximately measure bits per phoneme using the negative log-probability of that word under the model. This simple measure allows us to compare the entropy across languages, giving insight into how complex a language's phonotactics are. Using a collection of 1016 basic concept words across 106 languages, we demonstrate a very strong negative correlation of -0.74 between bits per phoneme and the average length of words.</p>Tiago PimentelBrian RoarkRyan D. Cotterell
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-01-232020-01-238118AMR-To-Text Generation with Graph Transformerhttps://transacl.org/index.php/tacl/article/view/1805
<p>Abstract meaning representation (AMR)-to-text generation is the challenging task of generating natural language texts from AMR graphs, where nodes represent concepts and edges denote relations. The current state-of-the-art methods use graph-to-sequence models; however, they still cannot significantly outperform the previous sequence-to-sequence models or statistical approaches. In this paper, we propose a novel graph-to-sequence model (Graph Transformer) to address the above-mentioned task. The model directly encodes the AMR graphs and learns the node representations. A pairwise interaction function is used for computing the semantic relations between the concepts. Moreover, attention mechanisms are employed for aggregating the information from the incoming and outgoing neighbors, which help the model to capture the semantic information effectively. Our model outperforms the state-of-the-art neural approach by 1.5 BLEU points on LDC2015E86 and 4.8 BLEU points on LDC2017T10 and achieves new state-of-the-art performances.</p>Tianming WangXiaojun WanHanqi Jin
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-02-012020-02-0181933What BERT is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Modelshttps://transacl.org/index.php/tacl/article/view/1852
<p>Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction -- and in particular, it shows clear insensitivity to the contextual impacts of negation.</p>Allyson Ettinger
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-02-012020-02-0183448Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?https://transacl.org/index.php/tacl/article/view/1779
<p>Data privacy is an important issue for "machine learning as a service" providers. We focus on the problem of membership inference attacks: given a data sample and black-box access to a model's API, determine whether the sample existed in the model's training data. Our contribution is an investigation of this problem in the context of sequence-to-sequence models, which are important in applications such as machine translation and video captioning. We define the membership inference problem for sequence generation, provide an open dataset based on state-of-the-art machine translation models, and report initial results on whether these models leak private information against several kinds of membership inference attacks. </p>Sorami HisamotoMatt PostKevin Duh
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-03-132020-03-1384963SpanBERT: Improving Pre-training by Representing and Predicting Spanshttps://transacl.org/index.php/tacl/article/view/1853
<p>We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT-Large, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.</p>Mandar JoshiDanqi ChenYinhan LiuDaniel S. WeldLuke ZettlemoyerOmer Levy
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-03-132020-03-1386477A Graph-based Model for Joint Chinese Word Segmentation and Dependency Parsinghttps://transacl.org/index.php/tacl/article/view/1876
<p>Chinese word segmentation and dependency parsing are two fundamental tasks for Chinese natural language processing. The dependency parsing is defined on word-level. Therefore, word segmentation is the precondition of dependency parsing, which makes dependency parsing suffer from error propagation and unable to directly make use of the character-level pre-trained language model (such as BERT). In this paper, we propose a graph-based model to integrate Chinese word segmentation and dependency parsing. Different from previous transition-based joint models, our proposed model is more concise, which results in fewer efforts of feature engineering. Our graph-based joint model achieves better performance than previous joint models and state-of-the-art results in both Chinese word segmentation and dependency parsing. Besides, when BERT is combined, our model can substantially reduce the performance gap of dependency parsing between joint models and gold-segmented word-based models. Our code is publicly available at https://github.com/fastnlp/JointCwsParser.</p>Hang YanXipeng QiuXuanjing Huang
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-03-132020-03-1387892A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationhttps://transacl.org/index.php/tacl/article/view/1886
<p>Story generation, namely generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling ﬂuency and local coherence, existing neural language generation models (e.g., GPT-2) still suffer from repetition, logic conﬂicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difﬁculty of associating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper temporal order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories. To further capture the causal and temporal dependencies between the sentences in a reasonable story, we employ multi-task learning which combines a discriminative objective to distinguish true and fake stories during ﬁne-tuning. Automatic and manual evaluation shows that our model can generate more reasonable stories than state-of-the-art baselines, particularly in terms of logic and global coherence.</p>Jian GuanFei HuangMinlie HuangZhihao ZhaoXiaoyan Zhu
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-012020-04-01893108Improving Candidate Generation for Low-resource Cross-lingual Entity Linkinghttps://transacl.org/index.php/tacl/article/view/1906
<p>Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts. The first step of (X)EL is candidate generation, which retrieves a list of plausible candidate entities from the target-language KB for each mention. Approaches based on resources from Wikipedia have proven successful in the realm of relatively high-resource languages (HRL), but these do not extend well to low-resource languages (LRL) with few, if any, Wikipedia pages. Recently, transfer learning methods have been shown to reduce the demand for resources in the LRL by utilizing resources in closely-related languages, but the performance still lags far behind their high-resource counterparts. In this paper, we first assess the problems faced by current entity candidate generation methods for low-resource XEL, then propose three improvements that (1) reduce the disconnect between entity mentions and KB entries, and (2) improve the robustness of the model to low-resource scenarios. The methods are simple, but effective: we experiment with our approach on seven XEL datasets and find that they yield an average gain of 16.9% in Top-30 gold candidate recall, compared to state-of-the-art baselines. Our improved model also yields an average gain of 7.9% in in-KB accuracy of end-to-end XEL.</p>Shuyan ZhouShruti RijhwaniJohn WietingJaime CarbonellGraham Neubig
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-012020-04-018109124Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networkshttps://transacl.org/index.php/tacl/article/view/1892
<p>Learners that are exposed to the same training data might generalize differently due to differing inductive biases. In neural network models, inductive biases could in theory arise from any aspect of the model architecture. We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection. For both tasks, the training set is consistent with a generalization based on hierarchical structure and a generalization based on linear order. All architectural factors that we investigated qualitatively affected how models generalized, including factors with no clear connection to hierarchical structure. For example, LSTMs and GRUs displayed qualitatively different inductive biases. However, the only factor that consistently contributed a hierarchical bias across tasks was the use of a tree-structured model rather than a model with sequential recurrence, suggesting that human-like syntactic generalization requires architectural syntactic structure.</p>R. Thomas McCoyRobert FrankTal Linzen
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-012020-04-018125140Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensionhttps://transacl.org/index.php/tacl/article/view/1882
<p>Machine reading comprehension tasks require a machine reader to answer questions relevant to the given document. In this paper, we present the first free-form multiple-Choice Chinese machine reading Comprehension dataset (C^3), containing 13,369 documents (dialogues or more formally written mixed-genre texts) and their associated 19,577 multiple-choice free-form questions collected from Chinese-as-a-second-language examinations.</p><p><br /> We present a comprehensive analysis of the prior knowledge (i.e., linguistic, domain-specific, and general world knowledge) needed for these real-world problems. We implement rule-based and popular neural methods and find that there is still a significant performance gap between the best performing model (68.5%) and human readers (96.0%), especially on problems that require prior knowledge. We further study the effects of distractor plausibility and data augmentation based on translated relevant datasets for English on model performance. We expect C^3 to present great challenges to existing systems as answering 86.8% of questions requires both knowledge within and beyond the accompanying document, and we hope that C^3 can serve as a platform to study how to leverage various kinds of prior knowledge to better understand a given written or orally oriented text. C^3 is available at <a href="https://dataset.org/c3/">https://dataset.org/c3/</a>.</p>Kai SunDian YuDong YuClaire Cardie
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-012020-04-018141155Theoretical Limitations of Self-Attention in Neural Sequence Modelshttps://transacl.org/index.php/tacl/article/view/1815
Transformers are emerging as the new workhorse of NLP, showing great success across tasks.<br />Unlike LSTMs, transformers process input sequences entirely through self-attention. Previous work has suggested that the computational capabilities of self-attention to process hierarchical structures are limited. In this work, we mathematically investigate the computational power of self-attention to model formal languages. Across both soft and hard attention, we show strong theoretical limitations of the computational abilities of self-attention, finding that it cannot model periodic finite-state languages, nor hierarchical structure, unless the number of layers or heads increases with input length. These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.Michael Hahn
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-012020-04-018156171Target-Guided Structured Attention Network for Target-dependent Sentiment Analysishttps://transacl.org/index.php/tacl/article/view/1843
<p>Target-dependent sentiment analysis (TDSA) aims to classify the sentiment of a text towards a given target. The major challenge of this task lies in modeling the semantic relatedness between a target and its context sentence. This paper proposes a novel Target-Guided Structured Attention Network (TG-SAN), which captures target-related contexts for TDSA in a fine-to-coarse manner. Given a target and its context sentence, the proposed TG-SAN first identifies multiple semantic segments from the sentence using a target-guided structured attention mechanism. It then fuses the extracted segments based on their relatedness with the target for sentiment classification. We present comprehensive comparative experiments on three benchmarks with three major findings. Firstly, TG-SAN outperforms the state-of-the-art by up to 1.61% and 3.58% in terms of accuracy and Marco-F1 respectively. Secondly, it shows a strong advantage in determining the sentiment of a target when the context sentence contains multiple semantic segments. Lastly, the attention results produced by TG-SAN are highly interpretable as visualization results shown.</p>Ji ZhangChengyao ChenPengfei LiuChao HeCane Wing-Ki Leung
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-182020-04-188172182Break It Down: A Question Understanding Benchmarkhttps://transacl.org/index.php/tacl/article/view/1845
<p>Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the Break dataset, containing over 83K pairs of questions and their QDMRs.</p><p>We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HotpotQA dataset, (b) it can be deterministically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use Break to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.</p>Tomer WolfsonMor GevaAnkit GuptaYoav GoldbergMatt GardnerDaniel DeutchJonathan Berant
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-182020-04-188183198Acoustic-Prosodic and Lexical Cues to Deception and Trust: Deciphering How People Detect Lieshttps://transacl.org/index.php/tacl/article/view/1834
<p>Humans rarely perform better than chance at lie detection. To better understand human perception of deception, we created a game framework, LieCatcher, to collect ratings of perceived deception using a large corpus of deceptive and truthful interviews. We analyzed the acoustic-prosodic and linguistic characteristics of language trusted and mistrusted by raters and compared these to characteristics of actual truthful and deceptive language to understand how perception aligns with reality. With this data we built classifiers to automatically distinguish trusted from mistrusted speech, achieving an F1 of 66.1%. We next evaluated whether the strategies raters said they used to discriminate between truthful and deceptive responses were in fact useful. Our results show that, while several prosodic and lexical features were consistently perceived as trustworthy, they were not reliable cues. Also, the strategies that judges reported using in deception detection were not helpful for the task. Our work sheds light on the nature of trusted language and provides insight into the challenging problem of human deception detection.</p>Xi (Leslie) ChenSarah Ita LevitanMichelle LevineMarko MandicJulia Hirschberg
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-182020-04-188199214Unsupervised Discourse Constituency Parsing Using Viterbi EMhttps://transacl.org/index.php/tacl/article/view/1811
<p>In this paper, we introduce an unsupervised discourse constituency parsing algorithm. We use Viterbi EM with a margin-based criterion to train a span-based discourse parser in an unsupervised manner. We also propose initialization methods for Viterbi training of discourse constituents based on our prior knowledge of text structures. Experimental results demonstrate that our unsupervised parser achieves comparable or even superior performance to fully supervised parsers. We also investigate discourse constituents that are learned by our method.</p>Noriki NishidaHideki Nakayama
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-182020-04-188215230Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension using Distributional Semantic Modelshttps://transacl.org/index.php/tacl/article/view/1912
<p>Recent years have seen a growing interest within the natural language processing (NLP) community in evaluating the ability of semantic models to capture human meaning representation in the brain. Existing research has mainly focused on applying semantic models to decode brain activity patterns associated with the meaning of individual words, and, more recently, this approach has been extended to sentences and larger text fragments. Our work is the first to investigate metaphor processing in the brain in this context. We evaluate a range of semantic models (word embeddings, compositional, and visual models) in their ability to decode brain activity associated with reading of both literal and metaphoric sentences. Our results suggest that compositional models and word embeddings are able to capture differences in the processing of literal and metaphoric sentences, providing support for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension.</p>Vesna G. DjokicJean MaillardLuana BulatEkaterina Shutova
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-282020-04-288231246Machine Learning Driven Language Assessmenthttps://transacl.org/index.php/tacl/article/view/1743
<p>We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales based on a given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with other high-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements.</p>Burr SettlesMasato HagiwaraGeoffrey T. LaFlair
Copyright (c) 2020 Association for Computational Linguistics
http://creativecommons.org/licenses/by/4.0
2020-04-282020-04-288247263