Frames: a corpus for adding memory to goal-oriented dialogue systems (El Asri et al., 2017)

Another paper about a dataset of dialogues, but this time with structure. Like the paper from yesterday, the aim is a dataset of task-oriented conversations, but with more complexity than prior work. The difference is that this work includes a structured representation of the state of the conversation: frames.

A frame is essentially a tuple describing a query, e.g. (Destination: Sydney, Origin: Ann Arbor, price: 1500 USD). There are multiple frames in a dialogue (a departure from DSTC tasks), and utterances are labeled with dialogue acts that capture modifications to the frames as well as references to them. This structure sounds fairly general, though the focus here was on vacation planning, where the user is buying a package. The setup doesn’t maximise the potential complexity though, as there are a small number of set packages available, rather than the complex tradeoffs of flight+hotel combinations that exist in practise. Looking at the example dialogues in the paper, it has complete sentences of some complexity. One thing I’m still curious about is disagreements between annotators, as for the complete task the score was 0.62 +/- 5 (with dialogue acts being trickier than slot values, and no scores for frame references on their own).

Comparing to the Stanford dataset this is smaller (11k vs. 1.4k), but has more turns per dialogue (11 vs. 15) and probably longer turns too, judging by the examples. The tasks are completely different, but both come with small tables of information that are private to the two participants and required for almost every turn in the conversation. Evaluating on both could be a great way to show the flexibility of a dialogue system, but the lack of frames for the Stanford data and the difficulty of running a human evaluation for this data limits the feasible types of multi-domain experiments.



  author    = {El Asri, Layla  and  Schulz, Hannes  and  Sharma, Shikhar  and  Zumer, Jeremie  and  Harris, Justin  and  Fine, Emery  and  Mehrotra, Rahul  and  Suleman, Kaheer},
  title     = {Frames: a corpus for adding memory to goal-oriented dialogue systems},
  booktitle = {Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue},
  month     = {August},
  year      = {2017},
  address   = {Saarbrucken, Germany},
  publisher = {Association for Computational Linguistics},
  pages     = {207--219},
  url       = {}


comments powered by Disqus