Annotation Information

The data for the 2019 paper was selected and annotated in various ways. The supplementary material to the paper discusses these, and below we provide additional details.

Training Set

Each file was annotated by one person. All annotators had gone through the training process.

Part A

Selected by:

  1. Calculate stats for every hour
  2. Determine cutoffs for 0-25%, 25-50%, 50-75%, and 75-100% on each axis (users, messages, directed)
  3. For each case, divide the data into four, and select four hours from each section

Note, the identification of directed messages used an earlier version of our code that was less accurate. However, the difference is probably small (1-5%).

Cutoffs were:

Stats used:

Part B

Chosen by:

  1. Filter out all hours that are in the max or min 5% for users, messages, or addressing
  2. Randomly select 10 hours
  3. For busy hours, keep the first 100 messgaes, for quiet hours, add messages from the following hour to get up to 100

Part C

Selected by choosing a random point in the logs and keeping 1,500 messages after that point (1,000 as context, 500 to annotate).

2004-12-25 2005-02-06 2005-02-27 2005-05-14 2005-06-06 2005-06-12 2005-06-16
2005-07-29 2005-09-26 2005-10-07 2005-10-12 2005-12-03 2005-12-04 2005-12-16
2005-12-23 2006-01-02 2006-01-12 2006-02-20 2006-02-28 2006-03-05 2006-05-02
2006-05-15 2006-05-27 2006-05-29 2006-06-08 2006-06-21 2006-06-28 2006-07-01
2006-08-06 2006-08-11 2006-08-13 2006-08-15 2006-09-13 2006-09-24 2006-11-01
2006-12-06 2006-12-20 2007-01-12 2007-01-21 2007-01-29 2007-02-06 2007-02-07
2007-02-15 2007-06-01 2007-08-22 2007-08-24 2007-10-24 2008-01-02 2008-01-03
2008-02-07 2008-02-14 2008-03-01 2008-05-24 2008-07-03 2008-10-02 2009-05-04
2009-05-08 2009-07-02 2009-11-13 2009-12-05 2010-01-04 2010-03-08 2010-03-20
2010-04-12 2010-05-30 2010-06-21 2010-08-15 2010-10-17 2010-10-27 2011-02-13
2011-02-23 2011-03-18 2011-04-14 2011-04-17 2011-08-22 2011-11-24 2011-12-07
2012-03-24 2012-06-20 2013-05-28 2013-08-29 2013-09-16 2014-01-08 2014-08-14
2014-09-29 2014-12-21 2014-12-27 2015-05-08 2017-02-06 2017-03-02 2017-03-23
2017-05-09 2017-07-15 2017-09-02 2018-02-27

Development Set

Selected by choosing a random point in the logs and keeping 1,250 messages after that point (1,000 as context, 250 to annotate).

2004-11-15_03 2005-06-27_12 2005-08-08_01 2008-12-11_11 2009-02-23_10
2009-03-03_10 2009-10-01_17 2011-05-29_19 2011-11-13_02 2016-12-19_20

Test Set

Selected by choosing a random point in the logs and keeping 1,500 messages after that point (1,000 as context, 500 to annotate).

2005-07-06_14 2007-01-11_12 2007-12-01_03 2008-07-14_18 2010-08-17_18
2013-09-01_02 2014-06-18_13 2015-03-18_05 2016-02-22_17 2016-06-08_07

Pilot Data

Used in the process of developing the annotation scheme, NOT intended for use in developing or evaluating models. If you use this data for either training or tuning your model your results with NOT be comparable with those in the paper. This is included mainly for completeness.

Overall 1,250 lines (counting is a little subtle as it includes lines that didn’t get a label)

Phase 1:

Phase 2: