A range of services exist for collecting annotations from paid workers. This post gives an overview of a bunch of them.
For a more flexible dialogue system, use the crowd to propose and vote on responses, then introduce agents and a model for voting, gradually learning to replace the crowd.
By dividing a task up among multiple annotators carefully we can achieve high-quality real-time annotation of data, in this case transcription of audio.
By using a generative model to explain worker annotations, we can more effectively predict the correct label, and which workers are spamming.