An interactive action set detection in natral language hybrid system

In this paper, we introduce an interface for natural language dialog system and the Microsoft Excel API. This interface enables users to describe actions in unrestricted natural language interactively and run these actions in the technical domain like Microsoft Excel. We implement a hybrid system that supports natural language interactions similar as a chat bot and gesture control to follow the user’s showing actions. For this purpose and also to avoid the lack of inconsistent code documentation, we dynamically export the provided documentation for coding in Visual Studio and provide it in the xml-format to the hybrid system. Users can use already existing actions and provide new action sets to the hybrid system at runtime. For our first prototype of the hybrid system, we concentrate on extending for handling graphs and charts. Evaluated by a user study the hybrid system can successfully resolve more that 80 % of the given user tasks.