Skip to main content
Faculty of Science

research & innovation

Paper of the Month

Using DCNN-Based Methods to Extract Software Usage Information

By Connie Jeske Crane

 

Ping Gong

How to improve your software product? For software developers, this is an ongoing challenge. Building test scenarios that mimic how your software is used in the field can help. But when you have a larger and diverse customer base, the task is rather daunting, not to mention time-consuming.

At Ryerson, Domenic Curro, a Computer Science student completing his master’s degree, has devoted his studies to enhancing current approaches to extracting software usage information – with an end goal of advancing software engineering activities.

Recently, Curro, supervised by two Ryerson professors, Dr. Kosta Derpanis and Dr. Andriy Miranskyy, undertook an investigation featuring a unique and readily available source of raw material, as Curro explains: “A collection of 236 publicly available Microsoft Word tutorial videos (published on YouTube).”


“Our goal was to construct an automatic approach to extract information about user actions from instructional videos. To achieve this goal, we use a Deep Convolutional Neural Network (DCNN) to recognize user actions,” explains Curro. “We hoped to show that image and video data have become more accessible, with modern image processing methods.”


Briefly, DCNNs are a class of deep, feed-forward artificial neural networks and in machine learning, they’re driving rapid advances in high-level computer vision tasks. Already successfully applied to analyze visual imagery and video recognition, DCNNs haves many more potential applications – from powering vision in robots to self-driving cars.
 

The algorithm pipeline from input video to a Usage Profile, predicting the goal of the user.

As for Curro’s study, while his methods included a laborious component – “I spent an entire week, from waking to sleeping, hand-labelling Microsoft Word video tutorials!” – he says it was gauged a success. “Our pilot study shows that a DCNN trained to recognize user actions in video can classify five different actions in our collection of Microsoft Word tutorial videos. In our empirical evaluation we reported a mean average precision of 94.42 per cent across all actions.” Speaking to the work’s ultimate significance, he says, “This study demonstrates the efficacy of DCNN-based methods for extracting software usage information from videos. Moreover, this approach may aid in other software engineering activities that require information about customer usage of a product.”


Moreover, last May, Curro got the opportunity to travel to Argentina and present his findings at a conference in Buenos Aires at the 39th International Conference on Software Engineering. His paper was also published by IEEE Press in ICSE-NIER '17: Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results Track.

*1*
*2* The six frames are from one video. The first image shows the original frames, and the second image show the neural network predictions for what action the user is performing in that frame.

 

Curro is equally enthusiastic about his choice to study at Ryerson. “Ryerson hosts a knowledgeable, helpful and inspiring set of peers and faculty. With their help I have had my ideas nurtured and grown. My greatest takeaway from graduate studies at Ryerson is my newfound appreciation for a strong sense of community.”


Curro’s research was funded through NSERC grants held by Derpanis and Miranskyy. Travel funding was provided by Ryerson's Computer Science Department, the Yeates School of Graduate Studies, Ryerson International and CUPE/3904.