Issue-Report-Classification-Using-RoBERTa

This repository contains the notebook for training and testing the classifiers for our participation in the tool competition organized in the scope of the 1st International Workshop on Natural Language-based Software Engineering.

The model is available on HuggingFace

Link to Classifier 1 Colab notebook
Link to Classifier 2 Colab notebook

If you use the software in this repository please consider citing our paper as follows:

@inproceedings{Colavito-2022,
  title = {Issue Report Classification Using Pre-trained Language Models},
  booktitle = {2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE)},
  author = {Colavito, Giuseppe and Lanubile, Filippo and Novielli, Nicole},
  year = {2022},
  month = may,
  pages = {29--32},
  doi = {10.1145/3528588.3528659},
  abstract = {This paper describes our participation in the tool competition organized in the scope of the 1st International Workshop on Natural Language-based Software Engineering. We propose a supervised approach relying on fine-tuned BERT-based language models for the automatic classification of GitHub issues. We experimented with different pre-trained models, achieving the best performance with fine-tuned RoBERTa (F1 = .8591).},
  keywords = {Issue classification, BERT, deep learning, labeling unstructured data,
software maintenance and evolution},
}

Comparison with CodeBERT

As show in the table, the two models are equivalent. For CodeBERT, we removed the preprocessing to keep all the code, since it is trained on code also. Note that also removing the code, the performances are lower, but remain similar to RoBERTa

	CodeBERT	RoBERTa	CodeBERT	RoBERTa	CodeBERT	RoBERTa
	precision	precision	recall	recall	f1-score	f1-score	support
bug	0.8729	0.8750	0.8997	0.8988	0.8861	0.8867	40122
enhancement	0.8729	0.8713	0.8716	0.8743	0.8722	0.8728	33073
question	0.6664	0.6760	0.5528	0.5591	0.6043	0.6120	6943
micro avg	0.8580	0.8591	0.8580	0.8591	0.8580	0.8591	80138
macro avg	0.8041	0.8074	0.7747	0.7774	0.7875	0.7905	80138

Combination of the filters

The filters combined are the following:

• The project must have an age of more than 4 year
• The project must have more than 1500 stars
• The issue should have a single label (and should still be available) on GitHub

Results are compared with a model trained on a random sampling of the training set, tested on the filtered test set

RANDOM SAMPLING	Precision	Recall	F1-score	Support
bug	0.8935	0.7937	0.8407	2094
enhancement	0.8496	0.7569	0.8005	1164
question	0.5140	0.8233	0.6329	600
microavg	0.7872	0.7872	0.7872	3858
macroavg	0.7524	0.7913	0.7580	3858

FILTERED	Precision	Recall	F1-score	Support
bug	0.9032	0.7751	0.8342	2094
enhancement	0.7763	0.8290	0.8018	1164
question	0.5587	0.7617	0.6446	600
microavg	0.7893	0.7893	0.7893	3858
macroavg	0.7461	0.7886	0.7602	3858