Leaderboard



The leaderboard shows the results of different models of four tasks: Code Retrieval, Code Summarization, Code Completion, and Typ inference.

If you would like to report your results here, please submit an issue to CodeMind GitHub repository. All results will be updated if they pass our check.





MRR of our model and baseline methods for the task of code retrieval over CodeSearchNet dataset. (Best scores are in boldface.)

Rank Model Go Java JS PHP Python Ruby

1

07/01/2021
SelfAttn

Code Retrieval baseline

78.45 66.55 50.38 65.78 79.09 47.96

2

06/07/2021
Conv1D

Code Retrieval baseline

70.87 60.49 38.81 61.92 67.29 36.53

3

07/01/2021
NBOW

Code Retrieval baseline

66.59 59.92 47.15 54.75 63.33 42.86

4

07/01/2021
BiRNN

Code Retrieval baseline

65.80 48.60 23.23 51.36 48.28 19.35

Performance of our model and baseline methods for the task of code summarization over Python-Doc dataset. (Best scores are in boldface.)

Rank Model BLEU-4 METEOR BOUGE-L

1

07/01/2021
PLBART

Code Summarization baseline

32.71 18.13 46.05

2

07/01/2021
Transformer + BPE

Code Summarization baseline

31.57 17.74 45.18

3

07/01/2021
Transformer

Code Summarization baseline

30.64 17.65 44.59

4

07/01/2021
Seq2Seq + Attn

Code Summarization baseline

25.57 14.40 39.41

5

07/01/2021
Tree2Seq + Attn

Code Summarization baseline

23.35 12.59 36.49

MRR of our model and baseline methods for the task of code completion over Py150 dataset. (Attr: attribute; Num: numeric constant; Name: variable, module; Func: function parameter name; Token: all tokens. Best scores are in boldface.)

Rank Model Accuracy
Attr Num Name Param Token

1

07/01/2021
TravTrans

Code Completion baseline

72.08 68.55 76.33 71.08 83.17

2

07/01/2021
GPT-2

Code Completion baseline

70.37 62.20 63.84 73.54 82.17

3

07/01/2021
LSTM

Code Completion baseline

51.67 47.45 46.52 66.06 73.73

Accuracy of our model and baseline methods for the task of type inference over Py150 dataset.

Rank Model All types Any types
Acc@1 Acc@5 Acc@1 Acc@5

1

07/01/2021
DeepTyper

Type Inference baseline

0.52 0.67 0.43 0.67

2

07/01/2021
Transformer

Type Inference baseline

0.34 0.64 0.37 0.75