subword level kenLM

[Dreahim]

Hello,
I am trying to train kenLM using subword tokenization model
I am using the same steps here
but I get unwanted results

This a secrrenshot from my kenLM model

They should be arabic words, not like this

and this is output of Command line instruction if it is useful

[NeMo I 2024-01-23 16:22:03 train_kenlm:148] Running lmplz command 
    
    /home/user4/training/final_project/project6_nemo/LM/NeMo/nemo/decoders/kenlm/build/bin/lmplz -o 2 --arpa /home/user4/training/final_project/project6_nemo/LM/LM_models/LM_2gram_new.tmp.arpa --discount_fallback --prune 0
    
    
Read 0 lines: 0 lines [00:00, ? lines/s]=== 1/5 Counting and sorting n-grams ===
File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
Read 0 lines: 59765 lines [00:00, 1040437.38 lines/s]
[NeMo I 2024-01-23 16:22:12 kenlm_utils:204] Finished writing 8 chunks to <_io.BufferedWriter name=17>. Current chunk index = 8
Unigram tokens 2255330 types 130
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:1560 2:26847343411
Substituting fallback discounts for order 0: D1=0.5 D2=1 D3+=1.5
Substituting fallback discounts for order 1: D1=0.5 D2=1 D3+=1.5
Statistics:
1 130 D1=0.5 D2=1 D3+=1.5
2 4762 D1=0.5 D2=1 D3+=1.5
Memory estimate for binary LM:
type    kB
probing 87 assuming -p 1.5
probing 87 assuming -r models -p 1.5
trie    25 without quantization
trie    13 assuming -q 8 -b 8 quantization 
trie    25 assuming -a 22 array pointer compression
trie    13 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
=== 3/5 Calculating and sorting initial probabilities ===
Chain sizes: 1:1560 2:76192
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
####################################################################################################
=== 4/5 Calculating and writing order-interpolated probabilities ===
Chain sizes: 1:1560 2:76192
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
####################################################################################################
=== 5/5 Writing ARPA model ===
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Name:lmplz	VmPeak:26366536 kB	VmRSS:5356 kB	RSSMax:7149184 kB	user:0.398472	sys:2.69766	CPU:3.09615	real:9.02089
[NeMo I 2024-01-23 16:22:12 train_kenlm:173] Running binary_build command 
    
    /home/user4/training/final_project/project6_nemo/LM/NeMo/nemo/decoders/kenlm/build/bin/build_binary trie /home/user4/training/final_project/project6_nemo/LM/LM_models/LM_2gram_new.tmp.arpa /home/user4/training/final_project/project6_nemo/LM/LM_models/LM_2gram_new
    
    
Reading /home/user4/training/final_project/project6_nemo/LM/LM_models/LM_2gram_new.tmp.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Identifying n-grams omitted by SRI
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Writing trie
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
SUCCESS

I tried to use that LM model with ASR in both char level and subword level, but I always get theese results

output of LM: [[(-440.85137939453125, ''), (-2417.015869140625, '▁وَ'), (-2424.113525390625, '▁ال'), (-2425.376953125, 'ص'), (-2428.3291015625, 'غ'), (-2428.492431640625, 'ج'), (-2429.395263671875, 'ِي'), (-2430.468505859375, '▁مِ')]]

[titu1994]

@karpnv could you take a look

[Dreahim]

Please, I am sill waiting the answer