Handwritten character recognition is a hot topic for research now-a-days. If we can convert a handwritten piece of paper into a text-searchable document using the Optical Character Recognition (OCR) technique, we can easily understand the content and do not need to read a handwritten document. OCR in the English language is very common, but in the Bengali language, it is very hard to find a good quality OCR application. If we can merge machine learning and deep learning with OCR, it can be a huge contribution to this field. A variety of techniques have been suggested by various researchers on Bengali handwritten character recognition. A lot of machine learning algorithms and deep neural networks were used in their work, but the explanations of their models are not available. In our work, we have used different machine learning models and CNN also to recognize handwritten Bengali digits. We have got acceptable accuracy from some ML models, and CNN has given us great testing accuracy. Grad-CAM was used as an XAI method on our CNN model, which gave us insights about the model and helped us to detect the origin of interest for recognizing a digit from an image.