Nitrogen fertilizer is inevitable for rice production to ensure that the crop's nitrogen need is adequately supplied, during the growing season. International Rice Research Institute (IRRI) has proposed Leaf Color Chart (LCC) to detect the exact nitrogen need of paddy. Farmers generally monitor the plant's growth (which is also an indicator of the nitrogen concentration of leaves) by comparing the leaf color with the corresponding color of the LCC. Currently, in most cases, LCC is used manually to determine the fertilizer need and thus, there is a chance of either overestimating or underestimating the amount of fertilizer. To avoid this problem, a smart fertilizer recommendation system is proposed in this paper. The proposed method is able to automate the manual acquisition and interpretation of leaf color for classification through LCC. The experimentation considers a sample of 6000 Aman paddy leaf images. The data acquisition process was performed according to IRRI's guidance of taking the paddy leaf images within the body shade by our developed application. The data/images have already been made public in Kaggle-a well-known dataset website. The semantic segmentation of the dataset was performed by a powerful Convolutional Neural Network (CNN) backbone architecture-DeepLabV3+. Color classification into 4 categories of the LCC was performed by CNN architecture which consists of seven layers. Information gain based evaluation was performed in the Decision Tree (DT) approach to select features and with the selected features DT classified images into 4 categories. Color classification by our two proposed methods achieved 94.22% accuracy in CNN Model and 91.22% accuracy in the DT classifier.