image-captioning-using-keras.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<center> <h2 style=\"background-color:Orange; color:white\" ><br>Teaching Computers to describe pictures<br></h2></center>\n\n\n**`Image Captioning`**<br> \n* Image Captioning is process of generating textual description of an image\n* It uses both Natural Language Processing and Computer Vision to generate captions\n","metadata":{"id":"FlNPjpU79WGB"}},{"cell_type":"markdown","source":"**`Table of Contents`** :\n1. Introduction\n2. Motive\n3. Prerequisites\n4. Data Collection\n5. Interpreting data\n6. Data Cleaning\n7. Loading the training set\n8. Data Preprocessing — Images\n9. Data Preprocessing — Captions\n10. Data Preparation using Generator Function\n11. Word Embeddings\n12. Model Architecture\n13. Inference\n14. Evaluation\n15. Conclusion and Future work\n16. References\n\n","metadata":{"id":"d0RhX1H99c64"}},{"cell_type":"markdown","source":"<center> <h3 style=\"background-color:red; color:white\" ><br>1 - Introduction<br></h3></center>\n\n\n`What is in this image??`\n<center><img src=\"https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Ftse2.mm.bing.net%2Fth%3Fid%3DOIP.BPMGsi39UsNLDf7-qBZwmwHaC7%26pid%3DApi&f=1\"/></center>\n\nWell some might say “A white dog in a grassy area with collar belt”, some may say “White dog with brown spots and collar belt” and yet some others might say “A dog on grass, eating grass and collar belt” maybe\n\nDont be happy man : Definitely all of these captions are relevant for this image and there may be many others also<br>\nPoint I want to make is, it’s so easy for human beings to just have a glance at a picture and describe it in an appropriate language. Even a 5 year old could do this with utmost ease now in 2023 yes.\n\n**`Problem Statement`** <br>\n`Can you write a computer program that takes an image as input and produces a relevant caption as output for given image?` <br>\nWith advent of Deep Learning this problem can be solved very easily if one have required dataset.<br>\n`Andrej Karapathy` well researched this problem in his PhD thesis at Stanford, who is also now `Director of AI at Tesla`\n\n### AIM\nAim is to explain (in layman term possible) that \"\"How Deep Learning can be used to solve this problem\"\" of generating a caption for a given image, hence name **`Image Captioning`** <br>\n[Caption Bot](https://captionbot.net/home) ==> state-of-the-art system created by Microsoft\n","metadata":{"id":"6XEpZlr494B_"}},{"cell_type":"markdown","source":"<center> <h3 style=\"background-color:red; color:white\" ><br>2 - Motive<br></h3></center>\n\n\nTrying to connect this problem to some real world scenarios [Click](https://mobidev.biz/blog/exploring-deep-learning-image-captioning)<br> \n* `Image tagging for ecommerce, Photo-Sharing Seervices and Online Catalogs`\n* `Automatic image Annotations for blind peoples`\n* `CCTV Cameras` are everywhere today, but along with viewing world, we can also generate relevant captions, then we can raise alarms as soon as there is some malicious activity going on somewhere(malls, roads). This could probably help reduce some crime and/or accidents\n* `Automatic Captioning` make a Image Search as good as Google Search here every image could be first converted into a caption and then search can be performed based on caption\n","metadata":{"id":"DZ-HiOz7HY9P"}},{"cell_type":"markdown","source":"<center> <h3 style=\"background-color:red; color:white\" ><br>3 - Prerequisites<br></h3></center>\n\nKnow basic Deep Learning concepts before diving in: \n1. Multi-layered Perceptrons\n2. Convolution Neural Networks(CNN)\n3. Recurrent Neural Networks (RNN)\n4. Transfer Learning\n5. Gradient Descent\n6. Backpropagation\n7. Overfitting\n8. Probability\n9. Text Processing\n10. Python (syntax and data structures) \n11. Keras library\n\n**`Encoder`**<br>\nConvolutional Neural Network(CNN) can be thought of as an encoder. Input image is given to CNN to extract features. Last hidden state of CNN is connected to Decoder\n\n**`Decoder`**<br>\nDecoder is a Recurrent Neural Network(RNN) which does language modelling up to word level. First time step receives encoded output from encoder and also <START> vector\n\n[Understanding Encoder and Decoder Link](https://towardsdatascience.com/understanding-encoder-decoder-sequence-to-sequence-model-679e04af4346)\n","metadata":{"id":"ml-COpWiz6dN"}},{"cell_type":"markdown","source":"<center> <h3 style=\"background-color:red; color:white\" ><br>4 - Data Collection<br></h3></center>\n\nPlethora of open source datasets are available for this problem like `Flickr_8k_dataset`(containing8k images), `Flickr_30k_dataset`(containing 30k images), `MS_COCO_dataset`(containing 180k images) more many. <br>\nFor now I am using `Flickr_8k_dataset` ==>[Download  Dataset here](https://forms.illinois.edu/sec/1713398)<br> \nWhy I am using only `Flickr_8k_dataset` not other??? <br>\nI dont have a good harware in my rescure alash. No worry I will do it like you can scale it. <br>\nThis dataset `contains appx8000_images :: 1.1GB apx` `each with 5 captions` ==> (as we have already seen in the Introduction section that an image can have multiple captions, all being relevant simultaneously)<br>\n\n* Dataset will be in form `[image → captions]`\n* Dataset consists of input images and their corresponding output captions\n<br>\n\nThese images are divided into two branches or parts(bifurcated) as follows:\n1. `Test Set` — 1000 images\n2. `Training Set` — 6000 images\n3. `Dev Set` — 1000 images\n","metadata":{"id":"Gh169a4G0nej"}},{"cell_type":"markdown","source":"<center> <h3 style=\"background-color:red; color:white\" ><br>4 - Interpreting Data<br></h3></center>\n\nDownloas data from above provided link \n* Along with images, will also get some text files related to images \n* Dataset will be in form [image → captions]\n* Dataset consists of input images and their corresponding output captions\n\n<br>\n`captions.txt` ==> contains name of each image along with its 5 captions(0-4)<br>\nReading this file as follows\n","metadata":{"id":"X4PHqQXq9G2V"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os","metadata":{"id":"F4MfuqWu9cFO","execution":{"iopub.status.busy":"2023-11-01T08:46:49.302003Z","iopub.execute_input":"2023-11-01T08:46:49.302480Z","iopub.status.idle":"2023-11-01T08:46:49.334390Z","shell.execute_reply.started":"2023-11-01T08:46:49.302358Z","shell.execute_reply":"2023-11-01T08:46:49.333616Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"code","source":"#Image data\npath_img = '../input/flickr-8k-images-with-captions/Images'\njpgs = os.listdir(path_img)\n#first 10 jpgs\nprint(f\"All jpgs count: {len(jpgs)}\\n\")\njpgs[:10]","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:49.336302Z","iopub.execute_input":"2023-11-01T08:46:49.337322Z","iopub.status.idle":"2023-11-01T08:46:49.549283Z","shell.execute_reply.started":"2023-11-01T08:46:49.337274Z","shell.execute_reply":"2023-11-01T08:46:49.548240Z"},"trusted":true},"execution_count":2,"outputs":[{"name":"stdout","text":"All jpgs count: 8091\n\n","output_type":"stream"},{"execution_count":2,"output_type":"execute_result","data":{"text/plain":"['3226254560_2f8ac147ea.jpg',\n '214543992_ce6c0d9f9b.jpg',\n '2366643786_9c9a830db8.jpg',\n '3368819708_0bfa0808f8.jpg',\n '2190227737_6e0bde2623.jpg',\n '2752809449_632cd991b3.jpg',\n '3097776588_312932e438.jpg',\n '1206506157_c7956accd5.jpg',\n '1319634306_816f21677f.jpg',\n '2465218087_fca77998c6.jpg']"},"metadata":{}}]},{"cell_type":"code","source":"#loadng doc into memory function\ndef load_txt(path_txt):\n    #open file as read only\n    file = open(path_txt, 'r')\n    #read all text\n    text = file.read()\n    #close file\n    file.close()\n    return text","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:49.551204Z","iopub.execute_input":"2023-11-01T08:46:49.553181Z","iopub.status.idle":"2023-11-01T08:46:49.560485Z","shell.execute_reply.started":"2023-11-01T08:46:49.553120Z","shell.execute_reply":"2023-11-01T08:46:49.559139Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"code","source":"path_txt = '../input/flickr8k-text/flickr8k.token.txt'\n#loading descriptions\ntext_doc = load_txt(path_txt)\n\nprint(text_doc[:500])","metadata":{"id":"NQILki029KkN","execution":{"iopub.status.busy":"2023-11-01T08:46:49.562553Z","iopub.execute_input":"2023-11-01T08:46:49.563199Z","iopub.status.idle":"2023-11-01T08:46:49.634839Z","shell.execute_reply.started":"2023-11-01T08:46:49.563145Z","shell.execute_reply":"2023-11-01T08:46:49.633726Z"},"trusted":true},"execution_count":4,"outputs":[{"name":"stdout","text":"1000268201_693b08cb0e.jpg#0\tA child in a pink dress is climbing up a set of stairs in an entry way .\n1000268201_693b08cb0e.jpg#1\tA girl going into a wooden building .\n1000268201_693b08cb0e.jpg#2\tA little girl climbing into a wooden playhouse .\n1000268201_693b08cb0e.jpg#3\tA little girl climbing the stairs to her playhouse .\n1000268201_693b08cb0e.jpg#4\tA little girl in a pink dress going into a wooden cabin .\n1001773457_577c3a7d70.jpg#0\tA black dog and a spotted dog are fighting\n1001773457_577c3a7\n","output_type":"stream"}]},{"cell_type":"markdown","source":"`NOTE` <br>\nEvery text line contains `<image name>#n <actual caption>`, where 0 ≤ n ≤ 4 <br>\ni.e. `name of image`, `caption number`(0 to 4) and `actual caption`","metadata":{}},{"cell_type":"code","source":"#reading above text file line after line\ntxt = []\nfor line in text_doc.split('\\n'): #splitting everyline using '/n'\n    #spliting_every_line in words like ==>  '1000268201_693b08cb0e.jpg#0', 'A', 'child', 'in', 'a',\n    spliting_every_line = line.split() #dtype : list\n    #special cases handling\n    if len(spliting_every_line) == 0:\n        continue\n    #imageName_captionNum_of_all_line list to save \n        # 1. all imageName_captionNum, say like this ==> ['1000268201_693b08cb0e.jpg#0']\n        # 2. actual captions,say like this ==> 'A child in a pink dress is climbing up a set of stairs in an entry way .'\n        # 3. joined 1 and 2 using \" \" say like this ==> ['1000268201_693b08cb0e.jpg#0', 'A child in a pink dress is climbing up a set of stairs in an entry way .']\n    imageName_captionNum_of_all_line = []\n    #list indexing for first element of every line ==> this part ['1000268201_693b08cb0e.jpg#0'] of all\n    imageName_captionNum_of_all_line.append(spliting_every_line[0]) #acessing all imageName_captionNum_of_all_line\n    #joining name of image, caption number with actual caption, like this ==> ['1000268201_693b08cb0e.jpg#0', 'A child in a pink dress is climbing up a set of stairs in an entry way .'] \n    imageName_captionNum_of_all_line.append(\" \".join(spliting_every_line[1:]))\n    \n    #spliting using \"#\" in name_of_image#caption_number, output :: ['1000268201_693b08cb0e.jpg', '0']\n    imgName_num_seprated = imageName_captionNum_of_all_line[0].split('#') #acessing & spliting this part => '1000268201_693b08cb0e.jpg#0' using indexing\n    #appending to above list names as txt\n    txt.append(imgName_num_seprated + [imageName_captionNum_of_all_line[1].lower()])\n\n############################################    \n#making text dataframe out of raw text    \ntext_df = pd.DataFrame(txt,columns=['image_name','image_index','image_caption'])\n#reindexing\ntext_df_reindexed = text_df.reindex(columns = ['image_index','image_name','image_caption'])\ntext_df_reindexed_1 = text_df_reindexed[text_df_reindexed.image_caption != '2258277193_586949ec62.jpg.1']\nuni_filename = np.unique(text_df_reindexed_1.image_name.values)","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:49.638040Z","iopub.execute_input":"2023-11-01T08:46:49.639008Z","iopub.status.idle":"2023-11-01T08:46:50.114757Z","shell.execute_reply.started":"2023-11-01T08:46:49.638944Z","shell.execute_reply":"2023-11-01T08:46:50.113452Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"code","source":"#final dataset for text corpus\ntext_df = text_df_reindexed_1\nprint(type(text_df),'\\n')\ntext_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.116622Z","iopub.execute_input":"2023-11-01T08:46:50.116996Z","iopub.status.idle":"2023-11-01T08:46:50.145596Z","shell.execute_reply.started":"2023-11-01T08:46:50.116957Z","shell.execute_reply":"2023-11-01T08:46:50.143447Z"},"trusted":true},"execution_count":6,"outputs":[{"name":"stdout","text":"<class 'pandas.core.frame.DataFrame'> \n\n","output_type":"stream"},{"execution_count":6,"output_type":"execute_result","data":{"text/plain":"  image_index                 image_name  \\\n0           0  1000268201_693b08cb0e.jpg   \n1           1  1000268201_693b08cb0e.jpg   \n2           2  1000268201_693b08cb0e.jpg   \n3           3  1000268201_693b08cb0e.jpg   \n4           4  1000268201_693b08cb0e.jpg   \n\n                                       image_caption  \n0  a child in a pink dress is climbing up a set o...  \n1              a girl going into a wooden building .  \n2   a little girl climbing into a wooden playhouse .  \n3  a little girl climbing the stairs to her playh...  \n4  a little girl in a pink dress going into a woo...  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>image_index</th>\n      <th>image_name</th>\n      <th>image_caption</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>1000268201_693b08cb0e.jpg</td>\n      <td>a child in a pink dress is climbing up a set o...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>1000268201_693b08cb0e.jpg</td>\n      <td>a girl going into a wooden building .</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>1000268201_693b08cb0e.jpg</td>\n      <td>a little girl climbing into a wooden playhouse .</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>1000268201_693b08cb0e.jpg</td>\n      <td>a little girl climbing the stairs to her playh...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>1000268201_693b08cb0e.jpg</td>\n      <td>a little girl in a pink dress going into a woo...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"<center> <h3 style=\"background-color:orange; color:white\" ><br>Alternate & Effective Approach<br></h3></center>\n\n \nMaking dict, here: <br>\n* key : image_name without extension\n* values : all image_caption related with key","metadata":{}},{"cell_type":"code","source":"#function to make dict : here \"key\" ==> image_name without .jpg and #n , \"value\" as all related actual captions to keys\ndef load_text_doc(text_toke_data):\n    mapping = dict()\n    for line in text_toke_data.split('\\n'): #splitting everyline using '/n'\n        #spliting_every_line in words like ==>  '1000268201_693b08cb0e.jpg#0', 'A', 'child', 'in', 'a',\n        spliting_every_line_token = line.split() #dtype : list\n        #special cases handling\n        if len(line) < 2:\n            continue\n        \n        #first token as IMAGE_ID and rest as DESCRIPTION\n        image_id, image_desc = spliting_every_line_token[0], spliting_every_line_token[1:]\n        #taking '1000268201_693b08cb0e' only removing .jpg\n        image_id = image_id.split('.')[0] #output: 1000268201_693b08cb0e\n        image_desc = ' '.join(image_desc) #output: A child in a pink dress is climbing up a set of stairs in an entry way .\n        \n        if image_id not in mapping:\n            mapping[image_id] = list()\n        \n        #storing\n        mapping[image_id].append(image_desc)\n    return mapping\n","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.147632Z","iopub.execute_input":"2023-11-01T08:46:50.147988Z","iopub.status.idle":"2023-11-01T08:46:50.159101Z","shell.execute_reply.started":"2023-11-01T08:46:50.147951Z","shell.execute_reply":"2023-11-01T08:46:50.157843Z"},"trusted":true},"execution_count":7,"outputs":[]},{"cell_type":"code","source":"#dict with key:values pares, key along with all there descriptions\ndesc_dict = load_text_doc(text_doc)\n\nprint(f'Description Dictionary have :{len(desc_dict.keys())}: number of keys')","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.160968Z","iopub.execute_input":"2023-11-01T08:46:50.161349Z","iopub.status.idle":"2023-11-01T08:46:50.330489Z","shell.execute_reply.started":"2023-11-01T08:46:50.161295Z","shell.execute_reply":"2023-11-01T08:46:50.329440Z"},"trusted":true},"execution_count":8,"outputs":[{"name":"stdout","text":"Description Dictionary have :8092: number of keys\n","output_type":"stream"}]},{"cell_type":"code","source":"#accessing all desc_dict.keys() and then making it a list to use indexing and show first 10 keys\nlist(desc_dict.keys())[:10] #first 10 keys","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.331835Z","iopub.execute_input":"2023-11-01T08:46:50.332137Z","iopub.status.idle":"2023-11-01T08:46:50.343537Z","shell.execute_reply.started":"2023-11-01T08:46:50.332102Z","shell.execute_reply":"2023-11-01T08:46:50.342378Z"},"trusted":true},"execution_count":9,"outputs":[{"execution_count":9,"output_type":"execute_result","data":{"text/plain":"['1000268201_693b08cb0e',\n '1001773457_577c3a7d70',\n '1002674143_1b742ab4b8',\n '1003163366_44323f5815',\n '1007129816_e794419615',\n '1007320043_627395c3d8',\n '1009434119_febe49276a',\n '1012212859_01547e3f17',\n '1015118661_980735411b',\n '1015584366_dfcec3c85a']"},"metadata":{}}]},{"cell_type":"code","source":"#number of descriptions for key ==> 1000268201_693b08cb0e\nlen(desc_dict['1000268201_693b08cb0e'])","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.345664Z","iopub.execute_input":"2023-11-01T08:46:50.346467Z","iopub.status.idle":"2023-11-01T08:46:50.360086Z","shell.execute_reply.started":"2023-11-01T08:46:50.346407Z","shell.execute_reply":"2023-11-01T08:46:50.358969Z"},"trusted":true},"execution_count":10,"outputs":[{"execution_count":10,"output_type":"execute_result","data":{"text/plain":"5"},"metadata":{}}]},{"cell_type":"markdown","source":"There are total 5 description for `key:1000268201_693b08cb0e`","metadata":{}},{"cell_type":"code","source":"#descriptions for key ==> 1000268201_693b08cb0e\ndesc_dict['1000268201_693b08cb0e']","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.361661Z","iopub.execute_input":"2023-11-01T08:46:50.361962Z","iopub.status.idle":"2023-11-01T08:46:50.376632Z","shell.execute_reply.started":"2023-11-01T08:46:50.361921Z","shell.execute_reply":"2023-11-01T08:46:50.375568Z"},"trusted":true},"execution_count":11,"outputs":[{"execution_count":11,"output_type":"execute_result","data":{"text/plain":"['A child in a pink dress is climbing up a set of stairs in an entry way .',\n 'A girl going into a wooden building .',\n 'A little girl climbing into a wooden playhouse .',\n 'A little girl climbing the stairs to her playhouse .',\n 'A little girl in a pink dress going into a wooden cabin .']"},"metadata":{}}]},{"cell_type":"code","source":"#description for key ==> 1015584366_dfcec3c85a\ndesc_dict['1015584366_dfcec3c85a']","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.378354Z","iopub.execute_input":"2023-11-01T08:46:50.378698Z","iopub.status.idle":"2023-11-01T08:46:50.394438Z","shell.execute_reply.started":"2023-11-01T08:46:50.378649Z","shell.execute_reply":"2023-11-01T08:46:50.393318Z"},"trusted":true},"execution_count":12,"outputs":[{"execution_count":12,"output_type":"execute_result","data":{"text/plain":"['A black dog leaps over a log .',\n 'A grey dog is leaping over a fallen tree .',\n 'A large black dog leaps a fallen log .',\n 'A mottled black and grey dog in a blue collar jumping over a fallen tree .',\n 'The black dog jumped the tree stump .']"},"metadata":{}}]},{"cell_type":"markdown","source":"`Observation:` <br>\nNow we have image_name without extension i.e(.jpg) along with all associated description to it <br>\n\n","metadata":{}},{"cell_type":"markdown","source":"---\n---","metadata":{}},{"cell_type":"code","source":"import string \n\n#all punctuation  example \nstring.punctuation ","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.396187Z","iopub.execute_input":"2023-11-01T08:46:50.396561Z","iopub.status.idle":"2023-11-01T08:46:50.415199Z","shell.execute_reply.started":"2023-11-01T08:46:50.396520Z","shell.execute_reply":"2023-11-01T08:46:50.414133Z"},"trusted":true},"execution_count":13,"outputs":[{"execution_count":13,"output_type":"execute_result","data":{"text/plain":"'!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"},"metadata":{}}]},{"cell_type":"code","source":"#removing punctuation dummy\ntxt = 'A boy in a green shirt is looking down at many inflatable boats .'\nmytable = txt.maketrans(\" \", \" \", string.punctuation)\nprint(txt.translate(mytable))","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.419497Z","iopub.execute_input":"2023-11-01T08:46:50.419811Z","iopub.status.idle":"2023-11-01T08:46:50.436173Z","shell.execute_reply.started":"2023-11-01T08:46:50.419775Z","shell.execute_reply":"2023-11-01T08:46:50.434982Z"},"trusted":true},"execution_count":14,"outputs":[{"name":"stdout","text":"A boy in a green shirt is looking down at many inflatable boats \n","output_type":"stream"}]},{"cell_type":"markdown","source":"---\n---\n","metadata":{}},{"cell_type":"markdown","source":"## Removing any punctuation and 'a' and 's' from `image_caption`","metadata":{}},{"cell_type":"code","source":"def clean_image_captions(description_dic):\n    \n    for key, image_captions in desc_dict.items():\n        \n        for element_in_image_caption in range(len(image_captions)):\n            \n            #accessing every element(token) of each image_caption related to key\n            image_captions_elements = image_captions[element_in_image_caption]\n            #putting every element(token) in seperate list\n            image_captions_elements_list = image_captions_elements.split()\n            #all in lower case\n            image_captions_elements_list_lower_cased = [element.lower() for element in image_captions_elements_list]\n            #removing punctuations if exist from each token\n            image_captions_elements_punctuations_removed = [element.translate(str.maketrans('', '', string.punctuation)) for element in image_captions_elements_list_lower_cased]\n            #removing elements/tokens/word if it's length is less then 1 ==> focus is to remove \"s\" & \"a\"\n            image_captions_elements_word_cleaned = [element for element in image_captions_elements_punctuations_removed if len(element) > 1]\n            \n            #removing elements/tokens/word if they are alphanumeric ::: testing here can update later\n            image_captions_elements_alphanum_removed = [element for element in image_captions_elements_word_cleaned if element.isalpha()]\n            \n            #storing final usefull image_captions as strings\n            image_captions[element_in_image_caption] = ' '.join(image_captions_elements_alphanum_removed)\n\n##################################\n#calling function for work\nclean_image_captions(desc_dict)","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:50.438223Z","iopub.execute_input":"2023-11-01T08:46:50.438608Z","iopub.status.idle":"2023-11-01T08:46:52.130035Z","shell.execute_reply.started":"2023-11-01T08:46:50.438559Z","shell.execute_reply":"2023-11-01T08:46:52.128990Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"markdown","source":"**`Observation --`** `single letter are removed along with punctuations`","metadata":{}},{"cell_type":"code","source":"#cleared descriptions for key ==> 1000268201_693b08cb0e\ndesc_dict['1000268201_693b08cb0e']","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:52.131743Z","iopub.execute_input":"2023-11-01T08:46:52.133031Z","iopub.status.idle":"2023-11-01T08:46:52.141497Z","shell.execute_reply.started":"2023-11-01T08:46:52.132968Z","shell.execute_reply":"2023-11-01T08:46:52.140393Z"},"trusted":true},"execution_count":16,"outputs":[{"execution_count":16,"output_type":"execute_result","data":{"text/plain":"['child in pink dress is climbing up set of stairs in an entry way',\n 'girl going into wooden building',\n 'little girl climbing into wooden playhouse',\n 'little girl climbing the stairs to her playhouse',\n 'little girl in pink dress going into wooden cabin']"},"metadata":{}}]},{"cell_type":"code","source":"#cleared description for key ==> 1015584366_dfcec3c85a\ndesc_dict['1015584366_dfcec3c85a']","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:52.143980Z","iopub.execute_input":"2023-11-01T08:46:52.144668Z","iopub.status.idle":"2023-11-01T08:46:52.161007Z","shell.execute_reply.started":"2023-11-01T08:46:52.144603Z","shell.execute_reply":"2023-11-01T08:46:52.159998Z"},"trusted":true},"execution_count":17,"outputs":[{"execution_count":17,"output_type":"execute_result","data":{"text/plain":"['black dog leaps over log',\n 'grey dog is leaping over fallen tree',\n 'large black dog leaps fallen log',\n 'mottled black and grey dog in blue collar jumping over fallen tree',\n 'the black dog jumped the tree stump']"},"metadata":{}}]},{"cell_type":"markdown","source":"### Saving processed image_caption text","metadata":{}},{"cell_type":"code","source":"# function to save processed desc_dict to a new file, one per line\ndef saving_processed_desc_dict(descriptions_dict, filename):\n    lines = list()\n    for key, desc_list in desc_dict.items():\n        for desc in desc_dict:\n            lines.append(key + ' ' + desc)\n    data = '\\n'.join(lines)\n    file = open(filename, 'w')\n    file.write(data)\n    file.close()\n\n###########################\nsaving_processed_desc_dict(desc_dict, 'desc_dict_with_processed_image_caption.txt')\n","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:46:52.162936Z","iopub.execute_input":"2023-11-01T08:46:52.163506Z","iopub.status.idle":"2023-11-01T08:47:37.113086Z","shell.execute_reply.started":"2023-11-01T08:46:52.163453Z","shell.execute_reply":"2023-11-01T08:47:37.111246Z"},"trusted":true},"execution_count":18,"outputs":[]},{"cell_type":"markdown","source":"### Making img_captions_vocabulary","metadata":{}},{"cell_type":"code","source":"#function to make a set of vocab out of image_captions\ndef img_captions_vocabulary(descriptions):\n    \n    #set to store vocab which we will get out of image_captions of each key\n    image_captions_vocabulary_set = set()\n    \n    for key in descriptions.keys():\n        [image_captions_vocabulary_set.update(each_image_captions.split()) for each_image_captions in descriptions[key]]\n    \n    return image_captions_vocabulary_set\n\n###########################################\n#storing vocabulary in variable\nimage_captions_vocabulary = img_captions_vocabulary(desc_dict) #Result:: {'saddled', 'derby', 'facedown', 'bullbranded', 'obsured', 'cube'} \nprint(f'Final image_captions_vocabulary Size: {len(image_captions_vocabulary)}')","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:37.115812Z","iopub.execute_input":"2023-11-01T08:47:37.118271Z","iopub.status.idle":"2023-11-01T08:47:37.213556Z","shell.execute_reply.started":"2023-11-01T08:47:37.118202Z","shell.execute_reply":"2023-11-01T08:47:37.212172Z"},"trusted":true},"execution_count":19,"outputs":[{"name":"stdout","text":"Final image_captions_vocabulary Size: 8763\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Loading training file\nHere one will get image name with `.jpg` extension, each seprated with `\\n` <br>\nAll images name have number in them like ==> `'2513260012_03d33305cf.jpg\\n2903617548_d3e38d7f88.jpg`","metadata":{}},{"cell_type":"code","source":"#loadng doc into memory function :: making this function again just to make things more clear\ndef load_txt(path_txt):\n    \n    #open file as read only\n    file = open(path_txt, 'r')\n    #read all text\n    text = file.read()\n    #close file\n    file.close()\n    return text\n\n#######################################\n#loading train data\ntrain_path = '../input/flickr8k-text/'\ntrain_data = load_txt(train_path+'flickr_8k.trainImages.txt')\nprint(train_data[:200])","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:37.215620Z","iopub.execute_input":"2023-11-01T08:47:37.216249Z","iopub.status.idle":"2023-11-01T08:47:43.979174Z","shell.execute_reply.started":"2023-11-01T08:47:37.216182Z","shell.execute_reply":"2023-11-01T08:47:43.977801Z"},"trusted":true},"execution_count":20,"outputs":[{"name":"stdout","text":"2513260012_03d33305cf.jpg\n2903617548_d3e38d7f88.jpg\n3338291921_fe7ae0c8f8.jpg\n488416045_1c6d903fe0.jpg\n2644326817_8f45080b87.jpg\n218342358_1755a9cce1.jpg\n2501968935_02f2cd8079.jpg\n2699342860_5288e203e\n","output_type":"stream"}]},{"cell_type":"code","source":"#if you want to see new line statement by yourself\n###train_data","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:43.980647Z","iopub.execute_input":"2023-11-01T08:47:43.982968Z","iopub.status.idle":"2023-11-01T08:47:43.992173Z","shell.execute_reply.started":"2023-11-01T08:47:43.982851Z","shell.execute_reply":"2023-11-01T08:47:43.990017Z"},"trusted":true},"execution_count":21,"outputs":[]},{"cell_type":"markdown","source":"Actual values are seprated using `\\n` like this : `'2513260012_03d33305cf.jpg\\n2903617548_d3e38d7f88.jpg\\n3338291921_fe7ae0c8f8.jpg\\n`\n\nOne have to split using `\\n` to seprate images txt","metadata":{}},{"cell_type":"markdown","source":"### Preprocessing Loaded Training File","metadata":{"execution":{"iopub.status.busy":"2022-05-01T04:42:04.866879Z","iopub.execute_input":"2022-05-01T04:42:04.867225Z","iopub.status.idle":"2022-05-01T04:42:04.880744Z","shell.execute_reply.started":"2022-05-01T04:42:04.867191Z","shell.execute_reply":"2022-05-01T04:42:04.880084Z"},"jupyter":{"outputs_hidden":true}}},{"cell_type":"code","source":"#function to load and extract train_data image_id only by removing `.jpg` extension and removing empty strings\ndef load_process_train_data(train_txt_file):\n    train_data_final = list()\n\n    #using above function to load train_data\n    doc = load_txt(path_txt=train_txt_file)\n    \n    #spliting image names using `\\n` and then accessing every image name\n    for every_img_desc in doc.split('\\n'):\n        \n        #to remove any empty string\n        if len(every_img_desc) < 1:\n            continue\n        \n        #removing `.jpg` extension or getting image id\n        image_id = every_img_desc.split('.')[0]\n        train_data_final.append(image_id)\n    \n    return set(train_data_final)\n\n############################\nprocessed_train_data = load_process_train_data(train_txt_file=train_path+'flickr_8k.trainImages.txt') #dtype :: set\nprint(f'Total image_id in train_dataset are : {len(processed_train_data)}\\n\\n')\n\n\nimport random\n#see any 5 image_id of train set\nprint(f\"5 Randome train_id's: \\n{random.sample(processed_train_data,5)}\")","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:43.998029Z","iopub.execute_input":"2023-11-01T08:47:43.999905Z","iopub.status.idle":"2023-11-01T08:47:44.031941Z","shell.execute_reply.started":"2023-11-01T08:47:43.999770Z","shell.execute_reply":"2023-11-01T08:47:44.030882Z"},"trusted":true},"execution_count":22,"outputs":[{"name":"stdout","text":"Total image_id in train_dataset are : 6000\n\n\n5 Randome train_id's: \n['3124549928_10904a5a83', '3138562460_44227a35cf', '265528702_8653eab9fa', '378453580_21d688748e', '3265209567_b3b9c8e0fe']\n","output_type":"stream"}]},{"cell_type":"markdown","source":"### Loading and reading image files from Images caption file\n`using glob module to retrieve files/pathnames matching a specified pattern `.jpg``","metadata":{}},{"cell_type":"code","source":"#using glob module to retrieve files/pathnames matching a specified pattern `.jpg`\nimport glob\n\n#below path contains all images directory\nimages_data_path = '../input/flickr-8k-images-with-captions/Images/'\n#create a list of all image names inside directory Images\nevery_image_full_path = glob.glob(images_data_path + '*.jpg')\n\ntype(every_image_full_path)","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:44.033778Z","iopub.execute_input":"2023-11-01T08:47:44.036542Z","iopub.status.idle":"2023-11-01T08:47:44.776722Z","shell.execute_reply.started":"2023-11-01T08:47:44.036319Z","shell.execute_reply":"2023-11-01T08:47:44.775546Z"},"trusted":true},"execution_count":23,"outputs":[{"execution_count":23,"output_type":"execute_result","data":{"text/plain":"list"},"metadata":{}}]},{"cell_type":"markdown","source":"### All images in train data file\nLoading train files","metadata":{}},{"cell_type":"code","source":"train_images_all = set(open(train_path+'flickr_8k.trainImages.txt', 'r').read().strip().split('\\n'))\n\ntrain_images_after_check = []\nfor particular_image_full_path in every_image_full_path:\n    \n    if particular_image_full_path[len(images_data_path):] in train_images_all:\n        train_images_after_check.append(particular_image_full_path)\n        \n#first 3 images with there path :: list of all train images\ntrain_images_after_check[:3]","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:44.778503Z","iopub.execute_input":"2023-11-01T08:47:44.778903Z","iopub.status.idle":"2023-11-01T08:47:45.364852Z","shell.execute_reply.started":"2023-11-01T08:47:44.778840Z","shell.execute_reply":"2023-11-01T08:47:45.363570Z"},"trusted":true},"execution_count":24,"outputs":[{"execution_count":24,"output_type":"execute_result","data":{"text/plain":"['../input/flickr-8k-images-with-captions/Images/3226254560_2f8ac147ea.jpg',\n '../input/flickr-8k-images-with-captions/Images/214543992_ce6c0d9f9b.jpg',\n '../input/flickr-8k-images-with-captions/Images/3368819708_0bfa0808f8.jpg']"},"metadata":{}}]},{"cell_type":"markdown","source":"### All images in test data file\nLoading test files","metadata":{}},{"cell_type":"code","source":"test_images_all = set(open(train_path+'flickr_8k.testImages.txt', 'r').read().strip().split('\\n'))\n\ntest_images_after_check = []\nfor particular_image_full_path in every_image_full_path:\n    \n    if particular_image_full_path[len(images_data_path):] in test_images_all:\n        test_images_after_check.append(particular_image_full_path)\n        \n#first 3 images with there path :: list of all test images\ntest_images_after_check[:3]","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:45.367459Z","iopub.execute_input":"2023-11-01T08:47:45.368361Z","iopub.status.idle":"2023-11-01T08:47:45.396097Z","shell.execute_reply.started":"2023-11-01T08:47:45.368299Z","shell.execute_reply":"2023-11-01T08:47:45.394747Z"},"trusted":true},"execution_count":25,"outputs":[{"execution_count":25,"output_type":"execute_result","data":{"text/plain":"['../input/flickr-8k-images-with-captions/Images/280706862_14c30d734a.jpg',\n '../input/flickr-8k-images-with-captions/Images/929679367_ff8c7df2ee.jpg',\n '../input/flickr-8k-images-with-captions/Images/1317292658_ba29330a0b.jpg']"},"metadata":{}}]},{"cell_type":"code","source":"path_txt = '../input/flickr8k-text/flickr8k.token.txt'\n\n#loadng doc into memory function :: making this function again just to make things more clear\ndef load_txt(path_txt):\n    #open file as read only\n    file = open(path_txt, 'r')\n    #read all text\n    text = file.read()\n    #close file\n    file.close()\n    return text\n\n#loading clear description into memory\ndef load_clean_descriptions(path_txt, dataset):\n    doc = load_txt(path_txt)\n    \n    descriptions = dict()\n    for line in doc.split('\\n'):\n        #spliting lines by white space\n        tokens = line.split()\n        #split id from description\n        image_id, image_desc = tokens[0], tokens[1:]\n        \n        if image_id in dataset:\n            \n            if image_id not in descriptions:\n                descriptions[image_id] = list()\n            \n            #wraping description in token\n            desc = 'start ' + ' '.join(image_desc) + ' end'\n            #store\n            descriptions[image_id].append(desc)\n    return descriptions\n\n##using function load_clean_descriptions\ntrain_descriptions = load_clean_descriptions('./desc_dict_with_processed_image_caption.txt', processed_train_data)\nprint(len(train_descriptions))\n","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:47:45.398111Z","iopub.execute_input":"2023-11-01T08:47:45.398487Z","iopub.status.idle":"2023-11-01T08:49:45.036315Z","shell.execute_reply.started":"2023-11-01T08:47:45.398436Z","shell.execute_reply":"2023-11-01T08:49:45.035060Z"},"trusted":true},"execution_count":26,"outputs":[{"name":"stdout","text":"6000\n","output_type":"stream"}]},{"cell_type":"code","source":"#preprocessing image paths\ndef preprocess(image_path):\n    img = image.load_img(image_path, target_size=(299,299))\n    x = image.img_array(img)\n    x = np.expand_dims(x,axis=0)\n    x = preprocess_input(x)\n    return(x)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:49:45.038323Z","iopub.execute_input":"2023-11-01T08:49:45.038720Z","iopub.status.idle":"2023-11-01T08:49:45.048084Z","shell.execute_reply.started":"2023-11-01T08:49:45.038667Z","shell.execute_reply":"2023-11-01T08:49:45.046981Z"},"trusted":true},"execution_count":27,"outputs":[]},{"cell_type":"code","source":"for i in range(8):\n    print(i)\n    for j in range(8):\n        print(i)\n        print(j)\n    ","metadata":{"execution":{"iopub.status.busy":"2023-11-01T08:49:45.049951Z","iopub.execute_input":"2023-11-01T08:49:45.050703Z","iopub.status.idle":"2023-11-01T08:49:45.086204Z","shell.execute_reply.started":"2023-11-01T08:49:45.050635Z","shell.execute_reply":"2023-11-01T08:49:45.085211Z"},"trusted":true},"execution_count":28,"outputs":[{"name":"stdout","text":"0\n0\n0\n0\n1\n0\n2\n0\n3\n0\n4\n0\n5\n0\n6\n0\n7\n1\n1\n0\n1\n1\n1\n2\n1\n3\n1\n4\n1\n5\n1\n6\n1\n7\n2\n2\n0\n2\n1\n2\n2\n2\n3\n2\n4\n2\n5\n2\n6\n2\n7\n3\n3\n0\n3\n1\n3\n2\n3\n3\n3\n4\n3\n5\n3\n6\n3\n7\n4\n4\n0\n4\n1\n4\n2\n4\n3\n4\n4\n4\n5\n4\n6\n4\n7\n5\n5\n0\n5\n1\n5\n2\n5\n3\n5\n4\n5\n5\n5\n6\n5\n7\n6\n6\n0\n6\n1\n6\n2\n6\n3\n6\n4\n6\n5\n6\n6\n6\n7\n7\n7\n0\n7\n1\n7\n2\n7\n3\n7\n4\n7\n5\n7\n6\n7\n7\n","output_type":"stream"}]}]}