vader-sentiment.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2022-06-06T19:00:29.468567Z","iopub.execute_input":"2022-06-06T19:00:29.469191Z","iopub.status.idle":"2022-06-06T19:00:29.507750Z","shell.execute_reply.started":"2022-06-06T19:00:29.469040Z","shell.execute_reply":"2022-06-06T19:00:29.506742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction\n\nSentiment analysis refers to the use of natural language processing, text analysis, computational linguistics, and many more to identify and quantify the sentiment of some kind of text or audio.\n\n\nThere are two major techniques for sentiment analysis :-\n\n• Supervised machine learning\n\n• Unsupervised lexicon-based \n\n\nOften, you may not have the convenience of a well-labeled training dataset. \nIn those situations, you need to use unsupervised techniques for predicting the sentiment by using knowledgebases, ontologies, databases, and lexicons that have detailed information specially curated and prepared just for sentiment analysis.\n\n\nHere we use unsupervised lexicon based approach based on Vader lexicon for sentiment analysis.","metadata":{}},{"cell_type":"code","source":"# install vader if not already available\n!pip install vaderSentiment","metadata":{"execution":{"iopub.status.busy":"2022-07-24T04:04:51.036899Z","iopub.execute_input":"2022-07-24T04:04:51.037360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\nanalyser = SentimentIntensityAnalyzer()","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.status.busy":"2022-06-06T19:00:58.490823Z","iopub.execute_input":"2022-06-06T19:00:58.491314Z","iopub.status.idle":"2022-06-06T19:00:58.516487Z","shell.execute_reply.started":"2022-06-06T19:00:58.491272Z","shell.execute_reply":"2022-06-06T19:00:58.515337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def sentiment_analyzer_scores(sentence):\n    score = analyser.polarity_scores(sentence)\n    print(\"{:-<40} {}\".format(sentence, str(score)))","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:00.105636Z","iopub.execute_input":"2022-06-06T19:01:00.106089Z","iopub.status.idle":"2022-06-06T19:01:00.119722Z","shell.execute_reply.started":"2022-06-06T19:01:00.106040Z","shell.execute_reply":"2022-06-06T19:01:00.118002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sentiment_analyzer_scores(\"I am doing good!.\")","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:16.123340Z","iopub.execute_input":"2022-06-06T19:01:16.124158Z","iopub.status.idle":"2022-06-06T19:01:16.130621Z","shell.execute_reply.started":"2022-06-06T19:01:16.124112Z","shell.execute_reply":"2022-06-06T19:01:16.129150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Punctuation\nThe use of an exclamation mark(!), increases the magnitude of the intensity without modifying the semantic orientation. \nFor example, “The food here is good!” is more intense than “The food here is good.” and an increase in the number of (!), increases the magnitude accordingly.","metadata":{}},{"cell_type":"code","source":"sentiment_analyzer_scores(\"The phone is super cool!\")\nsentiment_analyzer_scores(\"The phone is super cool!!\")\nsentiment_analyzer_scores(\"The phone is super cool!!!\")","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:25.411236Z","iopub.execute_input":"2022-06-06T19:01:25.411708Z","iopub.status.idle":"2022-06-06T19:01:25.417956Z","shell.execute_reply.started":"2022-06-06T19:01:25.411672Z","shell.execute_reply":"2022-06-06T19:01:25.417139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Capitalization\n\nUsing upper case letters to emphasize a sentiment-relevant word in the presence of other non-capitalized words, increases the magnitude of the sentiment intensity. \nFor example, “The food here is GREAT!” conveys more intensity than “The food here is great!”","metadata":{}},{"cell_type":"code","source":"sentiment_analyzer_scores(\"The phone is super COOL!\")","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:34.924655Z","iopub.execute_input":"2022-06-06T19:01:34.925170Z","iopub.status.idle":"2022-06-06T19:01:34.931310Z","shell.execute_reply.started":"2022-06-06T19:01:34.925131Z","shell.execute_reply":"2022-06-06T19:01:34.930329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Degree Modifiers\n\nAlso called intensifiers, they impact the sentiment intensity by either increasing or decreasing the intensity. For example, “The service here is extremely good” is more intense than “The service here is good”, whereas “The service here is marginally good” reduces the intensity.","metadata":{}},{"cell_type":"code","source":"sentiment_analyzer_scores(\"Food here is good.\")\nsentiment_analyzer_scores(\"Food here is moderately good.\")\nsentiment_analyzer_scores(\"Food here is extremely good.\")","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:40.554574Z","iopub.execute_input":"2022-06-06T19:01:40.555637Z","iopub.status.idle":"2022-06-06T19:01:40.562714Z","shell.execute_reply.started":"2022-06-06T19:01:40.555595Z","shell.execute_reply":"2022-06-06T19:01:40.561138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conjunctions\n\nUse of conjunctions like “but” signals a shift in sentiment polarity, with the sentiment of the text following the conjunction being dominant. “The food here is great, but the service is horrible” has mixed sentiment, with the latter half dictating the overall rating.","metadata":{}},{"cell_type":"code","source":"sentiment_analyzer_scores(\"Food here is extremely good but service is horrible.\")","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:51.700480Z","iopub.execute_input":"2022-06-06T19:01:51.701133Z","iopub.status.idle":"2022-06-06T19:01:51.709865Z","shell.execute_reply.started":"2022-06-06T19:01:51.701078Z","shell.execute_reply":"2022-06-06T19:01:51.708726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preceding Tri-gram\n\nBy examining the tri-gram preceding a sentiment-laden lexical feature, we catch nearly 90% of cases where negation flips the polarity of the text. A negated sentence would be “The food here isn’t really all that great”.","metadata":{}},{"cell_type":"code","source":"sentiment_analyzer_scores(\"The food here isn’t really all that great\")\nsentiment_analyzer_scores(\"The food here isn’t that great\")\n\nsentiment_analyzer_scores(\"The food here is not really all that great\")\nsentiment_analyzer_scores(\"The food here is not that great\")\n","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:55.210500Z","iopub.execute_input":"2022-06-06T19:01:55.210977Z","iopub.status.idle":"2022-06-06T19:01:55.217269Z","shell.execute_reply.started":"2022-06-06T19:01:55.210944Z","shell.execute_reply":"2022-06-06T19:01:55.216432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Handling Emojis, Slangs and Emoticons\n\nVADER performs very well with emojis, slangs and acronyms in sentences. \n\nLet us see each with an example.","metadata":{}},{"cell_type":"code","source":"print(sentiment_analyzer_scores('I am 😄 today'))\nprint(sentiment_analyzer_scores('😊'))\nprint(sentiment_analyzer_scores('😥'))\nprint(sentiment_analyzer_scores('☹️'))","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:01:58.265889Z","iopub.execute_input":"2022-06-06T19:01:58.266366Z","iopub.status.idle":"2022-06-06T19:01:58.274491Z","shell.execute_reply.started":"2022-06-06T19:01:58.266332Z","shell.execute_reply":"2022-06-06T19:01:58.273077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Slangs","metadata":{}},{"cell_type":"code","source":"print(sentiment_analyzer_scores(\"Today SUX!\"))\nprint(sentiment_analyzer_scores(\"Today only kinda sux! But I'll get by, lol\"))","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:02:01.523438Z","iopub.execute_input":"2022-06-06T19:02:01.524442Z","iopub.status.idle":"2022-06-06T19:02:01.529826Z","shell.execute_reply.started":"2022-06-06T19:02:01.524405Z","shell.execute_reply":"2022-06-06T19:02:01.529066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Emoticons","metadata":{}},{"cell_type":"code","source":"print(sentiment_analyzer_scores(\"Make sure you :) or :D today!\"))","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:02:03.451132Z","iopub.execute_input":"2022-06-06T19:02:03.452227Z","iopub.status.idle":"2022-06-06T19:02:03.457578Z","shell.execute_reply.started":"2022-06-06T19:02:03.452186Z","shell.execute_reply":"2022-06-06T19:02:03.456452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Processing Multiple Sentences","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nscores =[]\nsentences = [\"A really bad, horrible book.\",\"A good, awesome, wonderful, cool book !!!  :)\"]\n\nfor sentence in sentences:\n    score = analyser.polarity_scores(sentence)\n    scores.append(score)\n    \n#Converting List of Dictionaries into Dataframe\ndataFrame= pd.DataFrame(scores)\n\nprint(dataFrame)\n\nprint(\"Overall Sentiment Score for the multiple sentences :- \",dataFrame.mean())","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:02:09.763564Z","iopub.execute_input":"2022-06-06T19:02:09.763930Z","iopub.status.idle":"2022-06-06T19:02:09.790197Z","shell.execute_reply.started":"2022-06-06T19:02:09.763903Z","shell.execute_reply":"2022-06-06T19:02:09.789412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=pd.read_csv(\"/kaggle/input/gameloft-android-games-collection-2022/data.csv\")\nr=pd.read_csv(\"/kaggle/input/gameloft-android-games-collection-2022/reviews.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:02:55.410514Z","iopub.execute_input":"2022-06-06T19:02:55.411262Z","iopub.status.idle":"2022-06-06T19:02:55.432857Z","shell.execute_reply.started":"2022-06-06T19:02:55.411224Z","shell.execute_reply":"2022-06-06T19:02:55.431561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"com.gameloft.android.ANMP.GloftA8HM","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"as8=r[r['appId']==\"com.gameloft.android.ANMP.GloftA8HM\"]['reviews']","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:04:10.812142Z","iopub.execute_input":"2022-06-06T19:04:10.812621Z","iopub.status.idle":"2022-06-06T19:04:10.819656Z","shell.execute_reply.started":"2022-06-06T19:04:10.812584Z","shell.execute_reply":"2022-06-06T19:04:10.818556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import ast","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:16:48.517638Z","iopub.execute_input":"2022-06-06T19:16:48.518053Z","iopub.status.idle":"2022-06-06T19:16:48.523522Z","shell.execute_reply.started":"2022-06-06T19:16:48.518021Z","shell.execute_reply":"2022-06-06T19:16:48.522538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores =[]\nsentences = data\n\nfor sentence in sentences:\n    score = analyser.polarity_scores(sentence)\n    scores.append((sentence,score))\n    \n#Converting List of Dictionaries into Dataframe\ndataF= pd.DataFrame(scores)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataF[1][0]","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:18:57.109885Z","iopub.execute_input":"2022-06-06T19:18:57.110294Z","iopub.status.idle":"2022-06-06T19:18:57.118686Z","shell.execute_reply.started":"2022-06-06T19:18:57.110264Z","shell.execute_reply":"2022-06-06T19:18:57.117283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df = (pd.DataFrame(dataF[1].tolist(), index=dataF[0])\n            .reset_index()\n            .reset_index(drop=True))\nnew_df","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:27:04.273786Z","iopub.execute_input":"2022-06-06T19:27:04.274238Z","iopub.status.idle":"2022-06-06T19:27:04.302518Z","shell.execute_reply.started":"2022-06-06T19:27:04.274207Z","shell.execute_reply":"2022-06-06T19:27:04.301760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Process(df,reviews):\n    out=False\n    app_list=list(df['appId'])\n    for appId in app_list:\n        review_list=get_reviews(appId,reviews)\n        scores= get_scores(appId,review_list)\n        if out==False:\n            result=scores.copy()\n            out=True\n        else :\n            result=result.append(scores)\n        \n    result=result.rename(columns={0: \"reviews\"})\n    result=result.reset_index()\n        \n    return result","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:59:09.324342Z","iopub.execute_input":"2022-06-06T19:59:09.324789Z","iopub.status.idle":"2022-06-06T19:59:09.332460Z","shell.execute_reply.started":"2022-06-06T19:59:09.324757Z","shell.execute_reply":"2022-06-06T19:59:09.331310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_reviews(appId,reviews):\n    appReviews=reviews[reviews['appId']==appId]['reviews']\n    appReviews=list(appReviews)[0]\n    appReviews=ast.literal_eval(appReviews)\n    return appReviews","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:36:53.558445Z","iopub.execute_input":"2022-06-06T19:36:53.558976Z","iopub.status.idle":"2022-06-06T19:36:53.565968Z","shell.execute_reply.started":"2022-06-06T19:36:53.558939Z","shell.execute_reply":"2022-06-06T19:36:53.564504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_scores(appId,review_list):\n    scores =[]\n    sentences = review_list\n    \n    for sentence in sentences:\n        score = analyser.polarity_scores(sentence)\n        scores.append((sentence,score))\n        \n    dataF= pd.DataFrame(scores)\n    new_df = (pd.DataFrame(dataF[1].tolist(), index=dataF[0])\n            .reset_index()\n            .reset_index(drop=True))\n\n    new_df['appId']=[appId]*len(review_list)\n    \n    return new_df","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:37:46.692539Z","iopub.execute_input":"2022-06-06T19:37:46.693000Z","iopub.status.idle":"2022-06-06T19:37:46.700530Z","shell.execute_reply.started":"2022-06-06T19:37:46.692967Z","shell.execute_reply":"2022-06-06T19:37:46.699338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fggg=Process(df,r)\nfggg","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:59:12.789388Z","iopub.execute_input":"2022-06-06T19:59:12.789816Z","iopub.status.idle":"2022-06-06T19:59:14.641765Z","shell.execute_reply.started":"2022-06-06T19:59:12.789782Z","shell.execute_reply":"2022-06-06T19:59:14.628799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:55:08.777689Z","iopub.execute_input":"2022-06-06T19:55:08.778187Z","iopub.status.idle":"2022-06-06T19:55:08.783320Z","shell.execute_reply.started":"2022-06-06T19:55:08.778150Z","shell.execute_reply":"2022-06-06T19:55:08.782299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fggg","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:59:18.651943Z","iopub.execute_input":"2022-06-06T19:59:18.652787Z","iopub.status.idle":"2022-06-06T19:59:18.674224Z","shell.execute_reply.started":"2022-06-06T19:59:18.652724Z","shell.execute_reply":"2022-06-06T19:59:18.673269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set(rc={'figure.figsize':(25,30)})\nfor i, column in enumerate([\"neg\",\"neu\",\"pos\",\"compound\"], 1):\n    plt.subplot(4,1,i)\n    p=sns.histplot(x=column,data=fggg,hue=\"appId\",stat='count',kde=True,palette='magma')\n    p.legend(loc='center left', bbox_to_anchor=(1, 1),ncol=4)","metadata":{"execution":{"iopub.status.busy":"2022-06-06T20:04:41.897319Z","iopub.execute_input":"2022-06-06T20:04:41.897736Z","iopub.status.idle":"2022-06-06T20:04:53.570000Z","shell.execute_reply.started":"2022-06-06T20:04:41.897705Z","shell.execute_reply":"2022-06-06T20:04:53.568851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fggg.groupby(\"appId\")[\"neg\",\"neu\",\"pos\",\"compound\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-06-06T19:47:58.650278Z","iopub.execute_input":"2022-06-06T19:47:58.650701Z","iopub.status.idle":"2022-06-06T19:47:58.683472Z","shell.execute_reply.started":"2022-06-06T19:47:58.650672Z","shell.execute_reply":"2022-06-06T19:47:58.682588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Note\n\nPlease share, upvote and comment to help me create and share more content for the community.\nThank you all.","metadata":{}}]}