Historical, Live Data related API's not working

[divyankm]

Issue description

The Historical, Live Stock and F&O Api's not working post 11 AM Yesterday.
Response is coming empty.

Stock Quote

# Download stock data to pandas dataframe
from jugaad_data.nse import stock_df
df = stock_df(symbol="SBIN", from_date=date(2020,1,1),
            to_date=date(2020,1,30), series="EQ")
df

Error

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    973         try:
--> 974             return complexjson.loads(self.text, **kwargs)
    975         except JSONDecodeError as e:

13 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    976             # Catch JSON-related errors and raise as requests.JSONDecodeError
    977             # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978             raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    979 
    980     @property

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Live Data, market Quote

from jugaad_data.nse import NSELive

n = NSELive()
status = n.market_status()

# Print the raw response text
print("Raw response:", status.text)  # Check if status has a .text attribute

Error:

--------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    973         try:
--> 974             return complexjson.loads(self.text, **kwargs)
    975         except JSONDecodeError as e:

7 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    976             # Catch JSON-related errors and raise as requests.JSONDecodeError
    977             # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978             raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    979 
    980     @property

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Code

from jugaad_data.nse import NSELive
from pandas import json_normalize
n = NSELive()
quotes = n.stock_quote_fno("TCS")
quotes

ERROR:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/requests/models.py](https://localhost:8080/#) in json(self, **kwargs)
    973         try:
--> 974             return complexjson.loads(self.text, **kwargs)
    975         except JSONDecodeError as e:

7 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/requests/models.py](https://localhost:8080/#) in json(self, **kwargs)
    976             # Catch JSON-related errors and raise as requests.JSONDecodeError
    977             # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978             raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    979 
    980     @property

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[bharathg017]

I am also facing the same issue.
Request for guidance from experienced persons in this

[divyankm]

Seems like data is coming encoded format from api's from live.py resulting in error.

Encoded Api response:

Code

from datetime import datetime
from requests import Session

class NSELive:
    time_out = 5
    base_url = "https://www.nseindia.com/api"
    page_url = "https://www.nseindia.com/get-quotes/equity?symbol=LT"
    _routes = {
        "market_status": "/marketStatus",
        # Add other routes as necessary
    }

    def __init__(self):
        self.s = Session()
        h = {
            "Host": "www.nseindia.com",
            "Referer": "https://www.nseindia.com/get-quotes/equity?symbol=SBIN",
            "X-Requested-With": "XMLHttpRequest",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate, br",
            "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
        }
        self.s.headers.update(h)
        self.s.get(self.page_url)  # Initial request to set cookies

    def get(self, route, payload={}):
        url = self.base_url + self._routes[route]
        r = self.s.get(url, params=payload)
        print("URL:", url)
        print("Response Status Code:", r.status_code)
        print("Raw Response:", r.text)
        
        try:
            return r.json()
        except Exception as e:
            print("Failed to decode JSON:", e)
            return None

    def market_status(self):
        return self.get("market_status", {})

# Test the NSELive class
if __name__ == "__main__":
    n = NSELive()
    market_status = n.market_status()
    print("Market Status:", market_status)

Output:

URL: https://www.nseindia.com/api/marketStatus
Response Status Code: 200
�ikh��O-l�Q���{�ڡΧ)��R��Հ�|� ��ీX>��W߶}R��g����(��M�S{��I�R�������ބk�@�?h�e�C������1�7�]��ѷ����Z���QZ��0��V_�&��2A`.���������r���OZ���,�(����t��x�7��f��u��/_���=>·4�A���B8��p!<���'�i`I����zAJ������$E��i$��������X�r�ve��������9һTO:y� N�d@��V��b��z��˺��)�O���4������e�<K��۳�
� +;������t�iH��h���hƑ��뻧����
Failed to decode JSON: Expecting value: line 1 column 2 (char 1)
Market Status: None

Normal Request using requests showing output:

Code

import requests
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'}
# url = 'https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY'
url = 'https://www.nseindia.com/api/marketStatus'

with requests.session() as s:
    data = s.get(url, headers=headers).json()
    print(type(data))
    print(data)

Output:

<class 'dict'>
{'marketState': [{'market': 'Capital Market', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024 11:04', 'index': 'NIFTY 50', 'last': 25068.55, 'variation': -59.400000000001455, 'percentChange': -0.24, 'marketStatusMessage': 'Normal Market is Open'}, {'market': 'Currency', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'Commodity', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'Debt', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'currencyfuture', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '84.1100', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open', 'expiryDate': '29-Oct-2024', 'underlying': 'USDINR', 'updated_time': '15-Oct-2024 11:02', 'tradeDateFormatted': '15-Oct-2024 11:02', 'slickclass': 'slick-item'}], 'marketcap': {'timeStamp': '14-Oct-2024', 'marketCapinTRDollars': 5.47, 'marketCapinLACCRRupees': 459.9931470995962, 'marketCapinCRRupees': 45999314.71, 'marketCapinCRRupeesFormatted': '45,999,314.71', 'marketCapinLACCRRupeesFormatted': '459.99', 'underlying': 'Market Cap'}, 'indicativenifty50': {'dateTime': '14-Oct-2024 15:30', 'indicativeTime': None, 'indexName': 'NIFTY 50', 'indexLast': None, 'indexPercChange': None, 'indexTimeVal': None, 'closingValue': 25127.95, 'finalClosingValue': 25127.95, 'change': 163.71, 'perChange': 0.66, 'status': 'CLOSE'}}

[aman15012]

Just try replacing the header dictionary in jugaad_data/nse/history.py with the header you have mentioned. It works for me.

[divyankm]

Earlier till Yesterday, with existing headers, data was coming in original format for all cloud providers wherever the api is running. Now data is coming in encoded format for original header , due to which api's are failing.

Existing Header:

        h = {
            "Host": "www.nseindia.com",
            "Referer": "https://www.nseindia.com/get-quotes/equity?symbol=SBIN",
            "X-Requested-With": "XMLHttpRequest",
            "pragma": "no-cache",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate, br",
            "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            }

Below Header: Only working in Local Machine:

The issue with this header: headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'} is API request are getting blocked by NSE as its giving IP of Cloud Providers, NSE Blocks all IP's of Cloud Providers (AWS, GCP, Azure).

[gopikrishnan17]

Facing the same issue, please guide in resolving the same

[k-prabin]

How to solve this. Please guide

[amritjain123]

Hi All,
I resolved this by navigating to live.py file in installation directory and locating live.py file.(use pip show jugaad-data)
In that file locate header dictoniary by 'h' name.
and replace "Accept-Encoding": "gzip, deflate, br" with this one "Accept-Encoding": "gzip, deflate",
It starts running for me.

I hope this helps temporarily until fix is released.

[sahvinay]

Hi amritjain123, Great!!! Thanks for temp solution.
Could you please elaborate how removing "br" for Accept-Encoding solve this issue?
I'm still in learning phase, just curious.

To all,
Do you think this api is loophole for NSE, will the NSE api will get remove sooner or later?
as we know the api is not free by NSE, also NSE endpoint contain api in it, making me doubt.

Do we need to limit API access per second to avoid getting blocked?

[rmantena]

Thanks for the workaround @amritjain123!
This worked for me.

[gopikrishnan17]

Thanks @amritjain123 , the fix works!!

[k-prabin]

The fix works for local machine. Somebody Pls update the library.

[Craven-Pirate]

The fix doesn't work well with batch commands. JSON Decode Error popping up if trying to iterate historical data retrieval for over 15 tickers.

[rmantena]

Seems like this repository is not being actively managed anymore. Any contributors subscribed to this thread?
There are several open issues on this repository.

What does it take for someone to push this change and to get it approved?
I am not sure how pushing this workaround to production would cause issues, but something simple as github.com//issues/70 should be doable by anyone. It's just updating text in a file.

[buzzvolt]

PR#88 should help. brotli python module is required to handle encoded responses (check requests to also handle it)

Hi All, I resolved this by navigating to live.py file in installation directory and locating live.py file.(use pip show jugaad-data) In that file locate header dictoniary by 'h' name. and replace "Accept-Encoding": "gzip, deflate, br" with this one "Accept-Encoding": "gzip, deflate", It starts running for me.

I hope this helps temporarily until fix is released.

if this does not work, then move to use nsepython

[divyankm]

#86 #87 #89 fixed