Indexing Error in realtime updating

[wolfcuring]

Requirements (place an x in each of the [ ])**

• [x ] I realize finplot is not a web lib. (Hint: it's native!)
• [x ] I've read the snippets and not found what I'm looking for.
• [x ] I've searched for any related issues and avoided creating a duplicate issue.
• [x ] I've updated finplot (pip install -U finplot).
• [x ] I've supplied the required data to run my code below.

Code to reproduce

import finplot as plt
import pandas as pd
import talib as ta



def chart_it(list_of_bars):
    """
    Plotting method

    :param data_ready_for_plot: bool
    :param es: pd.DataFrame
    :return: None
    """

    def update_minute_bar():
        """

        :return:
        """
        # retrive data.

        es = pd.DataFrame(list_of_bars)
        es = es.astype({'date': 'datetime64[ns]'})

        # calculate indicators.
        es['sma26'] = ta.SMA(es['close'], 26).round(2)
        es['sma52'] = ta.SMA(es['close'], 52).round(2)
        
        # RSI
        es['rsi'] = ta.RSI(es['close'])

        # prepare df sources for candle_sticks and indicators.
        candles = es[['date', 'open', 'close', 'high', 'low']]
        sma26 = es[['date', 'sma26']]
        sma52 = es[['date', 'sma52']]
        rsi = es[['date', 'rsi']]
        

        # plot candles and indicators.
        plot_candles.candlestick_ochl(candles, ax=ax0)
        plot_sma26.plot(sma26, ax=ax0, color='red', width=1.5, legend='SMA26')
        plot_sma52.plot(sma52, ax=ax0, color='black', width=4, legend='SMA52')
        plot_rsi.plot(rsi, ax=ax1, legend='RSI(14)', color='#927', style='^')
        
    # decorating
    name = 'MES Minute Bars'
    ax0, ax1 = plt.create_plot(title=name, rows=2)
    plt.add_band(30, 70, color='#cccc78', ax=ax1)
    plt.set_y_range(0, 100, ax=ax1)

    # set timezone.
    plt.display_timezone = None  # use plain data without timezone info.

    # main loop.
    plot_candles, plot_sma26, plot_sma52, plot_rsi = plt.live(4)
    update_minute_bar()
    plt.timer_callback(update_minute_bar, 5)
    plt.show()


The input data is list of dicts retrieved from native ibapi, updating every minute. Like this:

[{'date': '20230613  03:39:00',
  'open': 4354.5,
  'high': 4355.5,
  'low': 4354.25,
  'close': 4355.5,
  'volume': 23},
 {'date': '20230613  03:40:00',
  'open': 4355.5,
  'high': 4356.0,
  'low': 4355.5,
  'close': 4355.75,
  'volume': 36},
 {'date': '20230613  03:41:00',
  'open': 4355.75,
  'high': 4355.75,
  'low': 4355.0,
  'close': 4355.25,
  'volume': 40},
 {'date': '20230613  03:42:00',
  'open': 4355.25,
  'high': 4355.5,
  'low': 4354.75,
  'close': 4354.75,
  'volume': 25},
 {'date': '20230613 03:43:00',
  'open': 4354.5,
  'high': 4354.5,
  'low': 4354.5,
  'close': 4354.5,
  'volume': 1}]

Describe the bug

It plots the chart, but not updating properly. PyCharm is throwing out the following error:

/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/finplot/init.py:434: FutureWarning: reindexing with a non-unique Index is deprecated and will raise in a future version.
output_df[col] = df[col]
Traceback (most recent call last):
File "/Users/war/miniconda3/envs/ib/chart.py", line 46, in update_minute_bar
plot_sma26.plot(sma26, ax=ax0, color='red', width=1.5, legend='SMA26')
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/finplot/live.py", line 43, in wrap_call
item.update_data(*args, **ka)
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/finplot/init.py", line 2422, in _update_data
item.datasrc.update(ds)
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/finplot/init.py", line 434, in update
output_df[col] = df[col]
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/frame.py", line 3980, in setitem
self._set_item(key, value)
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/frame.py", line 4174, in _set_item
value = self._sanitize_column(value)
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/frame.py", line 4912, in _sanitize_column
return _reindex_for_setitem(Series(value), self.index)
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/frame.py", line 12025, in _reindex_for_setitem
raise err
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/frame.py", line 12020, in _reindex_for_setitem
reindexed_value = value.reindex(index)._values
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/series.py", line 5094, in reindex
return super().reindex(**kwargs)
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/generic.py", line 5289, in reindex
return self._reindex_axes(
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/generic.py", line 5309, in _reindex_axes
obj = obj._reindex_with_indexers(
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/generic.py", line 5355, in _reindex_with_indexers
new_data = new_data.reindex_indexer(
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 737, in reindex_indexer
self.axes[axis]._validate_can_reindex(indexer)
File "/Users/war/miniconda3/envs/ib/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 4316, in _validate_can_reindex
raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels

Expected behavior

The chart is expected to update every 5 seconds, with all indicators.

Screenshots

If applicable, add screenshots to help explain your problem.

Reproducible in:

OS: Mac OS 13.4
finplot version: 1.9.1
pyqtgraph version: 0.13.3
pyqt version: PyQt6
es_week.csv

[highfestiva]

What Pandas version are you using? In v1.5.3 and v2.0.2 all examples work. If you want specific help, please send in a complete working example. For instance list_of_bars is undefined in your code above.

[wolfcuring]

The pandas version is 1.5.3, the format of the list_of_bars is given above. To reproduce the error. One has to install IB's TradersWorkStation, the ibapi package, and several files that overwrite default method in ibapi to tweak the data format retrived from the api. I really wonder whether I should bother people that much.

Several other errors are thrown when I try to make a plot. And they are all about the pd.index, So I wonder whether it is safe to manipulate the pandas index object. For example, when using pd.reset_index, pandas will leave the original index as a column. So, I have to sacrifice efficiency, giving up calculating indicators in one place for all purposes, then to calculate the indicators in the plot block.

I remember this code ran perfectly the first time. And kept throwing out the error now.

[highfestiva]

Cut it down to the smallest possible example, including data. No additional packages needed if you dump a small part of the data at some point.

You could try es = es.reset_index(drop=True) right after es = pd.DataFrame(list_of_bars). If that doesn't work, please post the output of print(es) to let me know what the DataFrame looks like.

[highfestiva]

@wolfcuring is this still a problem?