fix graph migration 'no such file or directory' rpc_call error bug #36
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
描述:
进行graph迁移时,会出现migrate.go:301: open /opt/open-falcon/graph/data/6070/35/356aa82f9c00c1edd11102696313c636_GAUGE_60.rrd: no such file or directory这样的报错,并且该报错会持续发生,每次落盘周期30分钟一次,经查证,为代码段:
if os.IsNotExist(err) {
并未成功判断该err为os.IsNotExist(“文件不存在”)这个错误类型所造成,查看go源码,发现os.IsNotExist只针对PathError有效,可判断其是否为os.IsNotExist,但对rpc_call返回的普通error无法判断,造成if语句无法按预期判断,迁移时,一旦有新的指标项传入新的graph,则会报fetch_s_error,并无法落盘,只能写入内存。
可能的隐患:
如果迁移长时间持续进行,中途未进行重启graph的操作,新的指标监控项会造成内存数据持续增加,直至OOM,并且会造成log日志持续写入,日志增长过快。
解决方式:
修改
if os.IsNotExist(err) {
为if strings.Contains(err.Error(), "no such file or directory")
,直接对rpc_call返回的error进行内容判断,经测试验证通过Signed-off-by: Xin He hexin@jiedaibao.com