Toniq.JobImporter should gracefully restart on redis issues

[mspanc]

Using toniq 1.2.0

2017-01-06T10:20:29.226317+00:00 app[web.1]: 10:20:29.215 [error] GenServer Toniq.JobImporter terminating
2017-01-06T10:20:29.226331+00:00 app[web.1]: ** (UndefinedFunctionError) function Toniq.JobImporter.terminate/2 is undefined or private
2017-01-06T10:20:29.226333+00:00 app[web.1]:     (toniq) Toniq.JobImporter.terminate({:timeout, {:gen_server, :call, [#PID<0.993.0>, {:request, [["*", '2', "\r\n"], [["$", '8', "\r\n", "SMEMBERS", "\r\n"], ["$", '56', "\r\n", "toniq:b9f474c2-d3f9-11e6-9462-72b23b5ea2e9:incoming_jobs", "\r\n"]]]}, 5000]}}, [])
2017-01-06T10:20:29.226333+00:00 app[web.1]:     (stdlib) gen_server.erl:643: :gen_server.try_terminate/3
2017-01-06T10:20:29.226334+00:00 app[web.1]:     (stdlib) gen_server.erl:809: :gen_server.terminate/7
2017-01-06T10:20:29.226335+00:00 app[web.1]:     (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

That can lead to crashing whole application

2017-01-06T10:21:10.668318+00:00 app[web.1]: 10:21:10.667 [error] GenServer Toniq.JobImporter terminating
2017-01-06T10:21:10.668332+00:00 app[web.1]: ** (UndefinedFunctionError) function Toniq.JobImporter.terminate/2 is undefined or private
2017-01-06T10:21:10.668333+00:00 app[web.1]:     (toniq) Toniq.JobImporter.terminate({:timeout, {:gen_server, :call, [#PID<0.1356.0>, {:request, [["*", '2', "\r\n"], [["$", '8', "\r\n", "SMEMBERS", "\r\n"], ["$", '56', "\r\n", "toniq:cf1f79d2-d3f9-11e6-bbe6-72b23b5ea2e9:incoming_jobs", "\r\n"]]]}, 5000]}}, [])
2017-01-06T10:21:10.668334+00:00 app[web.1]:     (stdlib) gen_server.erl:643: :gen_server.try_terminate/3
2017-01-06T10:21:10.668335+00:00 app[web.1]:     (stdlib) gen_server.erl:809: :gen_server.terminate/7
2017-01-06T10:21:10.668335+00:00 app[web.1]:     (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
2017-01-06T10:21:10.780690+00:00 app[web.1]: 10:21:10.776 [info] Application toniq exited: shutdown
2017-01-06T10:21:12.363953+00:00 app[web.1]: {"Kernel pid terminated",application_controller,"{application_terminated,toniq,shutdown}"}
2017-01-06T10:21:12.364269+00:00 app[web.1]:
2017-01-06T10:21:13.638112+00:00 app[web.1]: Crash dump is being written to: erl_crash.dump...*** stack smashing detected ***: /app/.platform_tools/erlang/erts-7.3/bin/beam.smp terminated
2017-01-06T10:21:13.824785+00:00 heroku[web.1]: Process exited with status 134
2017-01-06T10:21:13.846389+00:00 heroku[web.1]: State changed from up to crashed

[seivan]

We're also getting these issues. When a new node pops up and imports the jobs sometimes it duplicates them. I guess this is explained in the Readme but would be nice if it tried as much as possible not to do that. I am not talking about a full fledge locking system but at least mitigate it.