Fix Session Initialization Issue 850

[paulr34]

from #850 (comment)

"Here's my best attempt to properly list requirements....
@paulr34 , do note that we have discussed a lot "what is there now". This text does NOT AT ALL talk about "what is there now". It is talking only about "what should be there in the future". My opinion is that we should just rip out all the mess we have now, and replace it with this proper flow.

TERMINOLOGY:

static requests are HTTP requests that retrieve STATIC content, like images, plain HTML and so on. They don't need a "session" or the "user" for that matter.
dynamic requests are HTTP requests that retrieve data from the database. They require a "session" in the DB table, since all dynamic requests are made per-session. And session doesn't exist unless there is a "user" in the USER table, so they also require a USER.
session is linked to a "single ZAP configuration". If you are simultaneously editing 3 files, you have 3 sessions.
user is linked to a "single browser+cookie instance". If you are one user, using one zap, no matter how many files you have opened, that is ONE user. In terminology of a website: no matter how many tabs you have opened in a browser against the zap server, all those tabs are ONE user, but each one is DIFFERENT session.
zap home page: the main zap page, where you can see endpoints on the left and clusters on the right. The place where you end up after your session is fully linked with packages and you are "open for business".
zap meta selection page: the page we have in zap where user gets to select the packages, before going on to "zap home page".
FUNCTIONAL requirements:

First: note that this is talking about "Zap UI startup". That is not equivalent to "Zap startup". We might have already loaded some static ZCL packages, or gen-templates and created database and so on and so on. So when "Zap UI starts up", there ALREADY is a database, and there alredy may be some ZCL or template packages loaded. Please don't confuse up the "zap startup" with "zap UI startup". This "design" is addressing the "Zap UI startup" issues only.

Zap UI startup can start up in one of the following modes:

blank configuration and no qualifiers: this means that you start a blank configuration and you are not specifying either whether you want zigbee, matter, or anything else. In this case the functionality should be as follows:

browser window comes up
a "zap meta selection" (a.k.a "zap config") page comes up, listing ALL available toplevel packages: both zcl metafiles, custom XML files already loaded and the generation packages loaded. If NOTHING is loaded at all, then this page should be blank and prompt the user to at least load an zcl-properties package, since you can't proceed without one.
meta selection page REQUIRES user to pick ONE AND EXACTLY ONE zcl-properties package. They can also then pick ZERO OR MORE custom XML packages, and ZERO OR MORE template generation packages.
a special use case here is a "session recovery". The "zap meta selection" page also lists existing "lost" sessions in the database, and with this use case, user can "link" to one of the existing sessions. Personally, I'm starting to believe that we should remove this "lost session" case from here, and simply move it into electron menu layers, or have CLI option to list and/or recover lost session. The fact that you ALREADY have a session here, but then you want to RECOVER another session is wrong. We should OPEN A NEW WINDOW if you want to recover a session.
once user makes necessary selection, they land in zap home page, where they can start adding endpoints.
blank configuration with qualifiers: this means that you start with blank, but maybe you passed a qualifier like "category=matter". In this case the functionality should be as follows:

browser window comes up
if there are POSSIBLE choices on the "zap meta selection" page (such as multiple "matter" zcl packages or such), then the "zap meta selection" page comes up, with the packages that match qualifier clearly shown on top.
if there are NO POSSIBLE choices (for example, there is ONLY ONE "matter" zcl package, and ONLY ONE "matter" generation package"), then this page is skipped and we immediatelly go to the zap home page, where they can start adding endpoints.
existing ZAP file and no qualifiers: this means you are starting the UI with existing zap file and no additional information. The functionality should be as follows:

browser window comes up
if ALL the packages listed inside the ZAP file are 100% matched (path, types) with already preloaded packages, then file is loaded, packages are linked and we directly land on zap home page, seeing endpoints. No meta config page is shown.
if SOME packages are not yet loaded, but they do exist on the hard drive (by path), then the listed packages are automatically loaded and we land on zap home page. No meta config page is shown.
if packages are NOT loaded, we can NOT find the packages listed at the paths in zap file, but there is ONE and ONLY ONE match among the packages to the "category" listed in zap file, then we do NOT show the "zap meta selection" page, but immediately link the ZAP file with the only available choice per category (i.e: "matter", "zigbee", etc). We then immediately open the zap home page, but we DO SHOW THE WARNING, telling user what package we matched, and that we can't be 100% certain it's all good.
if there is any ambiguity or any lack of solution (such as multiple available packages matching category, or NO category in the zap file, or possibly NO matching categories and NO known paths where to load them from), we ALWAYS show the "zap meta selection" page, and we simply don't let the user go on until they load or choose some package. We show warnings and notifications explaining user what is happening.
existing ZAP file with qualifiers: (note, this is not a currently required use case, but it should be here for completeness). if ZAP file has qualifiers, and the additional qualifiers CONFLICT with the ones in the zap file (such as, for example, ZAP file saying "zigbee", but a qualifier saying "matter"), then the provided qualifiers OVERRIDE the ones in zap file, and you are assumed to be in a "migration" situation. So basically if you say "load file zigbee-app.zap, but override it with matter", then the file will be loaded, but the "zigbee" packages inside it will be ignored, and the file will effectively be in a "migrate zigbee app to matter" mode. Not that this is a bit more advanced case, which would require probably some additional work beyond just the "session init". For the purposes of this work, we should at least have hooks to know where to start when we start properly implementing this.

TECHNICAL requirements:

The UI should start with a URL pointing to existing index.html or / or whatever it is, with the correct parameters delineating the actual mode as described above. Examples:
localhost:9070/index.html?config=blank
localhost:9070/index.html?config=
localhost:9070/index.html?config=blank&qualifier=matter
etc.
The point are not specific names of parameters. The point is that the starting URL of the browser (the "location"), needs to contain ALL and ANY information required to initialize a UI.
ZAP SPA and all STATIC content, needs to be able to be loaded BEFORE any user or session is created on the server. The UI should be able to get to a point where entire HTML content is mounted into DOM before any user/session tables being affected.

After entire SPA is loaded and MOUNTED into DOM, we should land in "please wait...." progress indicator mode of the UI.

At this point some kind of "init" request is made to the server, that carries with it all the information about session creation: the configuration mode (blank vs zap file) and all the required qualifiers. It should provide enough information to the back-end to answer all the questions listed above in the "functional requirement". Server the creates a session acording to the functional rules above. Do note that it is POSSIBLE that created session is NOT YET USABLE, because NO toplevel zcl-properties is yet linked to it. This is main difference to the current implementation. In a current implementation, we create a session, and then blindly link the FIRST AVAILABLE zcl-properties package to the session, "just because". That is wrong. We should not be doing this any more. If we can't tell without ambiguity which package we need to use we have to ask the user. We can't guess.

Return value back to the browser from this process has 3 possible interpretations:

session was created, packages are ready, and you can move on to zap home page.
session was created, but packages are not yet ready, so you NEED TO SHOW the zap meta selection page to sort that out.
session could not even be created. Fatal error. Show fatal message and give up.
We have to make 100% certain that until a "session creation" request described in 3 is finished, we:

keep showing the progress indicator of "please wait...." in a browser.
we make NO OTHER dynamic requests to the server at all.
Once the session creation request described in 3 is finished we do:

if the session creation returned error, we go from "please wait..." page to "Fatal error!" page, where your only options are basically: close browser to give up, or "reload this whole URL to try again".
if the session was created, and we need to go to "zap meta selection" page, we forward on to "zap meta selection" page. We DO NOT YET go loading "initial state or anything" related to endpoint data. We first have to deal with "zap meta selection" page (see next point 6)
if the session was created, and we can go to "zap home page" to edit endpoint, we do that. At this point, we can make as many asyncronous dynamic requests as we want. It's "open for business".
If we need to go to "zap meta selection page", the "zap meta selection page" should be MODAL. Meaning, that as long as you are on this page, you do NOT YET start fetching all kind of session data from the back-end (like labels and session-based images, and endpoint config, and so on and so on). We have to STAY on the "zap meta selection page", and user has to make the necessary selection. Only AFTER this MODAL page goes away, we can land in "zap home page" and then we can start making dynamic requests and call ourselves "open for business".

Some side notes:

we need to 100% eliminate the weird business we're doing today, where when we load the file, we FIRST create a blank session, then we ALREADY link some packages to it, and then LATER, we do a "location.reload()" and we wipe everything again to load the file. That's a giant mess. We should NEVER reload the spa, unless user really reloaded the whole browser.
we need to 100% eliminate any dynamic requests to populate ANY part of the zap home page UI (endpoints, labels, clusters, etc, etc), UNTIL AFTER the point when session and session packages are ready. The only requests we are allowed to make to the server AFTER session is created, but BEFORE packages are properly linked, are the requests we have to make to read which packages are available, so basically stuff required to populate the "zap meta selection" page.
we should NEVER delete a session and replace it with a different session within a single instance of a browser window. We can ONLY delete a session if the browser window goes away."