Improve error handling and reporting in query side

[gortiz]

Error handling and reporting can be improved in Pinot. Sometimes, error messages are not easy for users to understand, and the fact that each query engine notifies errors differently makes it even more confusing. Here is a nonexclusive list of improvements we need to apply:

• On errors, Pinot controller query endpoints inconsistently return broker-like JSON error payloads (if the broker detects the error) or plain text (if the controller itself detected the error. Fixed in Cleanup controller and MSE broker entry points to keep error codes an… #15277
• The Pinot controller detected errors are not logged. Fixed in Cleanup controller and MSE broker entry points to keep error codes an… #15277
• MSE errors include the stack trace.
  • Partially fixed in Cleanup controller and MSE broker entry points to keep error codes an… #15277
• MSE errors don't include the actual error code. Instead, error code 200 is always returned. Fixed in Cleanup controller and MSE broker entry points to keep error codes an… #15277
• MSE errors are logged several times in the same server.
• MSE checks and SSE checks are different. Some are copied and pasted in both places, but some constraints are enforced differently.
  • MSE doesn't check if tables are disabled
• Introduce a correlation ID to track logs related to the same query. This ID may be defined by the customer (otherwise, it uses the initial request ID) and can be introduced in logs Query thread context and cid #15180
• Cleanup exception classes Refactor query error handling to use QueryErrorCode and QueryErrorMessage for improved clarity and consistency #15037

[gortiz]

Opened #14951 to fix the first two problems

[real-mj-song]

MSE errors are large and don't include the actual error code. Instead, error code 200 is always returned.

💯 agreed. We definitely need a throwable/exception class containing Pinot error code info. I described the reasoning in this comment.

[vrajat]

MSE errors include the stack trace and don't include the actual error code. Instead, error code 200 is always returned.

This is true for SSE as well. For example:

"exceptions": [
    {
      "message": "QueryExecutionError:\nQuery execution error on: Server_100.81.71.117_7050 org.apache.pinot.spi.exception.EarlyTerminationException: Interrupted while processing next block",
      "errorCode": 200
    },

The error code should have been 503 or QUERY_CANCELLATION_ERROR_CODE

[gortiz]

Please take a look at #14994

[gortiz]

Correlation ID PR has been merged