Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scql镜像制作问题咨询 #442

Closed
lisg666666 opened this issue Jan 21, 2025 · 4 comments
Closed

scql镜像制作问题咨询 #442

lisg666666 opened this issue Jan 21, 2025 · 4 comments

Comments

@lisg666666
Copy link

使用scql源码docker文件夹下的build.sh脚本构建镜像,我笔记本8c16g 起了个wsl2虚拟机,docker用的27.4.0,编译了大约8个小时,报错。请问这是什么原因呢?资源不够还是做镜像需要做其他配置呢?
部分日志如下所示:
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:556:48: warning: 'EC_KEY* EC_KEY_new_by_curve_name(int)' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
556 | if (!i || ((ecdh = EC_KEY_new_by_curve_name(i)) == NULL)) {
| ~~~~~~~~~~~~~~~~~~~~~~~~^~~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:33,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ec.h:1017:31: note: declared here
1017 | OSSL_DEPRECATEDIN_3_0 EC_KEY EC_KEY_new_by_curve_name(int nid);
| ^~~~~~~~~~~~~~~~~~~~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:563:16: warning: 'void EC_KEY_free(EC_KEY
)' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
563 | EC_KEY_free(ecdh);
| ~~~~~~~~~~~^~~~~~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:33,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ec.h:1022:28: note: declared here
1022 | OSSL_DEPRECATEDIN_3_0 void EC_KEY_free(EC_KEY key);
| ^~~~~~~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp: In function 'DH
brpc::SSLGetDH1024()':
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:725:20: warning: 'DH* DH_new()' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
725 | DH dh = DH_new();
| ~~~~~~^~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:210:27: note: declared here
210 | OSSL_DEPRECATEDIN_3_0 DH DH_new(void);
| ^~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:731:16: warning: 'int DH_set0_pqg(DH
, BIGNUM
, BIGNUM*, BIGNUM*)' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
731 | DH_set0_pqg(dh, p, NULL, g);
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:266:27: note: declared here
266 | OSSL_DEPRECATEDIN_3_0 int DH_set0_pqg(DH dh, BIGNUM p, BIGNUM q, BIGNUM g);
| ^~~~~~~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp: In function 'DH
brpc::SSLGetDH2048()':
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:748:20: warning: 'DH
DH_new()' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
748 | DH
dh = DH_new();
| ~~~~~~^~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:210:27: note: declared here
210 | OSSL_DEPRECATEDIN_3_0 DH DH_new(void);
| ^~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:754:16: warning: 'int DH_set0_pqg(DH
, BIGNUM
, BIGNUM*, BIGNUM*)' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
754 | DH_set0_pqg(dh, p, NULL, g);
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:266:27: note: declared here
266 | OSSL_DEPRECATEDIN_3_0 int DH_set0_pqg(DH dh, BIGNUM p, BIGNUM q, BIGNUM g);
| ^~~~~~~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp: In function 'DH
brpc::SSLGetDH4096()':
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:771:20: warning: 'DH
DH_new()' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
771 | DH dh = DH_new();
| ~~~~~~^~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:210:27: note: declared here
210 | OSSL_DEPRECATEDIN_3_0 DH DH_new(void);
| ^~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:777:16: warning: 'int DH_set0_pqg(DH
, BIGNUM
, BIGNUM
, BIGNUM
)' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
777 | DH_set0_pqg(dh, p, NULL, g);
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:266:27: note: declared here
266 | OSSL_DEPRECATEDIN_3_0 int DH_set0_pqg(DH dh, BIGNUM p, BIGNUM q, BIGNUM g);
| ^~~~~~~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp: In function 'DH
brpc::SSLGetDH8192()':
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:794:20: warning: 'DH
DH_new()' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
794 | DH dh = DH_new();
| ~~~~~~^~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:210:27: note: declared here
210 | OSSL_DEPRECATEDIN_3_0 DH DH_new(void);
| ^~~~~~
external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:800:16: warning: 'int DH_set0_pqg(DH
, BIGNUM
, BIGNUM
, BIGNUM
)' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
800 | DH_set0_pqg(dh, p, NULL, g);
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dsa.h:31,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/x509.h:37,
from bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/ssl.h:32,
from external/com_github_brpc_brpc/src/brpc/details/ssl_helper.cpp:24:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/dh.h:266:27: note: declared here
266 | OSSL_DEPRECATEDIN_3_0 int DH_set0_pqg(DH dh, BIGNUM p, BIGNUM q, BIGNUM g);
| ^~~~~~~~~~~
INFO: From Compiling src/brpc/controller.cpp:
external/com_github_brpc_brpc/src/brpc/controller.cpp: In member function 'void brpc::Controller::AppendServerIdentiy()':
external/com_github_brpc_brpc/src/brpc/controller.cpp:394:12: warning: 'unsigned char
MD5(const unsigned char
, size_t, unsigned char
)' is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
394 | MD5((const unsigned char
)ipbuf, len, digest);
| ~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from external/com_github_brpc_brpc/src/brpc/controller.cpp:20:
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/include/openssl/md5.h:52:38: note: declared here
52 | OSSL_DEPRECATEDIN_3_0 unsigned char *MD5(const unsigned char *d, size_t n,
| ^~~
[4,536 / 4,767] 8 actions running
[4,707 / 4,767] 8 actions running
Compiling engine/datasource/csvdb_adaptor.cc; 99s linux-sandbox
Compiling engine/util/tensor_util.cc; 81s linux-sandbox
Compiling engine/datasource/arrow_sql_adaptor.cc; 73s linux-sandbox
Compiling engine/datasource/csvdb_adaptor_factory.cc; 73s linux-sandbox
Compiling engine/datasource/datasource_adaptor_mgr.cc; 38s linux-sandbox
Compiling engine/framework/session.cc; 23s linux-sandbox
Compiling engine/framework/session_manager.cc; 17s linux-sandbox
Compiling engine/framework/exec.cc; 16s linux-sandbox
request returned Internal Server Error for API route and version http://%2Fvar%2Frun%2Fdocker.sock/v1.47/exec/b3bcc9fbe7bb877cbefe06859896866a751cfe10e3492bc8ff2236a4e5edc790/json, check if the server supports the requested API version

@jingshi-ant
Copy link
Collaborator

关于报错,看起来可能是超时或者硬件资源不足导致的bazel 报错。
scql的编译需要大量的计算资源,建议提供足够资源(32核64G,该资源下,预计需要半小时左右完成编译),也避免使用虚拟机编译。

@lisg666666
Copy link
Author

关于报错,看起来可能是超时或者硬件资源不足导致的bazel 报错。 scql的编译需要大量的计算资源,建议提供足够资源(32核64G,该资源下,预计需要半小时左右完成编译),也避免使用虚拟机编译。

谢谢,我们在研究scql有没有可能扩展hive数据源,想着验证下镜像制作,目前手头上没有这么大资源,能租用阿里云的资源进行镜像制作吗?如果可以的话建议租用啥配置的资源呢?

@jingshi-ant
Copy link
Collaborator

jingshi-ant commented Jan 22, 2025

阿里云ecs应该可以,建议32c64g起步,编译过程中需要访问外网,如果dockerhub的镜像拉不下来,建议修改下docker proxy

@lisg666666
Copy link
Author

好的 谢谢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants