关于glm4-9B-chat大模型

【xinference】（11）：在compshare上使用4090D运行xinf和chatgpt-web，部署大模型glm4-9B-chat大模型，显存18G

GLM-4-9B 是智谱 AI 推出的最新一代预训练模型 GLM-4 系列中的开源版本。 在语义、数学、推理、代码和知识等多方面的数据集测评中，GLM-4-9B 及其人类偏好对齐的版本 GLM-4-9B-Chat 均表现出较高的性能。 除了能进行多轮对话，GLM-4-9B-Chat 还具备网页浏览、代码执行、自定义工具调用（Function Call）和长文本推理（支持最大 128K 上下文）等高级功能。 本代模型增加了多语言支持，支持包括日语，韩语，德语在内的 26 种语言。我们还推出了支持 1M 上下文长度（约 200 万中文字符）的模型。

https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat/summary

模型文件分成了 10 个文件

xinf 也支持了：

https://inference.readthedocs.io/zh-cn/latest/models/builtin/llm/glm4-chat.html

创建带驱动的镜像

设置密码

2，需要选择23.08，cuda12.3版本，否则报错

<code> × Getting requirements to build wheel did not run successfully.

│ exit code: 1

╰─> [33 lines of output]

Traceback (most recent call last):

File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>

main()

File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main

json_out['return_val'] = hook(**hook_input['kwargs'])

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel

backend = _build_backend()

^^^^^^^^^^^^^^^^

File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend

obj = import_module(mod_path)

^^^^^^^^^^^^^^^^^^^^^^^

File "/home/ubuntu/miniconda3/lib/python3.12/importlib/__init__.py", line 90, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "<frozen importlib._bootstrap>", line 1387, in _gcd_import

File "<frozen importlib._bootstrap>", line 1360, in _find_and_load

File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed

File "<frozen importlib._bootstrap>", line 1387, in _gcd_import

File "<frozen importlib._bootstrap>", line 1360, in _find_and_load

File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 935, in _load_unlocked

File "<frozen importlib._bootstrap_external>", line 995, in exec_module

File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed

File "/tmp/pip-build-env-84tcwvda/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>

import setuptools.version

File "/tmp/pip-build-env-84tcwvda/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>

import pkg_resources

File "/tmp/pip-build-env-84tcwvda/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2191, in <module>

register_finder(pkgutil.ImpImporter, find_on_path)

^^^^^^^^^^^^^^^^^^^

AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

问题’pkgutil’ has no attribute ‘ImpImporter’,切换版本解决

使用 cuda12.3 版本的python是 3.10 版本。没有这个问题

是python3.12 版本造成的。

最终运行效果

占用显存情况18G：