下载模型

git lfs install

git clone https://www.modelscope.cn/qwen/Qwen-7B.git

中途下载报错，手动下载几个没有正常拉下来的模型文件

移动过来

Quickstart

创建工作目录以及env

要求及安装依赖

pip install transformers==4.32.0 accelerate tiktoken einops scipy transformers_stream_generator==0.0.4 peft deepspeed

pip install flash-attention

pip install modelscope

测试，非常缓慢！GPU显存接近极限

异常缓慢，调整代码报错，显存不足

fp16，bf16都会报错。

接近4分钟，没有意义了。。。

后续研究下量化方法，缩小模型对显存的消耗看看效果。

7B模型这个问题的回答乱七八糟

部署Qwen web

下载源代码

git clone https://gh-proxy.com/https://github.com/QwenLM/Qwen

创建venv安装依赖

 安装web的依赖

pip install -r requirements_web_demo.txt

 直接启动报错

python3 web_demo.py

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/home/yeqiang/下载/src/Qwen/web_demo.py", line 209, in <module>

    main()

  File "/home/yeqiang/下载/src/Qwen/web_demo.py", line 203, in main

    model, tokenizer, config = _load_model_tokenizer(args)

  File "/home/yeqiang/下载/src/Qwen/web_demo.py", line 41, in _load_model_tokenizer

    tokenizer = AutoTokenizer.from_pretrained(

  File "/home/yeqiang/下载/src/Qwen/venv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 773, in from_pretrained

    config = AutoConfig.from_pretrained(

  File "/home/yeqiang/下载/src/Qwen/venv/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1100, in from_pretrained

    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)

  File "/home/yeqiang/下载/src/Qwen/venv/lib/python3.10/site-packages/transformers/configuration_utils.py", line 634, in get_config_dict

    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)

  File "/home/yeqiang/下载/src/Qwen/venv/lib/python3.10/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict

    resolved_config_file = cached_file(

  File "/home/yeqiang/下载/src/Qwen/venv/lib/python3.10/site-packages/transformers/utils/hub.py", line 425, in cached_file

    raise EnvironmentError(

OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like Qwen/Qwen-7B-Chat is not the path to a directory containing a file named config.json.

Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

没有指定模型，自动连接到huggingface，报错

 查看help

(venv) (base) yeqiang@yeqiang-Default-string:~/Downloads/src/Qwen$ python3 web_demo.py --help

usage: web_demo.py [-h] [-c CHECKPOINT_PATH] [--cpu-only] [--share] [--inbrowser] [--server-port SERVER_PORT] [--server-name SERVER_NAME]

options:

  -h, --help            show this help message and exit

  -c CHECKPOINT_PATH, --checkpoint-path CHECKPOINT_PATH

                        Checkpoint name or path, default to 'Qwen/Qwen-7B-Chat'

  --cpu-only            Run demo with CPU only

  --share               Create a publicly shareable link for the interface.

  --inbrowser           Automatically launch the interface in a new tab on the default browser.

  --server-port SERVER_PORT

                        Demo server port.

  --server-name SERVER_NAME

                        Demo server name.

 指定模型路径，再次启动

python3 web_demo.py -c /home/yeqiang/Downloads/ai/Qwen-7B --server-port 8080

Wed Apr 10 09:49:48 2024       

+---------------------------------------------------------------------------------------+

| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |

|-----------------------------------------+----------------------+----------------------+

| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |

|                                         |                      |               MIG M. |

|=========================================+======================+======================|

|   0  NVIDIA GeForce RTX 4060 Ti     Off | 00000000:01:00.0  On |                  N/A |

|  0%   35C    P8              13W / 165W |  13219MiB / 16380MiB |     34%      Default |

|                                         |                      |                  N/A |

+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+

| Processes:                                                                            |

|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |

|        ID   ID                                                             Usage      |

|=======================================================================================|

|    0   N/A  N/A      3465      G   /usr/lib/xorg/Xorg                          200MiB |

|    0   N/A  N/A      3617      G   /usr/bin/gnome-shell                         62MiB |

|    0   N/A  N/A     66713      G   ...38243838,2569802313780412916,262144       52MiB |

|    0   N/A  N/A    258826      C   python3                                   12890MiB |

+---------------------------------------------------------------------------------------+

web模式需要下载Qwen-7B-Chat模型。

Qwen-7B-Chat模型

git clone https://www.modelscope.cn/qwen/Qwen-7B-Chat.git

速度有点慢，输出内容还行（显存90%左右，GPU 80%左右）

参考：

魔搭社区