<code>Model config Qwen2MoeConfig {

"_name_or_path": "/data/zengzijian/Qwen2-57B-A14B/",

"architectures": [

"Qwen2MoeForCausalLM"

],

"attention_dropout": 0.0,

"bos_token_id": 151643,

"decoder_sparse_step": 1,

"eos_token_id": 151643,

"hidden_act": "silu",

"hidden_size": 3584,

"initializer_range": 0.02,

"intermediate_size": 18944,

"max_position_embeddings": 131072,

"max_window_layers": 28,

"mlp_only_layers": [],

"model_type": "qwen2_moe",

"moe_intermediate_size": 2560,

"norm_topk_prob": false,

"num_attention_heads": 28,

"num_experts": 64,

"num_experts_per_tok": 8,

"num_hidden_layers": 28,

"num_key_value_heads": 4,

"output_router_logits": false,

"rms_norm_eps": 1e-06,

"rope_theta": 1000000.0,

"router_aux_loss_coef": 0.001,

"shared_expert_intermediate_size": 20480,

"sliding_window": 131072,

"tie_word_embeddings": false,

"torch_dtype": "bfloat16",

"transformers_version": "4.41.2",

"use_cache": true,

"use_sliding_window": false,

"vocab_size": 151936

}

[INFO|modeling_utils.py:3471] 2024-07-15 15:34:49,666 >> loading weights file /data/zengzijian/Qwen2-57B-A14B/model.safetensors.index.json

[INFO|modeling_utils.py:3614] 2024-07-15 15:34:49,670 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model

[INFO|configuration_utils.py:962] 2024-07-15 15:34:49,674 >> Generate config GenerationConfig {

"bos_token_id": 151643,

"eos_token_id": 151643

}

Token indices sequence length is longer than the specified maximum sequence length for this model (132576 > 131072). Running this sequence through the model will result in indexing errors

Qwen2-57B-A14B是阿里巴巴开源的第二代通义千问系列大模型中的一个混合专家(Mixture of Experts, MoE)版本. 以下是该模型的一些关键详细信息:

模型规模:

总参数规模: 570亿推理时激活参数: 140亿

发布信息:

发布日期: 2024年6月7日预训练文件大小: 114GB

模型架构:

基于Transformer架构使用SwiGLU激活函数采用混合专家(MoE)技术

性能表现:

Qwen2-57B-A14B在多项评测任务中表现优异,超越了当前主流的MoE开源模型. 具体表现如下:

MMLU: 75.4MMLU-Pro: 52.8GPQA: 34.3TheroemQA: 33.1MT-Bench: 8.55HumanEval (代码): 79.9GSM8K (数学): 79.6MATH: 49.1C-Eval (中文): 80.5AlignBench: 7.36

主要特点:

在自然语言理解、知识、代码、数学及多语言等多项能力上表现出色经过精细微调,提升了代码、数学、推理、指令遵循、多语言理解等能力对齐了人类价值观,更加有帮助、诚实和安全

训练方法:

采用多种自动方法获取高质量指令和偏好数据结合有监督微调、反馈模型训练和在线DPO等方法使用在线模型合并方法减少对齐成本

Qwen2-57B-A14B作为一个强大的MoE模型,在保持较小激活参数规模的同时,实现了优秀的性能表现,为大规模语言模型的应用提供了新的可能性。

deepspeed训练

日志展示了Qwen2-57B-A14B MoE模型的详细性能分析。我来解释一下关键信息:

模型架构:

总参数量:57B (57亿)每次前向传播激活参数:14B (14亿)使用了MoE (Mixture of Experts)结构共有28层Decoder层每层有64个专家(experts)每次激活8个专家

并行设置:

世界大小(world size):16,表示使用16个GPU数据并行度(data parallel size):16模型并行度(model parallel size):1每个GPU的batch size:16

计算性能:

每个GPU的参数量:433.66K每个GPU的前向传播MACs:449.07 TMACs每个GPU的前向传播FLOPS:37.63 TFLOPS总体FLOPS:31.01 TFLOPS每秒处理样本数:2.95

延迟:

前向传播延迟:23.87s后向传播延迟:61.66s参数更新延迟:1.37s总迭代延迟:86.9s

关键模块:

Embedding层:151936 x 3584自注意力层:使用FlashAttention2MLP层:使用SparseeMoeBlock每个专家MLP的隐藏维度:2560

MoE结构:

门控网络:Linear layer (3584 -> 64)64个专家,每次激活8个每个专家是一个MLP,包含gate_proj, up_proj, down_proj和SiLU激活

这个MoE模型通过在每一层使用多个专家网络,实现了在保持较低激活参数量(14B)的同时,拥有较大的总参数量(57B)。这种设计能够在计算资源有限的情况下,训练和部署更大规模的模型,从而可能获得更好的性能。

-------------------------- DeepSpeed Flops Profiler --------------------------

gpu009: Profile Summary at step 2:

gpu009: Notations:

gpu009: data parallel size (dp_size), model parallel size(mp_size),

gpu009: number of parameters (params), number of multiply-accumulate operations(MACs),

gpu009: number of floating-point operations (flops), floating-point operations per second (FLOPS),

gpu009: fwd latency (forward propagation latency), bwd latency (backward propagation latency),

gpu009: step (weights update latency), iter latency (sum of fwd, bwd and step latency)

gpu009:

gpu009: world size: 16

gpu009: data parallel size: 16

gpu009: model parallel size: 1

gpu009: batch size per GPU: 16

gpu009: params per GPU: 433.66 K

gpu009: params of model = params per GPU * mp_size: 433.66 K

gpu009: fwd MACs per GPU: 449.07 TMACs

gpu009: fwd flops per GPU: 898.15 T

gpu009: fwd flops of model = fwd flops per GPU * mp_size: 898.15 T

gpu009: fwd latency: 23.87 s

gpu009: fwd FLOPS per GPU = fwd flops per GPU / fwd latency: 37.63 TFLOPS

gpu009: bwd latency: 61.66 s

gpu009: bwd FLOPS per GPU = 2 * fwd flops per GPU / bwd latency: 29.13 TFLOPS

gpu009: fwd+bwd FLOPS per GPU = 3 * fwd flops per GPU / (fwd+bwd latency): 31.5 TFLOPS

gpu009: step latency: 1.37 s

gpu009: iter latency: 86.9 s

gpu009: FLOPS per GPU = 3 * fwd flops per GPU / iter latency: 31.01 TFLOPS

gpu009: samples/second: 2.95

gpu009:

gpu009: ----------------------------- Aggregated Profile per GPU -----------------------------

gpu009: Top 3 modules in terms of params, MACs or fwd latency at different model depths:

gpu009: depth 0:

gpu009: params - {'Qwen2MoeForCausalLM': '433.66 K'}

gpu009: MACs - {'Qwen2MoeForCausalLM': '449.07 TMACs'}

gpu009: fwd latency - {'Qwen2MoeForCausalLM': '23.73 s'}

gpu009: depth 1:

gpu009: params - {'Qwen2MoeModel': '433.66 K', 'Linear': '0'}

gpu009: MACs - {'Qwen2MoeModel': '431.22 TMACs', 'Linear': '17.84 TMACs'}

gpu009: fwd latency - {'Qwen2MoeModel': '23.59 s', 'Linear': '65.17 ms'}

gpu009: depth 2:

gpu009: params - {'ModuleList': '430.08 K', 'Qwen2MoeRMSNorm': '3.58 K', 'Embedding': '0'}

gpu009: MACs - {'ModuleList': '431.22 TMACs', 'Embedding': '0 MACs', 'Qwen2MoeRMSNorm': '0 MACs'}

gpu009: fwd latency - {'ModuleList': '23.47 s', 'Qwen2MoeRMSNorm': '1.58 ms', 'Embedding': '1.48 ms'}

gpu009: depth 3:

gpu009: params - {'Qwen2MoeDecoderLayer': '430.08 K'}

gpu009: MACs - {'Qwen2MoeDecoderLayer': '431.22 TMACs'}

gpu009: fwd latency - {'Qwen2MoeDecoderLayer': '23.47 s'}

gpu009: depth 4:

gpu009: params - {'Qwen2MoeRMSNorm': '200.7 K', 'Qwen2MoeFlashAttention2': '129.02 K', 'Qwen2MoeSparseMoeBlock': '100.35 K'}

gpu009: MACs - {'Qwen2MoeSparseMoeBlock': '404.28 TMACs', 'Qwen2MoeFlashAttention2': '26.94 TMACs', 'Qwen2MoeRMSNorm': '0 MACs'}

gpu009: fwd latency - {'Qwen2MoeSparseMoeBlock': '22.17 s', 'Qwen2MoeFlashAttention2': '779.38 ms', 'Qwen2MoeRMSNorm': '90.4 ms'}

gpu009: depth 5:

gpu009: params - {'Linear': '229.38 K', 'Qwen2MoeRotaryEmbedding': '0', 'ModuleList': '0'}

gpu009: MACs - {'ModuleList': '202.04 TMACs', 'Qwen2MoeMLP': '202.04 TMACs', 'Linear': '27.15 TMACs'}

gpu009: fwd latency - {'ModuleList': '16.49 s', 'Qwen2MoeMLP': '1.14 s', 'Linear': '224.08 ms'}

gpu009: depth 6:

gpu009: params - {'Qwen2MoeMLP': '0', 'Linear': '0', 'SiLU': '0'}

gpu009: MACs - {'Qwen2MoeMLP': '202.04 TMACs', 'Linear': '202.04 TMACs', 'SiLU': '0 MACs'}

gpu009: fwd latency - {'Qwen2MoeMLP': '16.49 s', 'Linear': '713.84 ms', 'SiLU': '25.52 ms'}

gpu009:

gpu009: ------------------------------ Detailed Profile per GPU ------------------------------

gpu009: Each module profile is listed after its name in the following order:

gpu009: params, percentage of total params, MACs, percentage of total MACs, fwd latency, percentage of total fwd latency, fwd FLOPS

gpu009:

gpu009: Note: 1. A module can have torch.nn.module or torch.nn.functional to compute logits (e.g. CrossEntropyLoss). They are not counted as submodules, thus not to be printed out. However they make up the difference between a parent's MACs (or latency) and the sum of its submodules'.

gpu009: 2. Number of floating-point operations is a theoretical estimation, thus FLOPS computed using that could be larger than the maximum system throughput.

gpu009: 3. The fwd latency listed in the top module's profile is directly captured at the module forward function in PyTorch, thus it's less than the fwd latency shown above which is captured in DeepSpeed.

gpu009:

gpu009: Qwen2MoeForCausalLM(

gpu009: 433.66 K = 100% Params, 449.07 TMACs = 100% MACs, 23.73 s = 100% latency, 37.86 TFLOPS

gpu009: (model): Qwen2MoeModel(

gpu009: 433.66 K = 100% Params, 431.22 TMACs = 96.03% MACs, 23.59 s = 99.43% latency, 36.56 TFLOPS

gpu009: (embed_tokens): Embedding(0 = 0% Params, 0 MACs = 0% MACs, 1.48 ms = 0.01% latency, 0 FLOPS, 151936, 3584)

gpu009: (layers): ModuleList(

gpu009: (0): Qwen2MoeDecoderLayer(

gpu009: 15.36 K = 3.54% Params, 15.4 TMACs = 3.43% MACs, 1.56 s = 6.58% latency, 19.72 TFLOPS

gpu009: (self_attn): Qwen2MoeFlashAttention2(

gpu009: 4.61 K = 1.06% Params, 962.07 GMACs = 0.21% MACs, 29.87 ms = 0.13% latency, 64.41 TFLOPS

gpu009: (q_proj): Linear(3.58 K = 0.83% Params, 420.91 GMACs = 0.09% MACs, 2.03 ms = 0.01% latency, 415.15 TFLOPS, in_features=3584, out_features=3584, bias=True)

gpu009: (k_proj): Linear(512 = 0.12% Params, 60.13 GMACs = 0.01% MACs, 1.21 ms = 0.01% latency, 99.37 TFLOPS, in_features=3584, out_features=512, bias=True)

gpu009: (v_proj): Linear(512 = 0.12% Params, 60.13 GMACs = 0.01% MACs, 1.19 ms = 0.01% latency, 100.92 TFLOPS, in_features=3584, out_features=512, bias=True)

gpu009: (o_proj): Linear(0 = 0% Params, 420.91 GMACs = 0.09% MACs, 1.61 ms = 0.01% latency, 522 TFLOPS, in_features=3584, out_features=3584, bias=False)

gpu009: (rotary_emb): Qwen2MoeRotaryEmbedding(0 = 0% Params, 0 MACs = 0% MACs, 1.01 ms = 0% latency, 0 FLOPS)

gpu009: )

gpu009: (mlp): Qwen2MoeSparseMoeBlock(

gpu009: 3.58 K = 0.83% Params, 14.44 TMACs = 3.22% MACs, 1.47 s = 6.22% latency, 19.58 TFLOPS

gpu009: (gate): Linear(0 = 0% Params, 7.52 GMACs = 0% MACs, 1.14 ms = 0% latency, 13.15 TFLOPS, in_features=3584, out_features=64, bias=False)

gpu009: (experts): ModuleList(

gpu009: (0): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 97.55 GMACs = 0.02% MACs, 20.47 ms = 0.09% latency, 9.53 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 32.52 GMACs = 0.01% MACs, 1.15 ms = 0% latency, 56.58 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 32.52 GMACs = 0.01% MACs, 1.15 ms = 0% latency, 56.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 32.52 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 58.28 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 1.02 ms = 0% latency, 4.45 GFLOPS)

gpu009: )

gpu009: (1): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 113.57 GMACs = 0.03% MACs, 19.76 ms = 0.08% latency, 11.5 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 37.86 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 67.78 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 37.86 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 67.39 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 37.86 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 66.19 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 986.58 us = 0% latency, 5.35 GFLOPS)

gpu009: )

gpu009: (2): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 117.09 GMACs = 0.03% MACs, 20.56 ms = 0.09% latency, 11.39 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.03 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 69.57 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.03 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 68.38 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.03 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 69.6 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 996.11 us = 0% latency, 5.47 GFLOPS)

gpu009: )

gpu009: (3): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 124.58 GMACs = 0.03% MACs, 19.26 ms = 0.08% latency, 12.94 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 41.53 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 74.72 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 41.53 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 73.49 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 41.53 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 74.21 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 965.12 us = 0% latency, 6 GFLOPS)

gpu009: )

gpu009: (4): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 97.16 GMACs = 0.02% MACs, 20.14 ms = 0.08% latency, 9.65 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 32.39 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 57.43 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 32.39 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 57.6 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 32.39 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 57.38 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 969.89 us = 0% latency, 4.66 GFLOPS)

gpu009: )

gpu009: (5): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 119.35 GMACs = 0.03% MACs, 123.01 ms = 0.52% latency, 1.94 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.78 GMACs = 0.01% MACs, 1.16 ms = 0% latency, 68.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.78 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 70.66 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.78 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 71.29 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 967.74 us = 0% latency, 5.74 GFLOPS)

gpu009: )

gpu009: (6): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 106.63 GMACs = 0.02% MACs, 177.81 ms = 0.75% latency, 1.2 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 35.54 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 62.35 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 35.54 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 63.63 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 35.54 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 63.25 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 981.57 us = 0% latency, 5.05 GFLOPS)

gpu009: )

gpu009: (7): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 104.98 GMACs = 0.02% MACs, 85.97 ms = 0.36% latency, 2.44 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.99 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 61.93 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.99 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 62.02 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.99 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 62.59 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 977.99 us = 0% latency, 4.99 GFLOPS)

gpu009: )

gpu009: (8): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 96.28 GMACs = 0.02% MACs, 41.67 ms = 0.18% latency, 4.62 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 32.09 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 56.35 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 32.09 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 57.4 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 32.09 GMACs = 0.01% MACs, 1.56 ms = 0.01% latency, 41.03 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 973.7 us = 0% latency, 4.6 GFLOPS)

gpu009: )

gpu009: (9): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 108.72 GMACs = 0.02% MACs, 24.45 ms = 0.1% latency, 8.89 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 64.22 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 64.75 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 65.23 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 972.99 us = 0% latency, 5.2 GFLOPS)

gpu009: )

gpu009: (10): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 104.27 GMACs = 0.02% MACs, 21.43 ms = 0.09% latency, 9.73 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.76 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 61.52 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.76 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 61.86 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.76 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 61.52 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 965.6 us = 0% latency, 5.02 GFLOPS)

gpu009: )

gpu009: (11): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 107.73 GMACs = 0.02% MACs, 20.38 ms = 0.09% latency, 10.57 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 35.91 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 63.85 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 35.91 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 63.9 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 35.91 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 64.91 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 970.6 us = 0% latency, 5.16 GFLOPS)

gpu009: )

gpu009: (12): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 115.22 GMACs = 0.03% MACs, 11.8 ms = 0.05% latency, 19.52 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 38.41 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 68.36 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 38.41 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 69 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 38.41 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 69.08 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 980.85 us = 0% latency, 5.46 GFLOPS)

gpu009: )

gpu009: (13): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 107.9 GMACs = 0.02% MACs, 9.29 ms = 0.04% latency, 23.23 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 35.97 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 64.34 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 35.97 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 65.42 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 35.97 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 64.97 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 981.81 us = 0% latency, 5.11 GFLOPS)

gpu009: )

gpu009: (14): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 96.89 GMACs = 0.02% MACs, 9.83 ms = 0.04% latency, 19.72 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 32.3 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 57.7 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 32.3 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 58.17 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 32.3 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 58.24 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 987.29 us = 0% latency, 4.56 GFLOPS)

gpu009: )

gpu009: (15): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 116.6 GMACs = 0.03% MACs, 9.51 ms = 0.04% latency, 24.53 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 38.87 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 69.68 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 38.87 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 70.61 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 38.87 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 69.31 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 982.76 us = 0% latency, 5.52 GFLOPS)

gpu009: )

gpu009: (16): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 104.54 GMACs = 0.02% MACs, 9.83 ms = 0.04% latency, 21.27 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.85 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 62.37 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.85 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 63.23 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.85 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 61.31 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 985.15 us = 0% latency, 4.93 GFLOPS)

gpu009: )

gpu009: (17): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 100.08 GMACs = 0.02% MACs, 9.41 ms = 0.04% latency, 21.27 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 33.36 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 59.1 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 33.36 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 60.57 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 33.36 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 58.98 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 984.91 us = 0% latency, 4.73 GFLOPS)

gpu009: )

gpu009: (18): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 101.57 GMACs = 0.02% MACs, 10.06 ms = 0.04% latency, 20.18 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 33.86 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 59.9 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 33.86 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 60.07 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 33.86 GMACs = 0.01% MACs, 1.15 ms = 0% latency, 58.79 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 976.32 us = 0% latency, 4.84 GFLOPS)

gpu009: )

gpu009: (19): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 109.55 GMACs = 0.02% MACs, 9.31 ms = 0.04% latency, 23.53 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.52 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 64.48 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.52 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 65.75 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.52 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 65.82 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 973.46 us = 0% latency, 5.23 GFLOPS)

gpu009: )

gpu009: (20): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 109.94 GMACs = 0.02% MACs, 10.15 ms = 0.04% latency, 21.67 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.65 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 65.43 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.65 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 65.47 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.65 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 65.16 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 969.89 us = 0% latency, 5.27 GFLOPS)

gpu009: )

gpu009: (21): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 114.45 GMACs = 0.03% MACs, 9.45 ms = 0.04% latency, 24.23 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 38.15 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 68.1 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 38.15 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 68.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 38.15 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 67.69 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 971.32 us = 0% latency, 5.48 GFLOPS)

gpu009: )

gpu009: (22): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 107.46 GMACs = 0.02% MACs, 9.67 ms = 0.04% latency, 22.22 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 35.82 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 63.44 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 35.82 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 64 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 35.82 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 64.03 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 972.27 us = 0% latency, 5.14 GFLOPS)

gpu009: )

gpu009: (23): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 104.32 GMACs = 0.02% MACs, 9.43 ms = 0.04% latency, 22.12 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.77 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 62.48 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.77 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 63.04 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.77 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 62.61 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 981.09 us = 0% latency, 4.94 GFLOPS)

gpu009: )

gpu009: (24): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 123.53 GMACs = 0.03% MACs, 9.72 ms = 0.04% latency, 25.41 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 41.18 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 74.4 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 41.18 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 75.14 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 41.18 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 73.62 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 969.41 us = 0% latency, 5.93 GFLOPS)

gpu009: )

gpu009: (25): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 108.72 GMACs = 0.02% MACs, 9.49 ms = 0.04% latency, 22.92 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 64.66 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 65.93 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 65.8 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 971.08 us = 0% latency, 5.21 GFLOPS)

gpu009: )

gpu009: (26): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 102.06 GMACs = 0.02% MACs, 10.03 ms = 0.04% latency, 20.35 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.02 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 60.54 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.02 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 61.65 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.02 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 59.7 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 993.01 us = 0% latency, 4.78 GFLOPS)

gpu009: )

gpu009: (27): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 92.48 GMACs = 0.02% MACs, 9.55 ms = 0.04% latency, 19.36 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 30.83 GMACs = 0.01% MACs, 1.15 ms = 0% latency, 53.82 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 30.83 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 56.26 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 30.83 GMACs = 0.01% MACs, 1.16 ms = 0% latency, 53.29 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 988.72 us = 0% latency, 4.35 GFLOPS)

gpu009: )

gpu009: (28): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 108.83 GMACs = 0.02% MACs, 6.6 ms = 0.03% latency, 32.99 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.28 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 65.49 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.28 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 65.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.28 GMACs = 0.01% MACs, 1.08 ms = 0% latency, 67.19 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 956.06 us = 0% latency, 5.29 GFLOPS)

gpu009: )

gpu009: (29): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 117.97 GMACs = 0.03% MACs, 6.04 ms = 0.03% latency, 39.09 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.32 GMACs = 0.01% MACs, 1.08 ms = 0% latency, 73.16 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.32 GMACs = 0.01% MACs, 1.07 ms = 0% latency, 73.22 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.32 GMACs = 0.01% MACs, 1.06 ms = 0% latency, 74.48 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 936.75 us = 0% latency, 5.86 GFLOPS)

gpu009: )

gpu009: (30): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 123.86 GMACs = 0.03% MACs, 20.87 ms = 0.09% latency, 11.87 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 41.29 GMACs = 0.01% MACs, 1.07 ms = 0% latency, 76.93 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 41.29 GMACs = 0.01% MACs, 1.05 ms = 0% latency, 78.82 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 41.29 GMACs = 0.01% MACs, 1.08 ms = 0% latency, 76.19 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 912.43 us = 0% latency, 6.31 GFLOPS)

gpu009: )

gpu009: (31): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 119.02 GMACs = 0.03% MACs, 5.93 ms = 0.02% latency, 40.17 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.67 GMACs = 0.01% MACs, 1.07 ms = 0% latency, 73.92 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.67 GMACs = 0.01% MACs, 1.04 ms = 0% latency, 76.28 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.67 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 77.45 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 906.47 us = 0% latency, 6.11 GFLOPS)

gpu009: )

gpu009: (32): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 101.9 GMACs = 0.02% MACs, 6.17 ms = 0.03% latency, 33.04 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 33.97 GMACs = 0.01% MACs, 1.05 ms = 0% latency, 64.8 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 33.97 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 66.45 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 33.97 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 66.6 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 881.67 us = 0% latency, 5.37 GFLOPS)

gpu009: )

gpu009: (33): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 135.7 GMACs = 0.03% MACs, 19.29 ms = 0.08% latency, 14.07 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 45.23 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 88.53 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 45.23 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 89.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 45.23 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 89.74 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 906.23 us = 0% latency, 6.96 GFLOPS)

gpu009: )

gpu009: (34): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 96.45 GMACs = 0.02% MACs, 5.97 ms = 0.03% latency, 32.32 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 32.15 GMACs = 0.01% MACs, 995.16 us = 0% latency, 64.61 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 32.15 GMACs = 0.01% MACs, 989.2 us = 0% latency, 65 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 32.15 GMACs = 0.01% MACs, 978.23 us = 0% latency, 65.73 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 849.96 us = 0% latency, 5.28 GFLOPS)

gpu009: )

gpu009: (35): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 131.29 GMACs = 0.03% MACs, 5.62 ms = 0.02% latency, 46.76 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 43.76 GMACs = 0.01% MACs, 989.44 us = 0% latency, 88.46 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 43.76 GMACs = 0.01% MACs, 957.49 us = 0% latency, 91.42 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 43.76 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 85.98 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 827.79 us = 0% latency, 7.38 GFLOPS)

gpu009: )

gpu009: (36): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 120.95 GMACs = 0.03% MACs, 9.07 ms = 0.04% latency, 26.66 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 40.32 GMACs = 0.01% MACs, 1 ms = 0% latency, 80.29 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 40.32 GMACs = 0.01% MACs, 986.34 us = 0% latency, 81.75 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 40.32 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 78.98 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 859.98 us = 0% latency, 6.54 GFLOPS)

gpu009: )

gpu009: (37): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 119.9 GMACs = 0.03% MACs, 9.09 ms = 0.04% latency, 26.38 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.97 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 78.11 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.97 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 77.97 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.97 GMACs = 0.01% MACs, 1.04 ms = 0% latency, 76.88 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 883.1 us = 0% latency, 6.31 GFLOPS)

gpu009: )

gpu009: (38): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 132.29 GMACs = 0.03% MACs, 12.38 ms = 0.05% latency, 21.38 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 44.1 GMACs = 0.01% MACs, 1.04 ms = 0% latency, 84.8 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 44.1 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 87.49 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 44.1 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 86.53 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 868.56 us = 0% latency, 7.08 GFLOPS)

gpu009: )

gpu009: (39): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 126.56 GMACs = 0.03% MACs, 20.56 ms = 0.09% latency, 12.31 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 42.19 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 82.78 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 42.19 GMACs = 0.01% MACs, 984.43 us = 0% latency, 85.71 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 42.19 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 82.84 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 869.27 us = 0% latency, 6.77 GFLOPS)

gpu009: )

gpu009: (40): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 120.01 GMACs = 0.03% MACs, 15.5 ms = 0.07% latency, 15.49 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 40 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 78.39 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 40 GMACs = 0.01% MACs, 998.97 us = 0% latency, 80.09 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 40 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 77.62 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 876.19 us = 0% latency, 6.37 GFLOPS)

gpu009: )

gpu009: (41): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 112.36 GMACs = 0.03% MACs, 18.06 ms = 0.08% latency, 12.44 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 37.45 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 72.96 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 37.45 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 73.65 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 37.45 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 72.89 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 875.47 us = 0% latency, 5.97 GFLOPS)

gpu009: )

gpu009: (42): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 133.72 GMACs = 0.03% MACs, 9.34 ms = 0.04% latency, 28.64 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 44.57 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 87.12 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 44.57 GMACs = 0.01% MACs, 998.02 us = 0% latency, 89.32 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 44.57 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 87.05 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 872.85 us = 0% latency, 7.12 GFLOPS)

gpu009: )

gpu009: (43): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 126.01 GMACs = 0.03% MACs, 9.34 ms = 0.04% latency, 26.98 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 42 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 82.44 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 42 GMACs = 0.01% MACs, 982.28 us = 0% latency, 85.52 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 42 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 82.38 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 895.98 us = 0% latency, 6.54 GFLOPS)

gpu009: )

gpu009: (44): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 121.44 GMACs = 0.03% MACs, 9.39 ms = 0.04% latency, 25.88 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 40.48 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 79.92 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 40.48 GMACs = 0.01% MACs, 993.01 us = 0% latency, 81.53 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 40.48 GMACs = 0.01% MACs, 1.04 ms = 0% latency, 77.65 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 880 us = 0% latency, 6.42 GFLOPS)

gpu009: )

gpu009: (45): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 114.23 GMACs = 0.03% MACs, 9.05 ms = 0.04% latency, 25.23 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 38.08 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 74.02 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 38.08 GMACs = 0.01% MACs, 993.49 us = 0% latency, 76.65 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 38.08 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 75.39 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 882.86 us = 0% latency, 6.02 GFLOPS)

gpu009: )

gpu009: (46): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 123.53 GMACs = 0.03% MACs, 9.4 ms = 0.04% latency, 26.28 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 41.18 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 80.82 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 41.18 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 81.6 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 41.18 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 80.48 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 870.7 us = 0% latency, 6.6 GFLOPS)

gpu009: )

gpu009: (47): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 105.37 GMACs = 0.02% MACs, 9.05 ms = 0.04% latency, 23.28 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 35.12 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 69.24 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 35.12 GMACs = 0.01% MACs, 998.26 us = 0% latency, 70.37 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 35.12 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 69.02 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 870.23 us = 0% latency, 5.63 GFLOPS)

gpu009: )

gpu009: (48): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 118.91 GMACs = 0.03% MACs, 8.6 ms = 0.04% latency, 27.66 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.64 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 77.58 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.64 GMACs = 0.01% MACs, 997.54 us = 0% latency, 79.47 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.64 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 78.79 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 866.65 us = 0% latency, 6.38 GFLOPS)

gpu009: )

gpu009: (49): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 110.43 GMACs = 0.02% MACs, 8.55 ms = 0.04% latency, 25.83 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.81 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 71.91 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.81 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 72.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.81 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 72.52 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 894.55 us = 0% latency, 5.74 GFLOPS)

gpu009: )

gpu009: (50): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 111.75 GMACs = 0.02% MACs, 8.58 ms = 0.04% latency, 26.06 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 37.25 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 72.42 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 37.25 GMACs = 0.01% MACs, 987.53 us = 0% latency, 75.44 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 37.25 GMACs = 0.01% MACs, 993.97 us = 0% latency, 74.95 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 868.8 us = 0% latency, 5.98 GFLOPS)

gpu009: )

gpu009: (51): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 113.46 GMACs = 0.03% MACs, 8.57 ms = 0.04% latency, 26.49 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 37.82 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 74.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 37.82 GMACs = 0.01% MACs, 988.48 us = 0% latency, 76.52 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 37.82 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 74.4 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 869.51 us = 0% latency, 6.07 GFLOPS)

gpu009: )

gpu009: (52): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 134.82 GMACs = 0.03% MACs, 8.65 ms = 0.04% latency, 31.17 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 44.94 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 88.02 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 44.94 GMACs = 0.01% MACs, 989.44 us = 0% latency, 90.84 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 44.94 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 88.97 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 875 us = 0% latency, 7.17 GFLOPS)

gpu009: )

gpu009: (53): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 103 GMACs = 0.02% MACs, 8.52 ms = 0.04% latency, 24.18 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.33 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 66.47 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.33 GMACs = 0.01% MACs, 991.11 us = 0% latency, 69.28 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.33 GMACs = 0.01% MACs, 1 ms = 0% latency, 68.56 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 860.21 us = 0% latency, 5.57 GFLOPS)

gpu009: )

gpu009: (54): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 109.33 GMACs = 0.02% MACs, 12.27 ms = 0.05% latency, 17.82 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.44 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 72.12 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.44 GMACs = 0.01% MACs, 990.87 us = 0% latency, 73.56 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.44 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 71.8 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 865.22 us = 0% latency, 5.88 GFLOPS)

gpu009: )

gpu009: (55): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 108.72 GMACs = 0.02% MACs, 8.58 ms = 0.04% latency, 25.34 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 71.77 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 994.92 us = 0% latency, 72.85 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.24 GMACs = 0.01% MACs, 1 ms = 0% latency, 72.3 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 872.61 us = 0% latency, 5.79 GFLOPS)

gpu009: )

gpu009: (56): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 104.27 GMACs = 0.02% MACs, 8.45 ms = 0.04% latency, 24.69 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.76 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 68.78 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.76 GMACs = 0.01% MACs, 989.68 us = 0% latency, 70.24 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.76 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 68.65 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 871.9 us = 0% latency, 5.56 GFLOPS)

gpu009: )

gpu009: (57): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 104.98 GMACs = 0.02% MACs, 8.56 ms = 0.04% latency, 24.52 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.99 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 69.27 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.99 GMACs = 0.01% MACs, 983.72 us = 0% latency, 71.15 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.99 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 68.75 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 859.02 us = 0% latency, 5.68 GFLOPS)

gpu009: )

gpu009: (58): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 118.8 GMACs = 0.03% MACs, 8.67 ms = 0.04% latency, 27.42 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.6 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 77.13 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.6 GMACs = 0.01% MACs, 997.3 us = 0% latency, 79.41 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.6 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 78.59 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 866.89 us = 0% latency, 6.37 GFLOPS)

gpu009: )

gpu009: (59): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 114.56 GMACs = 0.03% MACs, 8.8 ms = 0.04% latency, 26.03 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 38.19 GMACs = 0.01% MACs, 1 ms = 0% latency, 76.18 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 38.19 GMACs = 0.01% MACs, 982.28 us = 0% latency, 77.75 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 38.19 GMACs = 0.01% MACs, 1 ms = 0% latency, 76.09 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 864.98 us = 0% latency, 6.16 GFLOPS)

gpu009: )

gpu009: (60): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 127.28 GMACs = 0.03% MACs, 8.7 ms = 0.04% latency, 29.26 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 42.43 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 84.31 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 42.43 GMACs = 0.01% MACs, 980.38 us = 0% latency, 86.55 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 42.43 GMACs = 0.01% MACs, 1 ms = 0% latency, 84.55 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 865.46 us = 0% latency, 6.84 GFLOPS)

gpu009: )

gpu009: (61): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 106.8 GMACs = 0.02% MACs, 8.57 ms = 0.04% latency, 24.92 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 35.6 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 70.25 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 35.6 GMACs = 0.01% MACs, 1 ms = 0% latency, 70.85 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 35.6 GMACs = 0.01% MACs, 996.83 us = 0% latency, 71.42 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 869.04 us = 0% latency, 5.71 GFLOPS)

gpu009: )

gpu009: (62): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 114.12 GMACs = 0.03% MACs, 8.61 ms = 0.04% latency, 26.51 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 38.04 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 74.89 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 38.04 GMACs = 0.01% MACs, 987.77 us = 0% latency, 77.02 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 38.04 GMACs = 0.01% MACs, 998.02 us = 0% latency, 76.23 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 857.11 us = 0% latency, 6.19 GFLOPS)

gpu009: )

gpu009: (63): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 109.72 GMACs = 0.02% MACs, 11.99 ms = 0.05% latency, 18.3 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.57 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 72.36 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.57 GMACs = 0.01% MACs, 989.68 us = 0% latency, 73.91 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.57 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 71.2 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 866.65 us = 0% latency, 5.89 GFLOPS)

gpu009: )

gpu009: )

gpu009: (shared_expert): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 7.22 TMACs = 1.61% MACs, 41.08 ms = 0.17% latency, 351.26 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 2.41 TMACs = 0.54% MACs, 8.57 ms = 0.04% latency, 561.51 TFLOPS, in_features=3584, out_features=20480, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 2.41 TMACs = 0.54% MACs, 8.57 ms = 0.04% latency, 560.98 TFLOPS, in_features=3584, out_features=20480, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 2.41 TMACs = 0.54% MACs, 8.35 ms = 0.04% latency, 575.79 TFLOPS, in_features=20480, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 897.41 us = 0% latency, 373.9 GFLOPS)

gpu009: )

gpu009: (shared_expert_gate): Linear(3.58 K = 0.83% Params, 117.44 MMACs = 0% MACs, 1.07 ms = 0% latency, 219.17 GFLOPS, in_features=3584, out_features=1, bias=False)

gpu009: )

gpu009: (input_layernorm): Qwen2MoeRMSNorm(3.58 K = 0.83% Params, 0 MACs = 0% MACs, 1.61 ms = 0.01% latency, 0 FLOPS)

gpu009: (post_attention_layernorm): Qwen2MoeRMSNorm(3.58 K = 0.83% Params, 0 MACs = 0% MACs, 1.61 ms = 0.01% latency, 0 FLOPS)

gpu009: )

gpu009: (1): Qwen2MoeDecoderLayer(

gpu009: 15.36 K = 3.54% Params, 15.4 TMACs = 3.43% MACs, 841.01 ms = 3.54% latency, 36.63 TFLOPS

gpu009: (self_attn): Qwen2MoeFlashAttention2(

gpu009: 4.61 K = 1.06% Params, 962.07 GMACs = 0.21% MACs, 28.23 ms = 0.12% latency, 68.17 TFLOPS

gpu009: (q_proj): Linear(3.58 K = 0.83% Params, 420.91 GMACs = 0.09% MACs, 1.88 ms = 0.01% latency, 446.88 TFLOPS, in_features=3584, out_features=3584, bias=True)

gpu009: (k_proj): Linear(512 = 0.12% Params, 60.13 GMACs = 0.01% MACs, 1.17 ms = 0% latency, 102.54 TFLOPS, in_features=3584, out_features=512, bias=True)

gpu009: (v_proj): Linear(512 = 0.12% Params, 60.13 GMACs = 0.01% MACs, 1.17 ms = 0% latency, 102.35 TFLOPS, in_features=3584, out_features=512, bias=True)

gpu009: (o_proj): Linear(0 = 0% Params, 420.91 GMACs = 0.09% MACs, 1.61 ms = 0.01% latency, 521.46 TFLOPS, in_features=3584, out_features=3584, bias=False)

gpu009: (rotary_emb): Qwen2MoeRotaryEmbedding(0 = 0% Params, 0 MACs = 0% MACs, 989.91 us = 0% latency, 0 FLOPS)

gpu009: )

gpu009: (mlp): Qwen2MoeSparseMoeBlock(

gpu009: 3.58 K = 0.83% Params, 14.44 TMACs = 3.22% MACs, 795.47 ms = 3.35% latency, 36.3 TFLOPS

gpu009: (gate): Linear(0 = 0% Params, 7.52 GMACs = 0% MACs, 1.1 ms = 0% latency, 13.63 TFLOPS, in_features=3584, out_features=64, bias=False)

gpu009: (experts): ModuleList(

gpu009: (0): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 61.38 GMACs = 0.01% MACs, 9.05 ms = 0.04% latency, 13.56 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 20.46 GMACs = 0% MACs, 1.12 ms = 0% latency, 36.4 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 20.46 GMACs = 0% MACs, 1.09 ms = 0% latency, 37.6 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 20.46 GMACs = 0% MACs, 1.14 ms = 0% latency, 36 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 969.17 us = 0% latency, 2.95 GFLOPS)

gpu009: )

gpu009: (1): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 128.98 GMACs = 0.03% MACs, 9.27 ms = 0.04% latency, 27.83 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 42.99 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 76.17 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 42.99 GMACs = 0.01% MACs, 1.09 ms = 0% latency, 78.8 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 42.99 GMACs = 0.01% MACs, 1.27 ms = 0.01% latency, 67.45 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 971.08 us = 0% latency, 6.18 GFLOPS)

gpu009: )

gpu009: (2): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 21.47 GMACs = 0% MACs, 12.64 ms = 0.05% latency, 3.4 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 7.16 GMACs = 0% MACs, 1.12 ms = 0% latency, 12.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 7.16 GMACs = 0% MACs, 1.12 ms = 0% latency, 12.73 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 7.16 GMACs = 0% MACs, 1.17 ms = 0% latency, 12.22 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 991.82 us = 0% latency, 1.01 GFLOPS)

gpu009: )

gpu009: (3): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 58.9 GMACs = 0.01% MACs, 9.17 ms = 0.04% latency, 12.85 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 19.63 GMACs = 0% MACs, 1.16 ms = 0% latency, 33.93 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 19.63 GMACs = 0% MACs, 1.13 ms = 0% latency, 34.88 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 19.63 GMACs = 0% MACs, 1.16 ms = 0% latency, 33.84 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 981.09 us = 0% latency, 2.79 GFLOPS)

gpu009: )

gpu009: (4): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 86.92 GMACs = 0.02% MACs, 9.2 ms = 0.04% latency, 18.91 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 28.97 GMACs = 0.01% MACs, 1.15 ms = 0% latency, 50.31 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 28.97 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 51.57 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 28.97 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 51.84 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 971.56 us = 0% latency, 4.16 GFLOPS)

gpu009: )

gpu009: (5): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 128.32 GMACs = 0.03% MACs, 9.25 ms = 0.04% latency, 27.76 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 42.77 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 75.67 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 42.77 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 77.58 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 42.77 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 76.69 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 987.29 us = 0% latency, 6.04 GFLOPS)

gpu009: )

gpu009: (6): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 168.62 GMACs = 0.04% MACs, 9.07 ms = 0.04% latency, 37.18 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 56.21 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 98.7 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 56.21 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 100.27 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 56.21 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 99.91 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 975.13 us = 0% latency, 8.04 GFLOPS)

gpu009: )

gpu009: (7): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 284.88 GMACs = 0.06% MACs, 9.01 ms = 0.04% latency, 63.22 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 94.96 GMACs = 0.02% MACs, 1.12 ms = 0% latency, 170.07 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 94.96 GMACs = 0.02% MACs, 1.1 ms = 0% latency, 173.14 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 94.96 GMACs = 0.02% MACs, 1.12 ms = 0% latency, 169.52 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 963.45 us = 0% latency, 13.75 GFLOPS)

gpu009: )

gpu009: (8): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 48.83 GMACs = 0.01% MACs, 9.15 ms = 0.04% latency, 10.67 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 16.28 GMACs = 0% MACs, 1.13 ms = 0% latency, 28.88 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 16.28 GMACs = 0% MACs, 1.09 ms = 0% latency, 29.73 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 16.28 GMACs = 0% MACs, 1.12 ms = 0% latency, 29.09 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 986.1 us = 0% latency, 2.3 GFLOPS)

gpu009: )

gpu009: (9): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 168.29 GMACs = 0.04% MACs, 9.49 ms = 0.04% latency, 35.48 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 56.1 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 99.76 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 56.1 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 101.88 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 56.1 GMACs = 0.01% MACs, 1.18 ms = 0% latency, 95.39 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 995.4 us = 0% latency, 7.86 GFLOPS)

gpu009: )

gpu009: (10): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 37.49 GMACs = 0.01% MACs, 9.15 ms = 0.04% latency, 8.2 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 12.5 GMACs = 0% MACs, 1.15 ms = 0% latency, 21.75 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 12.5 GMACs = 0% MACs, 1.1 ms = 0% latency, 22.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 12.5 GMACs = 0% MACs, 1.14 ms = 0% latency, 21.88 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 982.52 us = 0% latency, 1.77 GFLOPS)

gpu009: )

gpu009: (11): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 126.67 GMACs = 0.03% MACs, 14.19 ms = 0.06% latency, 17.86 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 42.22 GMACs = 0.01% MACs, 1.71 ms = 0.01% latency, 49.5 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 42.22 GMACs = 0.01% MACs, 1.62 ms = 0.01% latency, 52.22 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 42.22 GMACs = 0.01% MACs, 1.69 ms = 0.01% latency, 49.96 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 1.59 ms = 0.01% latency, 3.69 GFLOPS)

gpu009: )

gpu009: (12): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 34.19 GMACs = 0.01% MACs, 11.55 ms = 0.05% latency, 5.92 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 11.4 GMACs = 0% MACs, 1.57 ms = 0.01% latency, 14.5 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 11.4 GMACs = 0% MACs, 1.3 ms = 0.01% latency, 17.48 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 11.4 GMACs = 0% MACs, 1.7 ms = 0.01% latency, 13.41 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 1.46 ms = 0.01% latency, 1.09 GFLOPS)

gpu009: )

gpu009: (13): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 34.57 GMACs = 0.01% MACs, 11.93 ms = 0.05% latency, 5.8 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 11.52 GMACs = 0% MACs, 1.7 ms = 0.01% latency, 13.59 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 11.52 GMACs = 0% MACs, 1.66 ms = 0.01% latency, 13.9 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 11.52 GMACs = 0% MACs, 1.69 ms = 0.01% latency, 13.6 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 1.51 ms = 0.01% latency, 1.06 GFLOPS)

gpu009: )

gpu009: (14): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 69.25 GMACs = 0.02% MACs, 9.85 ms = 0.04% latency, 14.06 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 23.08 GMACs = 0.01% MACs, 1.63 ms = 0.01% latency, 28.28 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 23.08 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 41.47 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 23.08 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 41.06 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 1.17 ms = 0% latency, 2.74 GFLOPS)

gpu009: )

gpu009: (15): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 36.66 GMACs = 0.01% MACs, 8.89 ms = 0.04% latency, 8.25 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 12.22 GMACs = 0% MACs, 1.12 ms = 0% latency, 21.92 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 12.22 GMACs = 0% MACs, 1.1 ms = 0% latency, 22.31 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 12.22 GMACs = 0% MACs, 1.12 ms = 0% latency, 21.91 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 970.13 us = 0% latency, 1.76 GFLOPS)

gpu009: )

gpu009: (16): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 102.67 GMACs = 0.02% MACs, 8.98 ms = 0.04% latency, 22.86 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 34.22 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 61.33 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 34.22 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 61.98 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 34.22 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 62.17 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 979.19 us = 0% latency, 4.88 GFLOPS)

gpu009: )

gpu009: (17): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 98.32 GMACs = 0.02% MACs, 9.09 ms = 0.04% latency, 21.63 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 32.77 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 57.59 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 32.77 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 59.49 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 32.77 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 59.08 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 977.28 us = 0% latency, 4.68 GFLOPS)

gpu009: )

gpu009: (18): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 31.32 GMACs = 0.01% MACs, 9.09 ms = 0.04% latency, 6.89 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 10.44 GMACs = 0% MACs, 1.12 ms = 0% latency, 18.7 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 10.44 GMACs = 0% MACs, 1.09 ms = 0% latency, 19.08 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 10.44 GMACs = 0% MACs, 1.13 ms = 0% latency, 18.44 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 979.66 us = 0% latency, 1.49 GFLOPS)

gpu009: )

gpu009: (19): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 245.47 GMACs = 0.05% MACs, 9.16 ms = 0.04% latency, 53.6 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 81.82 GMACs = 0.02% MACs, 1.13 ms = 0% latency, 144.35 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 81.82 GMACs = 0.02% MACs, 1.1 ms = 0% latency, 149.02 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 81.82 GMACs = 0.02% MACs, 1.12 ms = 0% latency, 145.57 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 972.99 us = 0% latency, 11.73 GFLOPS)

gpu009: )

gpu009: (20): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 230.11 GMACs = 0.05% MACs, 8.97 ms = 0.04% latency, 51.33 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 76.7 GMACs = 0.02% MACs, 1.13 ms = 0% latency, 135.37 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 76.7 GMACs = 0.02% MACs, 1.09 ms = 0% latency, 140.61 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 76.7 GMACs = 0.02% MACs, 1.13 ms = 0% latency, 135.92 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 966.55 us = 0% latency, 11.07 GFLOPS)

gpu009: )

gpu009: (21): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 169.72 GMACs = 0.04% MACs, 9.08 ms = 0.04% latency, 37.36 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 56.57 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 100.27 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 56.57 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 103.3 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 56.57 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 100.02 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 967.74 us = 0% latency, 8.16 GFLOPS)

gpu009: )

gpu009: (22): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 89.95 GMACs = 0.02% MACs, 8.94 ms = 0.04% latency, 20.12 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 29.98 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 53.63 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 29.98 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 54.3 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 29.98 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 53.81 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 955.58 us = 0% latency, 4.38 GFLOPS)

gpu009: )

gpu009: (23): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 11.18 GMACs = 0% MACs, 9.1 ms = 0.04% latency, 2.46 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 3.73 GMACs = 0% MACs, 1.13 ms = 0% latency, 6.61 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 3.73 GMACs = 0% MACs, 1.1 ms = 0% latency, 6.78 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 3.73 GMACs = 0% MACs, 1.12 ms = 0% latency, 6.64 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 972.75 us = 0% latency, 534.24 MFLOPS)

gpu009: )

gpu009: (24): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 124.8 GMACs = 0.03% MACs, 13 ms = 0.05% latency, 19.21 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 41.6 GMACs = 0.01% MACs, 1.15 ms = 0% latency, 72.23 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 41.6 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 75.44 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 41.6 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 73.36 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 972.99 us = 0% latency, 5.96 GFLOPS)

gpu009: )

gpu009: (25): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 66.61 GMACs = 0.01% MACs, 9.17 ms = 0.04% latency, 14.53 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 22.2 GMACs = 0% MACs, 1.13 ms = 0% latency, 39.15 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 22.2 GMACs = 0% MACs, 1.11 ms = 0% latency, 39.88 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 22.2 GMACs = 0% MACs, 1.13 ms = 0% latency, 39.44 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 974.89 us = 0% latency, 3.18 GFLOPS)

gpu009: )

gpu009: (26): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 71.68 GMACs = 0.02% MACs, 9.62 ms = 0.04% latency, 14.91 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 23.89 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 42.03 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 23.89 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 43.55 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 23.89 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 42.07 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 977.04 us = 0% latency, 3.41 GFLOPS)

gpu009: )

gpu009: (27): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 26.81 GMACs = 0.01% MACs, 10.23 ms = 0.04% latency, 5.24 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 8.94 GMACs = 0% MACs, 1.13 ms = 0% latency, 15.79 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 8.94 GMACs = 0% MACs, 1.1 ms = 0% latency, 16.18 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 8.94 GMACs = 0% MACs, 1.15 ms = 0% latency, 15.59 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 984.43 us = 0% latency, 1.27 GFLOPS)

gpu009: )

gpu009: (28): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 90.72 GMACs = 0.02% MACs, 6.59 ms = 0.03% latency, 27.52 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 30.24 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 54.37 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 30.24 GMACs = 0.01% MACs, 1.09 ms = 0% latency, 55.57 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 30.24 GMACs = 0.01% MACs, 1.09 ms = 0% latency, 55.32 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 971.08 us = 0% latency, 4.34 GFLOPS)

gpu009: )

gpu009: (29): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 215.41 GMACs = 0.05% MACs, 6.13 ms = 0.03% latency, 70.32 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 71.8 GMACs = 0.02% MACs, 1.11 ms = 0% latency, 128.98 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 71.8 GMACs = 0.02% MACs, 1.09 ms = 0% latency, 132.24 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 71.8 GMACs = 0.02% MACs, 1.08 ms = 0% latency, 133.35 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 936.75 us = 0% latency, 10.69 GFLOPS)

gpu009: )

gpu009: (30): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 198.51 GMACs = 0.04% MACs, 20.09 ms = 0.08% latency, 19.76 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 66.17 GMACs = 0.01% MACs, 1.06 ms = 0% latency, 124.65 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 66.17 GMACs = 0.01% MACs, 1.06 ms = 0% latency, 124.6 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 66.17 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 119.99 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 913.14 us = 0% latency, 10.11 GFLOPS)

gpu009: )

gpu009: (31): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 201.92 GMACs = 0.04% MACs, 5.91 ms = 0.02% latency, 68.28 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 67.31 GMACs = 0.01% MACs, 1.06 ms = 0% latency, 126.51 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 67.31 GMACs = 0.01% MACs, 1.05 ms = 0% latency, 128 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 67.31 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 130.79 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 903.37 us = 0% latency, 10.39 GFLOPS)

gpu009: )

gpu009: (32): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 285.66 GMACs = 0.06% MACs, 6.2 ms = 0.03% latency, 92.22 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 95.22 GMACs = 0.02% MACs, 1.07 ms = 0% latency, 177.38 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 95.22 GMACs = 0.02% MACs, 1.01 ms = 0% latency, 188.61 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 95.22 GMACs = 0.02% MACs, 1.02 ms = 0% latency, 186.28 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 887.87 us = 0% latency, 14.96 GFLOPS)

gpu009: )

gpu009: (33): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 111.64 GMACs = 0.02% MACs, 18.6 ms = 0.08% latency, 12 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 37.21 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 73.28 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 37.21 GMACs = 0.01% MACs, 1 ms = 0% latency, 74.06 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 37.21 GMACs = 0.01% MACs, 996.35 us = 0% latency, 74.7 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 877.38 us = 0% latency, 5.92 GFLOPS)

gpu009: )

gpu009: (34): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 155.24 GMACs = 0.03% MACs, 6.2 ms = 0.03% latency, 50.05 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 51.75 GMACs = 0.01% MACs, 1 ms = 0% latency, 103.04 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 51.75 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 91.93 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 51.75 GMACs = 0.01% MACs, 980.38 us = 0% latency, 105.57 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 854.73 us = 0% latency, 8.45 GFLOPS)

gpu009: )

gpu009: (35): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 227.25 GMACs = 0.05% MACs, 5.62 ms = 0.02% latency, 80.81 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 75.75 GMACs = 0.02% MACs, 992.3 us = 0% latency, 152.67 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 75.75 GMACs = 0.02% MACs, 971.79 us = 0% latency, 155.9 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 75.75 GMACs = 0.02% MACs, 999.93 us = 0% latency, 151.51 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 819.21 us = 0% latency, 12.9 GFLOPS)

gpu009: )

gpu009: (36): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 114.94 GMACs = 0.03% MACs, 8.39 ms = 0.04% latency, 27.4 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 38.31 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 75.43 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 38.31 GMACs = 0.01% MACs, 1 ms = 0% latency, 76.42 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 38.31 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 75.08 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 864.27 us = 0% latency, 6.18 GFLOPS)

gpu009: )

gpu009: (37): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 196.14 GMACs = 0.04% MACs, 9.52 ms = 0.04% latency, 41.22 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 65.38 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 128.23 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 65.38 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 130 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 65.38 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 127.02 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 878.1 us = 0% latency, 10.39 GFLOPS)

gpu009: )

gpu009: (38): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 325.02 GMACs = 0.07% MACs, 8.62 ms = 0.04% latency, 75.43 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 108.34 GMACs = 0.02% MACs, 1.01 ms = 0% latency, 214.04 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 108.34 GMACs = 0.02% MACs, 993.25 us = 0% latency, 218.15 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 108.34 GMACs = 0.02% MACs, 1.03 ms = 0% latency, 210.91 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 864.03 us = 0% latency, 17.49 GFLOPS)

gpu009: )

gpu009: (39): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 109.61 GMACs = 0.02% MACs, 8.92 ms = 0.04% latency, 24.57 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 36.54 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 71.21 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 36.54 GMACs = 0.01% MACs, 995.64 us = 0% latency, 73.39 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 36.54 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 72.49 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 873.09 us = 0% latency, 5.84 GFLOPS)

gpu009: )

gpu009: (40): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 254.94 GMACs = 0.06% MACs, 8.64 ms = 0.04% latency, 59.03 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 84.98 GMACs = 0.02% MACs, 1.02 ms = 0% latency, 167.3 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 84.98 GMACs = 0.02% MACs, 984.67 us = 0% latency, 172.6 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 84.98 GMACs = 0.02% MACs, 1.03 ms = 0% latency, 165.55 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 859.5 us = 0% latency, 13.79 GFLOPS)

gpu009: )

gpu009: (41): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 220.97 GMACs = 0.05% MACs, 8.62 ms = 0.04% latency, 51.27 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 73.66 GMACs = 0.02% MACs, 1.02 ms = 0% latency, 144.26 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 73.66 GMACs = 0.02% MACs, 986.58 us = 0% latency, 149.32 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 73.66 GMACs = 0.02% MACs, 1.01 ms = 0% latency, 145.42 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 860.93 us = 0% latency, 11.94 GFLOPS)

gpu009: )

gpu009: (42): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 240.35 GMACs = 0.05% MACs, 8.67 ms = 0.04% latency, 55.44 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 80.12 GMACs = 0.02% MACs, 1.04 ms = 0% latency, 154.04 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 80.12 GMACs = 0.02% MACs, 1.05 ms = 0% latency, 153.26 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 80.12 GMACs = 0.02% MACs, 1.05 ms = 0% latency, 153.26 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 884.29 us = 0% latency, 12.64 GFLOPS)

gpu009: )

gpu009: (43): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 9.52 GMACs = 0% MACs, 8.92 ms = 0.04% latency, 2.14 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 3.17 GMACs = 0% MACs, 1.01 ms = 0% latency, 6.31 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 3.17 GMACs = 0% MACs, 983 us = 0% latency, 6.46 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 3.17 GMACs = 0% MACs, 1.22 ms = 0.01% latency, 5.22 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 862.84 us = 0% latency, 513.28 MFLOPS)

gpu009: )

gpu009: (44): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 118.63 GMACs = 0.03% MACs, 8.63 ms = 0.04% latency, 27.5 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 39.54 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 77.71 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 39.54 GMACs = 0.01% MACs, 999.93 us = 0% latency, 79.09 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 39.54 GMACs = 0.01% MACs, 993.73 us = 0% latency, 79.59 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 853.54 us = 0% latency, 6.46 GFLOPS)

gpu009: )

gpu009: (45): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 142.14 GMACs = 0.03% MACs, 8.45 ms = 0.04% latency, 33.63 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 47.38 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 92.05 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 47.38 GMACs = 0.01% MACs, 982.76 us = 0% latency, 96.42 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 47.38 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 93.87 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 859.02 us = 0% latency, 7.69 GFLOPS)

gpu009: )

gpu009: (46): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 105.81 GMACs = 0.02% MACs, 8.58 ms = 0.04% latency, 24.65 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 35.27 GMACs = 0.01% MACs, 1.04 ms = 0% latency, 67.69 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 35.27 GMACs = 0.01% MACs, 992.77 us = 0% latency, 71.05 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 35.27 GMACs = 0.01% MACs, 1.03 ms = 0% latency, 68.77 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 869.75 us = 0% latency, 5.66 GFLOPS)

gpu009: )

gpu009: (47): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 142.03 GMACs = 0.03% MACs, 8.55 ms = 0.04% latency, 33.21 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 47.34 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 92.86 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 47.34 GMACs = 0.01% MACs, 991.11 us = 0% latency, 95.54 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 47.34 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 92.94 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 855.45 us = 0% latency, 7.72 GFLOPS)

gpu009: )

gpu009: (48): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 126.67 GMACs = 0.03% MACs, 8.68 ms = 0.04% latency, 29.18 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 42.22 GMACs = 0.01% MACs, 991.58 us = 0% latency, 85.16 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 42.22 GMACs = 0.01% MACs, 980.14 us = 0% latency, 86.16 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 42.22 GMACs = 0.01% MACs, 1 ms = 0% latency, 84.05 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 859.5 us = 0% latency, 6.85 GFLOPS)

gpu009: )

gpu009: (49): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 4.9 GMACs = 0% MACs, 8.53 ms = 0.04% latency, 1.15 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 1.63 GMACs = 0% MACs, 1.02 ms = 0% latency, 3.2 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 1.63 GMACs = 0% MACs, 1.01 ms = 0% latency, 3.24 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 1.63 GMACs = 0% MACs, 1.01 ms = 0% latency, 3.22 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 864.98 us = 0% latency, 263.4 MFLOPS)

gpu009: )

gpu009: (50): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 1.27 GMACs = 0% MACs, 8.56 ms = 0.04% latency, 295.86 GFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 422.05 MMACs = 0% MACs, 1.1 ms = 0% latency, 769.16 GFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 422.05 MMACs = 0% MACs, 984.43 us = 0% latency, 857.45 GFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 422.05 MMACs = 0% MACs, 1.02 ms = 0% latency, 829.92 GFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 883.1 us = 0% latency, 66.67 MFLOPS)

gpu009: )

gpu009: (51): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 6.44 GMACs = 0% MACs, 8.52 ms = 0.04% latency, 1.51 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 2.15 GMACs = 0% MACs, 1.03 ms = 0% latency, 4.19 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 2.15 GMACs = 0% MACs, 993.25 us = 0% latency, 4.32 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 2.15 GMACs = 0% MACs, 1.02 ms = 0% latency, 4.23 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 867.13 us = 0% latency, 345.42 MFLOPS)

gpu009: )

gpu009: (52): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 61.38 GMACs = 0.01% MACs, 8.58 ms = 0.04% latency, 14.32 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 20.46 GMACs = 0% MACs, 995.4 us = 0% latency, 41.11 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 20.46 GMACs = 0% MACs, 981.81 us = 0% latency, 41.68 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 20.46 GMACs = 0% MACs, 1 ms = 0% latency, 40.78 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 870.23 us = 0% latency, 3.28 GFLOPS)

gpu009: )

gpu009: (53): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 81.75 GMACs = 0.02% MACs, 8.75 ms = 0.04% latency, 18.68 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 27.25 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 53.35 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 27.25 GMACs = 0.01% MACs, 990.87 us = 0% latency, 55 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 27.25 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 53.41 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 876.67 us = 0% latency, 4.34 GFLOPS)

gpu009: )

gpu009: (54): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 140.38 GMACs = 0.03% MACs, 8.74 ms = 0.04% latency, 32.12 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 46.79 GMACs = 0.01% MACs, 1.04 ms = 0% latency, 90.09 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 46.79 GMACs = 0.01% MACs, 1 ms = 0% latency, 93.57 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 46.79 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 91.52 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 869.51 us = 0% latency, 7.51 GFLOPS)

gpu009: )

gpu009: (55): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 129.92 GMACs = 0.03% MACs, 8.58 ms = 0.04% latency, 30.29 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 43.31 GMACs = 0.01% MACs, 1.06 ms = 0% latency, 81.73 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 43.31 GMACs = 0.01% MACs, 978.95 us = 0% latency, 88.48 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 43.31 GMACs = 0.01% MACs, 1 ms = 0% latency, 86.27 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 857.83 us = 0% latency, 7.04 GFLOPS)

gpu009: )

gpu009: (56): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 3.69 GMACs = 0% MACs, 8.55 ms = 0.04% latency, 862.63 GFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 1.23 GMACs = 0% MACs, 1.03 ms = 0% latency, 2.4 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 1.23 GMACs = 0% MACs, 980.85 us = 0% latency, 2.51 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 1.23 GMACs = 0% MACs, 1.01 ms = 0% latency, 2.44 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 863.79 us = 0% latency, 198.57 MFLOPS)

gpu009: )

gpu009: (57): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 75.97 GMACs = 0.02% MACs, 12.62 ms = 0.05% latency, 12.04 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 25.32 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 49.59 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 25.32 GMACs = 0.01% MACs, 990.15 us = 0% latency, 51.15 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 25.32 GMACs = 0.01% MACs, 1.04 ms = 0% latency, 48.86 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 4.8 ms = 0.02% latency, 736.28 MFLOPS)

gpu009: )

gpu009: (58): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 205.17 GMACs = 0.05% MACs, 8.48 ms = 0.04% latency, 48.4 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 68.39 GMACs = 0.02% MACs, 1.02 ms = 0% latency, 134.26 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 68.39 GMACs = 0.02% MACs, 989.91 us = 0% latency, 138.18 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 68.39 GMACs = 0.02% MACs, 1.02 ms = 0% latency, 134.61 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 866.17 us = 0% latency, 11.02 GFLOPS)

gpu009: )

gpu009: (59): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 99.37 GMACs = 0.02% MACs, 8.37 ms = 0.04% latency, 23.75 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 33.12 GMACs = 0.01% MACs, 1.02 ms = 0% latency, 64.8 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 33.12 GMACs = 0.01% MACs, 984.67 us = 0% latency, 67.28 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 33.12 GMACs = 0.01% MACs, 1.01 ms = 0% latency, 65.79 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 860.21 us = 0% latency, 5.37 GFLOPS)

gpu009: )

gpu009: (60): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 1.05 GMACs = 0% MACs, 8.56 ms = 0.04% latency, 244.29 GFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 348.65 MMACs = 0% MACs, 1.02 ms = 0% latency, 682.7 GFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 348.65 MMACs = 0% MACs, 980.14 us = 0% latency, 711.43 GFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 348.65 MMACs = 0% MACs, 1.03 ms = 0% latency, 675.76 GFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 866.89 us = 0% latency, 56.11 MFLOPS)

gpu009: )

gpu009: (61): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 33.36 GMACs = 0.01% MACs, 8.84 ms = 0.04% latency, 7.55 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 11.12 GMACs = 0% MACs, 1.03 ms = 0% latency, 21.49 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 11.12 GMACs = 0% MACs, 989.91 us = 0% latency, 22.47 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 11.12 GMACs = 0% MACs, 1.01 ms = 0% latency, 21.99 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 867.13 us = 0% latency, 1.79 GFLOPS)

gpu009: )

gpu009: (62): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 9.63 GMACs = 0% MACs, 8.66 ms = 0.04% latency, 2.22 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 3.21 GMACs = 0% MACs, 1.03 ms = 0% latency, 6.24 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 3.21 GMACs = 0% MACs, 983.48 us = 0% latency, 6.53 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 3.21 GMACs = 0% MACs, 1.02 ms = 0% latency, 6.32 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 862.36 us = 0% latency, 519.5 MFLOPS)

gpu009: )

gpu009: (63): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 4.07 GMACs = 0% MACs, 8.57 ms = 0.04% latency, 951.01 GFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 1.36 GMACs = 0% MACs, 1.02 ms = 0% latency, 2.68 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 1.36 GMACs = 0% MACs, 983 us = 0% latency, 2.76 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 1.36 GMACs = 0% MACs, 1.03 ms = 0% latency, 2.65 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 867.13 us = 0% latency, 218.47 MFLOPS)

gpu009: )

gpu009: )

gpu009: (shared_expert): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 7.22 TMACs = 1.61% MACs, 41.25 ms = 0.17% latency, 349.88 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 2.41 TMACs = 0.54% MACs, 8.54 ms = 0.04% latency, 563 TFLOPS, in_features=3584, out_features=20480, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 2.41 TMACs = 0.54% MACs, 8.54 ms = 0.04% latency, 562.95 TFLOPS, in_features=3584, out_features=20480, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 2.41 TMACs = 0.54% MACs, 8.35 ms = 0.04% latency, 576.38 TFLOPS, in_features=20480, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 909.81 us = 0% latency, 368.81 GFLOPS)

gpu009: )

gpu009: (shared_expert_gate): Linear(3.58 K = 0.83% Params, 117.44 MMACs = 0% MACs, 1.05 ms = 0% latency, 223.04 GFLOPS, in_features=3584, out_features=1, bias=False)

gpu009: )

gpu009: (input_layernorm): Qwen2MoeRMSNorm(3.58 K = 0.83% Params, 0 MACs = 0% MACs, 1.63 ms = 0.01% latency, 0 FLOPS)

gpu009: (post_attention_layernorm): Qwen2MoeRMSNorm(3.58 K = 0.83% Params, 0 MACs = 0% MACs, 1.57 ms = 0.01% latency, 0 FLOPS)

gpu009: )

gpu009: (2): Qwen2MoeDecoderLayer(

gpu009: 15.36 K = 3.54% Params, 15.4 TMACs = 3.43% MACs, 822.91 ms = 3.47% latency, 37.43 TFLOPS

gpu009: (self_attn): Qwen2MoeFlashAttention2(

gpu009: 4.61 K = 1.06% Params, 962.07 GMACs = 0.21% MACs, 27.81 ms = 0.12% latency, 69.2 TFLOPS

gpu009: (q_proj): Linear(3.58 K = 0.83% Params, 420.91 GMACs = 0.09% MACs, 1.9 ms = 0.01% latency, 442.51 TFLOPS, in_features=3584, out_features=3584, bias=True)

gpu009: (k_proj): Linear(512 = 0.12% Params, 60.13 GMACs = 0.01% MACs, 1.17 ms = 0% latency, 102.63 TFLOPS, in_features=3584, out_features=512, bias=True)

gpu009: (v_proj): Linear(512 = 0.12% Params, 60.13 GMACs = 0.01% MACs, 1.16 ms = 0% latency, 103.32 TFLOPS, in_features=3584, out_features=512, bias=True)

gpu009: (o_proj): Linear(0 = 0% Params, 420.91 GMACs = 0.09% MACs, 1.62 ms = 0.01% latency, 520.23 TFLOPS, in_features=3584, out_features=3584, bias=False)

gpu009: (rotary_emb): Qwen2MoeRotaryEmbedding(0 = 0% Params, 0 MACs = 0% MACs, 994.92 us = 0% latency, 0 FLOPS)

gpu009: )

gpu009: (mlp): Qwen2MoeSparseMoeBlock(

gpu009: 3.58 K = 0.83% Params, 14.44 TMACs = 3.22% MACs, 777.17 ms = 3.28% latency, 37.16 TFLOPS

gpu009: (gate): Linear(0 = 0% Params, 7.52 GMACs = 0% MACs, 1.11 ms = 0% latency, 13.58 TFLOPS, in_features=3584, out_features=64, bias=False)

gpu009: (experts): ModuleList(

gpu009: (0): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 70.46 GMACs = 0.02% MACs, 9.16 ms = 0.04% latency, 15.39 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 23.49 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 41.15 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 23.49 GMACs = 0.01% MACs, 1.1 ms = 0% latency, 42.56 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 23.49 GMACs = 0.01% MACs, 1.13 ms = 0% latency, 41.68 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 1.05 ms = 0% latency, 3.12 GFLOPS)

gpu009: )

gpu009: (1): Qwen2MoeMLP(

gpu009: 0 = 0% Params, 85.82 GMACs = 0.02% MACs, 9.17 ms = 0.04% latency, 18.72 TFLOPS

gpu009: (gate_proj): Linear(0 = 0% Params, 28.61 GMACs = 0.01% MACs, 1.14 ms = 0% latency, 50.28 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (up_proj): Linear(0 = 0% Params, 28.61 GMACs = 0.01% MACs, 1.11 ms = 0% latency, 51.44 TFLOPS, in_features=3584, out_features=2560, bias=False)

gpu009: (down_proj): Linear(0 = 0% Params, 28.61 GMACs = 0.01% MACs, 1.12 ms = 0% latency, 51.23 TFLOPS, in_features=2560, out_features=3584, bias=False)

gpu009: (act_fn): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 967.03 us = 0% latency, 4.13 GFLOPS)

gpu009: )

gpt4o-128k：Qwen2-57B-A14B Mixture-of-Experts (MoE) 模型详解

概述

Qwen2-57B-A14B 是阿里巴巴集团发布的 Qwen2 系列中的一种 Mixture-of-Experts 模型。该模型总参数量为 57B（570 亿），但每个 token 只激活其中的 14B（140 亿）参数。这种模型架构旨在平衡模型复杂度与计算效率，提升模型在不同任务上的表现。

主要参数解释

总参数量 (Total Parameters)：57B，即 570 亿参数。激活参数量 (Activated Parameters)：14B，即 140 亿参数。这意味着每个 token 在计算时，仅有 14B 参数被激活。隐藏层大小 (Hidden Size)：3584层数 (# Layers)：28查询头数 (# Query Heads)：28键值头数 (# KV Heads)：4头大小 (Head Size)：128中间层大小 (Intermediate Size)：2560专家数量 (# Experts)：64，其中激活的专家数量为 8，此外还有共享专家。词汇表大小 (Vocabulary Size)：151,646

模型架构

Qwen2-57B-A14B 采用的是 Transformer 架构，配合 Mixture-of-Experts (MoE) 机制。其核心组件包括：

DeepSpeed Flops Profiler 分析

以下是 DeepSpeed Flops Profiler 提供的一些关键性能指标：

Embedding 层：主要用于输入 token 的嵌入表示。Qwen2MoeDecoderLayer：包含自注意力机制（Qwen2MoeFlashAttention2）和 MoE 块（Qwen2MoeSparseMoeBlock）。

关键组件详解

自注意力机制 (Qwen2MoeFlashAttention2)：

主要包括查询、键、值的线性投影 (Linear) 和旋转嵌入 (RotaryEmbedding)。通过减少 KV 头的数量，优化了内存使用，提升了推理速度。

Mixture-of-Experts 块 (Qwen2MoeSparseMoeBlock)：

包含多个专家，每个专家为一个独立的前馈神经网络 (Qwen2MoeMLP)。使用门控网络 (gate) 来选择激活哪些专家。

前馈神经网络 (Qwen2MoeMLP)：

包含 gate_proj、up_proj 和 down_proj 三个线性层。使用 SiLU 作为激活函数。

归一化层 (Qwen2MoeRMSNorm)：

用于输入和注意力后的归一化处理。世界规模 (World Size)：16数据并行规模 (Data Parallel Size)：16模型并行规模 (Model Parallel Size)：1每 GPU 的批次大小 (Batch Size per GPU)：16每 GPU 的参数量 (Params per GPU)：433.66K每 GPU 的前向 MACs (Fwd MACs per GPU)：449.07 TMACs每 GPU 的前向 FLOPs (Fwd FLOPs per GPU)：898.15 T前向延迟 (Fwd Latency)：23.87 s前向+反向 FLOPs (Fwd+Bwd FLOPs per GPU)：31.5 TFLOPS迭代延迟 (Iter Latency)：86.9 s

Fwd MACs per GPU (449.07 TMACs): The number of multiply-accumulate operations (MACs) performed during a forward pass on each GPU, in teraMACs (trillions of MACs).乘法累加运算

细节分析

性能细节

Qwen2MoeForCausalLM：顶层模块，包含所有的参数和计算。

Qwen2MoeModel：包含嵌入层和多个解码层 (Qwen2MoeDecoderLayer)。

Qwen2MoeDecoderLayer：每层包含自注意力机制和 MoE 块。

自注意力机制 (Qwen2MoeFlashAttention2) 处理查询、键和值的投影，并包含旋转嵌入。MoE 块 (Qwen2MoeSparseMoeBlock) 包含多个前馈神经网络 (Qwen2MoeMLP)，每个前馈神经网络都有 gate_proj、up_proj 和 down_proj 线性层。

Qwen2MoeMLP：前馈神经网络的核心组件，使用 SiLU 作为激活函数，并通过 gate_proj、up_proj 和 down_proj 三个线性层进行计算。

Qwen2MoeRMSNorm：用于层归一化，确保训练的稳定性。

前向和反向传播性能：每 GPU 达到 31.5 TFLOPS，整体性能较高。内存和计算优化：通过减少 KV 头数量和使用高效的注意力机制，优化了内存使用和计算效率。

综上所述，Qwen2-57B-A14B 通过 Mixture-of-Experts 机制和优化的 Transformer 架构，实现了在模型规模和计算效率之间的平衡，能够在多种任务上表现出色，且具备较高的计算性能。

{'loss': 1.4182, 'grad_norm': 0.27865862445647444, 'learning_rate': 9.996199949332658e-06, 'epoch': 0.0}

gpu009: [2024-07-16 20:41:53,521] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 632.55

gpu009: [2024-07-16 20:41:53,522] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 13170.53 | bwd_microstep: 50444.48 | bwd_inner_microstep: 50416.57 | bwd_allreduce_microstep: 27.77 | step_microstep: 875.33

gpu009: [2024-07-16 20:41:53,523] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 13170.52 | bwd: 50444.48 | bwd_inner: 50416.57 | bwd_allreduce: 27.82 | step: 862.07

gpu009: [2024-07-16 20:42:59,993] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 589.17

gpu009: [2024-07-16 20:42:59,994] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 13154.64 | bwd_microstep: 50845.34 | bwd_inner_microstep: 50790.42 | bwd_allreduce_microstep: 54.82 | step_microstep: 920.00

gpu009: [2024-07-16 20:42:59,994] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 13154.64 | bwd: 50845.34 | bwd_inner: 50790.41 | bwd_allreduce: 54.84 | step: 920.00

gpu009: [2024-07-16 20:44:05,911] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 597.05

gpu009: [2024-07-16 20:44:05,912] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 13262.96 | bwd_microstep: 50931.91 | bwd_inner_microstep: 50892.46 | bwd_allreduce_microstep: 39.36 | step_microstep: 894.01

gpu009: [2024-07-16 20:44:05,912] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 13262.94 | bwd: 50931.90 | bwd_inner: 50892.46 | bwd_allreduce: 39.37 | step: 894.01

Claude-3.5-Sonnet

Poe

这个训练日志提供了关于模型训练过程的详细信息。让我逐条解释：

这些日志对于监控训练进度、诊断性能瓶颈以及优化训练过程非常有用。它们提供了训练过程中各个阶段的详细时间breakdown，有助于开发者了解哪些部分可能需要优化。

首先是训练指标:

复制

{'loss': 1.4182, 'grad_norm': 0.27865862445647444, 'learning_rate': 9.996199949332658e-06, 'epoch': 0.0}

loss: 当前批次的损失值，为1.4182grad_norm: 梯度范数，为0.2787learning_rate: 当前学习率，约为1e-5epoch: 当前训练轮数，为0.0（刚开始训练）

接下来是每个迭代步骤的时间统计（以毫秒为单位）:

对于每个迭代，日志记录了以下时间:

optimizer_step: 优化器更新参数所需时间fwd_microstep: 前向传播时间bwd_microstep: 反向传播总时间bwd_inner_microstep: 反向传播内部计算时间bwd_allreduce_microstep: 梯度的all-reduce操作时间（在分布式训练中用于同步梯度）step_microstep: 整个步骤的总时间

例如，第一个迭代

:

前向传播用时约13.17秒反向传播用时约50.44秒梯度all-reduce用时约27.77毫秒整个步骤用时约875.33毫秒

日志显示了三个连续的训练迭代，可以看到每个迭代的时间略有不同，但总体保持一致。

"gpu009"和"[Rank 0]"表示这些日志来自编号为009的GPU，且是分布式训练中的rank 0（主节点）。

时间戳（如[2024-07-16 20:41:53,521]）显示了每条日志的精确记录时间。