U ‰djã @sRddlZddlZddlZddlmZmZmZdddddgZdZ dd „Z d d „Z d d „Z dd„Z eejeejfddœdd„Zde ddfdd„Zdde ddfddœeejeeejeejfeeeeejjeeejddœdd„Zdeejeddœdd„Zdeejeejddœdd„Ze ddfeejeejeddœdd„ZdS)éN)ÚOptionalÚSequenceÚUnionÚ all_reduceÚreduceÚ broadcastÚ all_gatherÚreduce_scattercCstttjdƒst d¡dStƒ}|D]J}|jr4dS| ¡sBdS|jsNdS|  ¡}||krddS|  |¡q$dS)NÚ_nccl_all_reducez)PyTorch is not compiled with NCCL supportFT) ÚhasattrÚtorchÚ_CÚwarningsÚwarnÚsetZ is_sparseZ is_contiguousZis_cudaZ get_deviceÚadd)ZtensorsZdevicesZtensorZdevice©rú3/tmp/pip-unpacked-wheel-ua33x9lu/torch/cuda/nccl.pyÚ is_available s    rcCs0tj ¡}|d?}|d?d@}|d@}|||fS)Né éiÿÿ)r r Z _nccl_version)ÚverÚmajorÚminorÚpatchrrrÚversion"s   rcCs tj ¡S©N)r r Z_nccl_unique_idrrrrÚ unique_id*srcCstj |||¡Sr)r r Z_nccl_init_rank)Z num_ranksÚuidZrankrrrÚ init_rank.sr)ÚinputsÚreturncCs&t|tjjƒrt|tjƒr"tdƒ‚dS)Nz(Inputs should be a collection of tensors)Ú isinstanceÚ collectionsÚabcÚ Containerr ÚTensorÚ TypeError)r rrrÚ_check_sequence_type2sr(cCs4t|ƒ|dkr|}t|ƒtj |||||¡dSr)r(r r r ©r ÚoutputsÚopÚstreamsÚcommsrrrr7s )r*)r ÚoutputÚrootr+r,r*r!cCs’t|ƒ|dk r6|dk r"tdƒ‚qxt d¡||}nBt|tjƒsdt|tjj ƒrdt d¡||}n|dkrt||n|}tj   ||||||¡dS)Nzä'output' and 'outputs' can not be both specified. 'outputs' is deprecated in favor of 'output', taking in a single output tensor. The signature of reduce is: reduce(inputs, output=None, root=0, op=SUM, streams=None, comms=None).zƒnccl.reduce with an output tensor list is deprecated. Please specify a single output tensor with argument 'output' instead instead.z\nccl.reduce with an output tensor list is deprecated. Please specify a single output tensor.) r(Ú ValueErrorrrr"r r&r#r$rr Z _nccl_reduce)r r.r/r+r,r-r*Ú_outputrrrrAs"ÿÿ ÿ )r r/r!cCst|ƒtj ||||¡dSr)r(r r Z_nccl_broadcast)r r/r,r-rrrr`s)r r*r!cCs&t|ƒt|ƒtj ||||¡dSr)r(r r Z_nccl_all_gather)r r*r,r-rrrres)r r*r+r!cCs(t|ƒt|ƒtj |||||¡dSr)r(r r Z_nccl_reduce_scatterr)rrrr ks)rNN)NN)r#rZ torch.cudar ÚtypingrrrÚ__all__ZSUMrrrrr&r(rÚintZcudaZStreamrrrr rrrrÚsH  ûú  ú  ý ý