U 9%emS@srddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZddlmZddlmZdd lmZmZmZmZmZmZd d lm Z d d l!m"Z"m#Z#m$Z$d d l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1e2e3Z4dZ5dZ6eGdddeZ7dZ8ddZ9ddZ:d`e ej;edddZ?ej;e e>dddZ@d d!ZAdad$d%ZBd&d'ZCGd(d)d)ejDZEGd*d+d+ejFZGejHjIdbej;e>ej;d-d.d/ZJGd0d1d1ejFZKGd2d3d3ejFZLGd4d5d5ejFZMGd6d7d7e$ZNGd8d9d9ejFZOGd:d;d;ejFZPGdd?d?ejFZRGd@dAdAejFZSGdBdCdCejFZTGdDdEdEZUdcdGdHZVdIdJZWGdKdLdLejFZXGdMdNdNejFZYGdOdPdPejFZZGdQdRdRejFZ[GdSdTdTejFZ\GdUdVdVejFZ]GdWdXdXejFZ^GdYdZdZejFZ_Gd[d\d\ejFZ`ed]e"Gd^d_d_e$ZadS)dN) dataclass)partial)CallableDictListOptionalSequenceTupleUnion) LayerNorm)is_deepspeed_available) ModelOutput)ContextManagersadd_start_docstrings%add_start_docstrings_to_model_forwardis_scipy_availableloggingreplace_return_docstrings) EsmConfig)ESM_START_DOCSTRINGEsmModelEsmPreTrainedModel) OFProteinRigidRotationatom14_to_atom37 chunk_layercompute_predicted_aligned_error compute_tm-frames_and_literature_positions_to_atom14_posmake_atom14_masksresidue_constantsto_pdbtorsion_angles_to_frameszfacebook/esmfold_v1rc@sTeZdZUdZdZejed<dZejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed <dZejed <dZejed <dZejed <dZejed <dZejed<dZejed<dZejed<dZejed<dZejed<dZejed<dZejed<dZejed<dZejed<dZejed<dZejed<dZejed<dS)EsmForProteinFoldingOutputa! Output type of [`EsmForProteinFoldingOutput`]. Args: frames (`torch.FloatTensor`): Output frames. sidechain_frames (`torch.FloatTensor`): Output sidechain frames. unnormalized_angles (`torch.FloatTensor`): Predicted unnormalized backbone and side chain torsion angles. angles (`torch.FloatTensor`): Predicted backbone and side chain torsion angles. positions (`torch.FloatTensor`): Predicted positions of the backbone and side chain atoms. states (`torch.FloatTensor`): Hidden states from the protein folding trunk. s_s (`torch.FloatTensor`): Per-residue embeddings derived by concatenating the hidden states of each layer of the ESM-2 LM stem. s_z (`torch.FloatTensor`): Pairwise residue embeddings. distogram_logits (`torch.FloatTensor`): Input logits to the distogram used to compute residue distances. lm_logits (`torch.FloatTensor`): Logits output by the ESM-2 protein language model stem. aatype (`torch.FloatTensor`): Input amino acids (AlphaFold2 indices). atom14_atom_exists (`torch.FloatTensor`): Whether each atom exists in the atom14 representation. residx_atom14_to_atom37 (`torch.FloatTensor`): Mapping between atoms in the atom14 and atom37 representations. residx_atom37_to_atom14 (`torch.FloatTensor`): Mapping between atoms in the atom37 and atom14 representations. atom37_atom_exists (`torch.FloatTensor`): Whether each atom exists in the atom37 representation. residue_index (`torch.FloatTensor`): The index of each residue in the protein chain. Unless internal padding tokens are used, this will just be a sequence of integers from 0 to `sequence_length`. lddt_head (`torch.FloatTensor`): Raw outputs from the lddt head used to compute plddt. plddt (`torch.FloatTensor`): Per-residue confidence scores. Regions of low confidence may indicate areas where the model's prediction is uncertain, or where the protein structure is disordered. ptm_logits (`torch.FloatTensor`): Raw logits used for computing ptm. ptm (`torch.FloatTensor`): TM-score output representing the model's high-level confidence in the overall structure. aligned_confidence_probs (`torch.FloatTensor`): Per-residue confidence scores for the aligned structure. predicted_aligned_error (`torch.FloatTensor`): Predicted error between the model's prediction and the ground truth. max_predicted_aligned_error (`torch.FloatTensor`): Per-sample maximum predicted error. Nframessidechain_framesunnormalized_anglesangles positionsstatess_ss_zdistogram_logits lm_logitsaatypeatom14_atom_existsresidx_atom14_to_atom37residx_atom37_to_atom14atom37_atom_exists residue_index lddt_headplddt ptm_logitsptmaligned_confidence_probspredicted_aligned_errormax_predicted_aligned_error)__name__ __module__ __qualname____doc__r'torchZ FloatTensor__annotations__r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=rDrDg/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/esm/modeling_esmfold.pyr&;s0 6r&aJ Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) masking_pattern (`torch.LongTensor` of shape `({0})`, *optional*): Locations of tokens to mask during training as a form of regularization. Mask values selected in `[0, 1]`. num_recycles (`int`, *optional*, defaults to `None`): Number of times to recycle the input sequence. If `None`, defaults to `config.num_recycles`. "Recycling" consists of passing the output of the folding trunk back in as input to the trunk. During training, the number of recycles should vary with each batch, to ensure that the model learns to output valid predictions after each recycle. During inference, num_recycles should be set to the highest value that the model was trained with for maximum accuracy. Accordingly, when this value is set to `None`, config.max_recycles is used. cCsttjk}|ot}|SN)rBZget_autocast_gpu_dtypeZfloat16Zis_autocast_enabled)Z fp16_enabledrDrDrEis_fp16_enableds rGcCs:tr dSzddl}|jWStk r4YdSXdS)NFr)r deepspeedutilsZis_initialized Exception)rHrDrDrEis_deepspeed_initializeds rK)samplespad_vreturncCst|dkrtStdd|DdkrBtddd|Dtdd|D\}d dtd d|DD}tjt|f||dj|d }||t t|D],}||}||}||td d |j D<q|S)z Takes a list of tensors with the following dimensions: [(d_11, ..., d_1K), (d_21, ..., d_2K), ..., (d_N1, ..., d_NK)] and stack + pads them into a single tensor of: (N, max_i=1,N { d_i1 }, ..., max_i=1,N {diK}) rcSsh|] }|qSrDdim.0xrDrDrE sz(collate_dense_tensors..rz Samples has varying dimensions: cSsg|] }|qSrDrOrQrDrDrE sz)collate_dense_tensors..cSsh|] }|jqSrDdevicerQrDrDrErTscSsg|] }t|qSrD)max)rRlstrDrDrErUscSsg|] }|jqSrDshaperQrDrDrErUs)dtyperWcss|]}td|VqdS)rNslice)rRkrDrDrE sz(collate_dense_tensors..) lenrBTensor RuntimeErrortuplezipemptyr\fill_ranger[)rLrMrWZ max_shaperesultiZresult_itrDrDrEcollate_dense_tensorss " rlrkZno_dimscCs||jd| dS)N))reshaper[rmrDrDrEflatten_final_dimssrp)tensorindscsBdt|ttt|jd}||fdd|DS)Nrncsg|] }|qSrDrD)rRrjZ zero_indexrDrErUsz&permute_final_dims..)ralistrhr[permute)rqrrZ first_indsrDrsrEpermute_final_dimss rvcs\|d}i}|D]B\}fdd|D}t|tkrJt|||<q|||<q|S)Nrcsg|] }|qSrDrD)rRdr_rDrErUsz!dict_multimap..)itemstypedict dict_multimap)fnZdictsfirstZnew_dictvZall_vrDrxrEr|s r|?fan_incCs|j}|td|d}tsTtdt|}tjj j ||dj dd|dnbddl m }t||jd d ddd }|jd d d||d }t||}|tj||jd dS)NrzlThis init requires scipy, but scipy was not found, default to an approximation that might not be equivalent.)std@)minrXr) truncnorm)ablocscale)rrrrsizerV)r[rXrloggerwarningmathsqrtrBnninitZnormal_clampZ scipy.statsrrZrvsZnumelnprocopy_rqrW)weightsrfanr[rrrLrDrDrEtrunc_normal_init_s "  rc Cs&td}||W5QRXdS)NgabR?)rBno_gradrg)rZsoftplus_inverse_1rDrDrEipa_point_weights_init_s rc sDeZdZdZdeeeeeee j e j gdfdfdd Z Z S) EsmFoldLinearz A Linear layer with built-in nonstandard initializations. Called just like torch.nn.Linear. Implements the initializers in 1.11.4, plus some additional ones found in the code. TdefaultN)in_dimout_dimbiasrinit_fnc sVtj|||d|r6t|jdW5QRX||_||_|dkrRtddS)aM Args: in_dim: The final dimension of inputs to the layer out_dim: The final dimension of layer outputs bias: Whether to learn an additive bias. True by default init: The initializer to use. Choose from: "default": LeCun fan-in truncated normal initialization "relu": He initialization w/ truncated normal distribution "glorot": Fan-average Glorot uniform initialization "gating": Weights=0, Bias=1 "normal": Normal initialization with std=1/sqrt(fan_in) "final": Weights=0, Bias=0 Overridden by init_fn if the latter is not None. init_fn: A custom initializer taking weight and bias as inputs. Overrides init if not None. rr)rreluglorotgatingnormalfinalzInvalid init string.N) super__init__rBrrrgrr ValueError)selfrrrrr __class__rDrErs zEsmFoldLinear.__init__)TrN) r>r?r@rAintboolstrrrrBrbr __classcell__rDrDrrEr s rcs&eZdZdfdd ZddZZS)EsmFoldLayerNormh㈵>cs@t|f|_||_tt||_tt ||_ dSrF) rrc_inepsr ParameterrBZonesweightzerosr)rrrrrDrEr:s  zEsmFoldLayerNorm.__init__c Cs|j}|tjkr`ts`tjjjdd2tj ||j |j j |d|j j |d|j}W5QRXntj ||j |j |j |j}|S)NFenabledr\)r\rBbfloat16rKcudaampautocastr functional layer_normrrtorr)rrSrwoutrDrDrEforwardCs 8zEsmFoldLayerNorm.forward)rr>r?r@rrrrDrDrrEr9s rrn)rkrPrNc Cs\|j}|tjkrFtsFtjjjddtjjj ||d}W5QRXntjjj ||d}|S)z[ Softmax, but without automatic casting to fp32 when the input is of type bfloat16 FrrO) r\rBrrKrrrrrsoftmax)rkrPrwsrDrDrEsoftmax_no_castNs rc seZdZdZdeeeeeedfdd Zejeje ejejejfdddZ ejejejd d d Z dejeje e ejeeeeee ejejd ddZZS)EsmFoldAttentionzu Standard multi-head attention using AlphaFold's default layer initialization. Allows multiple bias vectors. T)c_qc_kc_vc_hiddenno_headsrcst||_||_||_||_||_||_t|j|j|jddd|_ t|j|j|jddd|_ t|j|j|jddd|_ t|j|j|jdd|_ d|_ |jrt|j|j|jdd|_ t|_dS)a Args: c_q: Input dimension of query data c_k: Input dimension of key data c_v: Input dimension of value data c_hidden: Per-head hidden dimension no_heads: Number of attention heads gating: Whether the output should be gated using query data FrrrrrNr)rrrrrrrrrlinear_qlinear_klinear_vlinear_olinear_grSigmoidsigmoid)rrrrrrrrrDrErbs zEsmFoldAttention.__init__)q_xkv_xrNcCs||}||}||}||jdd|jdf}||jdd|jdf}||jdd|jdf}|dd}|dd}|dd}|t|j }|||fS)Nrnr) rrrviewr[r transposerrr)rrrqr_rrDrDrE _prep_qkvs      zEsmFoldAttention._prep_qkv)orrNcCsX|jdk r@|||}||jdd|jdf}||}t|d}||}|S)Nrnr)rrrr[rrpr)rrrgrDrDrE_wrap_ups   zEsmFoldAttention._wrap_upNF) rrbiasesuse_memory_efficient_kerneluse_lmalma_q_chunk_sizelma_kv_chunk_size use_flash flash_maskrNc Cs|r|dks|dkrtd|r0|dk r0td|||g} t| dkrNtd|dkrZg}|||\} } } t| d} t| | }|D] }||7}qt|d}t|| }|dd }|||}|S) a{ Args: q_x: [*, Q, C_q] query data kv_x: [*, K, C_k] key data biases: List of biases that broadcast to [*, H, Q, K] use_memory_efficient_kernel: Whether to use a custom memory-efficient attention kernel. This should be the default choice for most. If none of the "use_<...>" flags are True, a stock PyTorch implementation is used instead use_lma: Whether to use low-memory attention (Staats & Rabe 2021). If none of the "use_<...>" flags are True, a stock PyTorch implementation is used instead lma_q_chunk_size: Query chunk size (for LMA) lma_kv_chunk_size: Key/Value chunk size (for LMA) Returns [*, Q, C_q] attention update NzPIf use_lma is specified, lma_q_chunk_size and lma_kv_chunk_size must be providedzSuse_flash is incompatible with the bias option. For masking, use flash_mask insteadrz2Choose at most one alternative attention algorithm)rrrnrr) rsumrrvrBmatmulrrr)rrrrrrrrrrZ attn_optionsquerykeyvalueoutputrrDrDrErs&!          zEsmFoldAttention.forward)T)NFFrrFN)r>r?r@rArrrrBrbr rrrrrrrDrDrrEr]s> /& rc sveZdZd fdd Zejjdejeeje e e e ejdddZ deje eje e e e e ejd d d Z ZS)EsmFoldTriangleAttentionTeAcsht||_||_||_||_||_t|j|_t ||jddd|_ t |j|j|j|j|j|_ dS)z Args: c_in: Input channel dimension c_hidden: Overall hidden channel dimension (not per-head) no_heads: Number of attention heads FrrN) rrrrrstartinginfr rrlinearrmha)rrrrrrrrDrErs  z!EsmFoldTriangleAttention.__init__F)rSr chunk_sizerr inplace_saferNcCs@|||d}tt|j||d||t|jdd|r8|nddS)ztriangle! triangle!)rrr)rrNr)rZ no_batch_dimsZ_out)rrrrar[)rrSrrrrrZ mha_inputsrDrDrE_chunks  zEsmFoldTriangleAttention._chunkN)rSmaskrrrrrNc Cs|dkr||jdd}|js:|dd}|dd}||}|j|ddddddddf}t||d}|d}||g} |dk r|j || ||||d }n|j ||| ||d }|js|dd}|S) z Args: x: [*, I, J, C_in] input tensor (e.g. the pair representation) Returns: [*, I, J, C_in] output tensor Nrnrrr.rrr)rrr)rrrrr) new_onesr[rrrrrvr unsqueezerr) rrSrrrrrZ mask_biasZ triangle_biasrrDrDrEr's>    $   z EsmFoldTriangleAttention.forward)Tr)FFF)NNFFF)r>r?r@rrBjitignorerbrrrrrrrrDrDrrErs8rcseZdZdZdfdd ZdejejeeejdddZ dejeejeee d d d Z dejeeje e eeejdddZ Z S)#EsmFoldTriangleMultiplicativeUpdatez* Implements Algorithms 11 and 12. Tcst|j}||_t|||_t||dd|_t|||_t||dd|_t||dd|_ t||dd|_ t ||_ t ||_ t|_dS)Nrrr)rrpairwise_state_dim _outgoingr linear_a_p linear_a_g linear_b_p linear_b_grlinear_zr layer_norm_inlayer_norm_outrrr)rconfigrrrrDrErhs     z,EsmFoldTriangleMultiplicativeUpdate.__init__N)rr_inplace_chunk_sizerNcCs|jrt|d}t|d}nt|d}t|d}|dk rtd|jd|D]l}|d|||ddddf}|d|||ddddf}t|||d|||ddddf<qJ|}n t||}t|dS)Nr)rrrrr.rrr)rrvrhr[rBr)rrrr rjZa_chunkb_chunkprDrDrE_combine_projectionsys       " z8EsmFoldTriangleMultiplicativeUpdate._combine_projections)zrinplace_chunk_sizewith_addc s8|dkr||jdd}|d}dfdd dfdd }|||ddd}dk r|jdd d d d jrn}d d fddfdd}t|j} | <|| } | } td| <| || d} ttd} ddt | | ddgD}tt}fdd|D}t | |||}|D]j\}}| s|kr|| |} d} |||||}|||||}| }|krڈ||||}nN| s|}td|<| |||||<n|}| |||}|||ddd}~t ||}t |d}|}|}||||}|}|~||9}|}t||||<|r|||7<n|||<qhn\|||dd}t ||}|}|}|}|||9}|r0||7}n|}|S)a Args: z: A [*, N, N, C_z] pair representation mask: A [*, N, N] pair mask inplace_chunk_size: Size of chunks used in the main computation. Increase to trade memory for speed. with_add: If True, z is overwritten with (z + update). Otherwise, it is overwritten with (update). Returns: A reference to the overwritten z More memory-efficient, inference-only version of the forward function. Uses in-place operations, fusion of the addition that happens after this module in the Evoformer, a smidge of recomputation, and a cache of overwritten values to lower peak memory consumption of this module from 5x the size of the input tensor z to 2.5x its size. Useful for inference on extremely long sequences. It works as follows. We will make reference to variables used in the default forward implementation below. Naively, triangle multiplication attention requires the manifestation of 5 tensors the size of z: 1) z, the "square" input tensor, 2) a, the first projection of z, 3) b, the second projection of b, 4) g, a z-sized mask, and 5) a z-sized tensor for intermediate computations. For large N, this is prohibitively expensive; for N=4000, for example, z is more than 8GB alone. To avoid this problem, we compute b, g, and all intermediate tensors in small chunks, noting that the chunks required to compute a chunk of the output depend only on the tensor a and corresponding vertical and horizontal chunks of z. This suggests an algorithm that loops over pairs of chunks of z: hereafter "columns" and "rows" of z, even though each "column" and "row" in fact contains inplace_chunk_size contiguous true columns and rows of z. Writing output chunks to a new tensor would bring total memory consumption down to 3x the size of z. However, more memory can be saved by writing output chunks directly to z in-place. WLOG, we choose to write output chunks vertically, overwriting the ith "column" of z at the end of the ith iteration of the main loop. Despite this overwriting, the ith column is always one column ahead of previously overwritten columns and can be recovered directly from z. After the first iteration, however, the ith row of z is always at least partially overwritten. For this reason, we introduce the z-cache, a tensor one-half the size of z. The z-cache initially contains the left half (2nd and 3rd quadrants) of z. For 0 < i < N/2, the missing left part of the ith row of z is recovered from this cache at the beginning of the ith iteration. Once i exceeds n/2, the cache is "reoriented" to encompass the 3rd and 4th quadrants of z instead. Though the 3rd quadrant of the original z is entirely overwritten at this point, it can be recovered from the z-cache itself. Thereafter, the ith row of z can be recovered in its entirety from the reoriented z-cache. After the final iteration, z has been completely overwritten and contains the triangular multiplicative update. If with_add is True, it instead contains the sum of z and the triangular multiplicative update. In either case, peak memory consumption is just 2.5x the size of z, disregarding memory used for chunks and other small variables. NrnTcsZ|rj}j}n j}j}|}||}||||9}||9}t|d}|S)Nr)rrrrr sigmoid_rv)pairrrrZlinear_prrrDrEcompute_projection_helpers   zYEsmFoldTriangleMultiplicativeUpdate._inference_forward..compute_projection_helperc s:j|A}|s.|||}|r*|dd}n|r8jnj}|jjd}|jdd|f|jdd}||}td|jdD]} |d| | ddddf} |d| | ddddf|d| | ddddf|} |r| dd} | |d| | f<n| |d| | ddf<~ q|S)Nrnrrr.)rrrrrr[ new_zerosrh) rrrchunkedZneed_transposerrcZ out_shaperjZ pair_chunk)rrrrDrEcompute_projections,   "   zREsmFoldTriangleMultiplicativeUpdate._inference_forward..compute_projection)rrrrcSsdd|jDS)NcSsg|] }tdqSrFr]rR_rDrDrErUsz`EsmFoldTriangleMultiplicativeUpdate._inference_forward..empty_slicer..rZ)rkrDrDrE empty_slicerszLEsmFoldTriangleMultiplicativeUpdate._inference_forward..empty_slicercs|}t||||<||SrFr])rkstartendrPr)rrDrE slice_tensorszLEsmFoldTriangleMultiplicativeUpdate._inference_forward..slice_tensorcs|d}|}|dddddddf}|}td|<|||<|d}|d}|}td|<|||<|S)N.rr)rr^)z_cacherZ quadrant_3Zfirst_half_slicerZ quadrant_4Zquadrant_3_slicer)col_dimrhalf_nnrow_dimr!rDrE flip_z_cache_ s  zMEsmFoldTriangleMultiplicativeUpdate._inference_forward..flip_z_cache_rFcSsg|]\}}||qSrDrD)rRZi_1Zi_2rDrDrErU4szJEsmFoldTriangleMultiplicativeUpdate._inference_forward..rcsg|]}qSrDrDr)rrDrErU6s)rrr )T)TT)rr[rrrtrr^rrhreclonerBrrvr rrr r) rrrrrrrZ b_chunk_dimr'Z z_cache_shaper"Zz_cache_slicerZz_cache_rotatedZi_rangeZinitial_offsetsZ after_halfZafter_half_offsetsZcombined_range_with_offsetsrjoffsetZ z_chunk_bZ mask_chunkZz_chunk_slicerZz_cache_offsetrZx_chunkZ z_chunk_gZg_chunkZz_slicerrrSrrD) r#rrr$rr%r&rr!rE_inference_forwards1 !     "           z6EsmFoldTriangleMultiplicativeUpdate._inference_forwardF)rrr_add_with_inplacer rNc Cs|r|j||||d}|S|dkr6||jdd}|d}||}|}||||}|||}|}||||}|| |}t rt j j jdd|||}W5QRXn |||}~~||}||}|||} || }|S)z Args: x: [*, N_res, N_res, C_z] input tensor mask: [*, N_res, N_res] input mask Returns: [*, N_res, N_res, C_z] output tensor )rrNrnFr)r*rr[rr rrrrrrGrBrrrrfloatr rr) rrrrr,r rSrrrrDrDrErvs8      z+EsmFoldTriangleMultiplicativeUpdate.forward)T)N)NNT)NFFr+)r>r?r@rArrBrbrrrrr*rrrDrDrrErcs>  grcs eZdZdZfddZZS)EsmFoldPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. c st|trt|jdk r0||j|jn|jdkrJt|jddn|jdkrdt|jddn~|jdkrt jj |jdd n`|jd kr|j d |jr|j dn6|jd krtj jj |jd dn|jdkr|j d W5QRXnt|t r t|jnt|trtj j|jjjtj j|jjjtj j|jjjtj j|jjjtj j|jjjjtj j|jjjjtj j|jjjjtj j|jjjjtj j|jjjtj j|jjjtj j|jjjtj j|jjjtj j|jjjtj j|jjdjtj j|jjdjtj j|j jdjtj j|j jdjn t!"|dS)zInitialize the weightsNrr)rrrrr)Zgainrrrr)Z nonlinearityrr)# isinstancerrBrrrrrrrZxavier_uniform_rgZkaiming_normal_EsmFoldInvariantPointAttentionr head_weights#EsmFoldTriangularSelfAttentionBlockzeros_ tri_mul_inr tri_mul_out tri_att_startrr tri_att_endsequence_to_pairo_projpair_to_sequencer seq_attentionmlp_seqmlpmlp_pairr _init_weights)rmodulerrDrEr?sN            z$EsmFoldPreTrainedModel._init_weights)r>r?r@rAr?rrDrDrrEr.sr.cs(eZdZdfdd ZdddZZS) EsmFoldSelfAttentionFcst|||kst||_||_||_tj||ddd|_tj||dd|_ ||_ |rt|||_ t jj |j jt jj |j j|jd|_t jj |j jdS)Nr FrT)rrAssertionError embed_dim num_heads head_widthrLinearprojr9gatedg_projrBrr3rZones_rrescale_factor)rrDrErFrIrrDrErs  zEsmFoldSelfAttention.__init__Nc Cs||j|jdd|jdf}|dddd}|jddd\}}}|j|}td||} |dk r|| |dddd} |dk r|ddddf}| |d kt j } t j j| dd} td | |} | j| jddd } |jr||| } || } | | ddddfS) a Basic self attention with optional mask and external pairwise bias. To handle sequences of different lengths, use mask. Inputs: x: batch of input sequneces (.. x L x C) mask: batch of boolean masks where 1=valid, 0=padding position (.. x L_k) bias: batch of scalar pairwise attention biases (.. x Lq x Lk x num_heads) Outputs: sequence projection (B x L x embed_dim), attention maps (B x L x L x num_heads) Nrrnrrr rOz...qc,...kc->...qkFz...hqk,...hkc->...qhc)rn)rHrr[rEruchunkrKrBZeinsum masked_fillrrrrrrorIrJrr9) rrSrrindicesrkrr_rryrDrDrErs" $  zEsmFoldSelfAttention.forward)F)NNNrrDrDrrErAsrAcsHeZdZdZeeeeefdfdd Ze j e j dddZ Z S)EsmFoldDropoutzl Implementation of dropout with the ability to share the dropout mask along a particular dimension. )r batch_dimcs:t||_t|tkr"|g}||_t|j|_dSrF) rrrQrzrrRrDropoutdropout)rrQrRrrDrEr#s   zEsmFoldDropout.__init__)rSrNcCs<t|j}|jdk r(|jD] }d||<q||||S)Nr)rtr[rRrTr)rrSr[ZbdrDrDrEr,s     zEsmFoldDropout.forward) r>r?r@rAr-r rrrrBrbrrrDrDrrErPs rPcs$eZdZfddZddZZS)EsmFoldSequenceToPaircsjtt||_tj||ddd|_tjd||dd|_tjj |jj tjj |jj dS)NrTr) rrrr layernormrGrHr9rBrr3r)rsequence_state_dim inner_dimrrrDrEr5s   zEsmFoldSequenceToPair.__init__cCst|jdkst||}||}|jddd\}}|dddddddf|dddddddf}|dddddddf|dddddddf}tj||gdd}||}|S)z Inputs: sequence_state: B x L x sequence_state_dim Output: pairwise_state: B x L x L x pairwise_state_dim Intermediate state: B x L x L x 2*inner_dim r rrnrON) rar[rCrVrHrLrBcatr9)rsequence_staterrr_proddiffrSrDrDrEr?s   88 zEsmFoldSequenceToPair.forwardrrDrDrrErU4s rUcs$eZdZfddZddZZS)EsmFoldPairToSequencecs,tt||_tj||dd|_dS)NFr)rrrr rVrGr)rrrErrDrEr[s  zEsmFoldPairToSequence.__init__cCs*t|jdkst||}||}|S)z Inputs: pairwise_state: B x L x L x pairwise_state_dim Output: pairwise_bias: B x L x L x num_heads )rar[rCrVr)rpairwise_staterZ pairwise_biasrDrDrEras  zEsmFoldPairToSequence.forwardrrDrDrrEr]Zs r]cs&eZdZdfdd ZddZZS)EsmFoldResidueMLPrc sBttt|t||tt||t||_dSrF) rrr Sequentialr rGReLUrSr=)rrDrXrTrrDrErps   zEsmFoldResidueMLP.__init__cCs|||SrF)r=)rrSrDrDrEr{szEsmFoldResidueMLP.forward)rrrDrDrrEr`os r`cs&eZdZfddZdddZZS)r2cst||_|j}|j}||j}||j}t||_ t ||d||_ t |||_ t|||jdd|_t|dd|_t|dd|_t||j|ddd|_t||j|ddd|_t|d||jd |_t|d||jd |_t|j|_t|jdd|_t|jdd |_dS) NrT)rI)rFr)rrr^)rTr)rrr rWrZsequence_head_widthZpairwise_head_widthrr layernorm_1rUr8r]r:rAr;rr5r4rr6r7r`rTr<r>rSdroprProw_dropcol_drop)rr rWrZsequence_num_headsZpairwise_num_headsrrDrErsH     z,EsmFoldTriangularSelfAttentionBlock.__init__NcKs0t|jdkr$tdt|jdt|jdkrHtdt|jd|dk rtt|jdkrttdt|jd|j\}}}|jd} ||jjkrtd |d |jjd | |jjkrtd | d |jjd ||jd krtd|d |jd d ||jdks"||jdkrJtd|d |jdd|jdd ||} ||} |j| || d\} } || | }| |}|| |}|dk r| d| dnd} || |j|| d}|||j|| d}|| |j|| |d}|||j|| |d}||}||fS)a* Inputs: sequence_state: B x L x sequence_state_dim pairwise_state: B x L x L x pairwise_state_dim mask: B x L boolean tensor of valid positions Output: sequence_state: B x L x sequence_state_dim pairwise_state: B x L x L x pairwise_state_dim r z,`sequence_state` should be a 3d-tensor, got  dims.r^z,`pairwise_state` should be a 4d-tensor, got Nr"`mask` should be a 2d-tensor, got zQ`sequence_state` last dimension should be equal to `self.sequence_state_dim`. Got != .zR`pairwise_state` last dimension should be equal to `self.pairwise_state_dim`. Got rzD`sequence_state` and `pairwise_state` have inconsistent batch size: rzI`sequence_state` and `pairwise_state` have inconsistent sequence length: z or )rrr)rr)rar[rr rWrr:rcr;rdr<r8rrer5rfr4r6r7r>)rrZr_rrZ,_EsmFoldTriangularSelfAttentionBlock__kwargsrRseq_dimrWrrrOrZtri_maskrDrDrErsR      "   " z+EsmFoldTriangularSelfAttentionBlock.forward)NNrrDrDrrEr2s "r2c@s&eZdZd ddZddZdd Zd S) EsmCategoricalMixture2rrcCsH||_tj|||d|jj|jjd}|dd|ddd|_dS)NrrWr\rnr)logitsrBlinspacerWr\v_bins)rparambinsrr rDrDrErs zEsmCategoricalMixture.__init__cCsN|d|jdg|jd}|jd}tj||ddd dS)NrnrO) rrrndimabsZargminrpZ log_softmaxrBZtake_along_dimsqueeze)rtrueZ true_indexZnllrDrDrElog_probs& zEsmCategoricalMixture.log_probcCs|jd|jddS)Nrnr)rprrrrrwrrDrDrEmeanszEsmCategoricalMixture.meanN)rnrr)r>r?r@rryrzrDrDrDrErms rmrncCst||dS)Nrt)rmrz)rprtrDrDrEcategorical_lddtsr|cCsb|dkr dSt|jdkr0tdt|jd|j\}}|d|||}||||}|S)z Helper to convert B x L mask of valid positions to axial mask used in row column attentions. Input: mask: B x L tensor of booleans Output: mask: B x L x L tensor of booleans Nrrhrgr)rar[rrexpandro)rrRrlmrDrDrEget_axial_masks  rcs&eZdZfddZdddZZS)EsmFoldRelativePositioncs2t|j|_tjd|jd|j|_dSNr) rrZ position_binsrtrBr Embeddingr embeddingrr rrDrErs z EsmFoldRelativePosition.__init__NcCs|jtjkrtd|jd|dk rL|j|jkrLtd|jd|jd|dddddf|dddddf}||j |j}||jd}|dk r|dddddf|dddddf}d||d k<||}|S) z Input: residue_index: B x L tensor of indices (dytpe=torch.long) mask: B x L tensor of booleans Output: pairwise_state: B x L x L x pairwise_state_dim tensor of embeddings z`residue_index` has dtype z, it should be `torch.long`.Nz5`residue_index` and `mask` have inconsistent shapes: rirjrrF)r\rBlongrr[rrtr)rr6rr\rrDrDrErs ,,  zEsmFoldRelativePosition.forward)NrrDrDrrErs rcs0eZdZfddZejejdddZZS)EsmFoldAngleResnetBlockcs@tt|j|jdd|_t|j|jdd|_t|_dSNrrr) rrr resnet_dimlinear_1linear_2rrbrrrrDrEr:s z EsmFoldAngleResnetBlock.__init__)rrNcCs4|}||}||}||}||}||SrF)rrr)rr s_initialrDrDrErBs     zEsmFoldAngleResnetBlock.forward)r>r?r@rrBrbrrrDrDrrEr9s rcsBeZdZdZfddZejejeejejfdddZZ S)EsmFoldAngleResnetz. Implements Algorithm 20, lines 11-14 cst||_t|j|j|_t|j|j|_t |_ t |j D]}t |}|j |qDt|j|jd|_t|_dSr)rrr r sequence_dimr linear_inlinear_initialr ModuleListlayersrhZnum_resnet_blocksrappendZ num_angles linear_outrbr)rr rlayerrrDrErRs  zEsmFoldAngleResnet.__init__)rrrNcCs||}||}||}||}||}|jD] }||}q6||}||}||jddd}|}ttj tj |dddd|j j d}||}||fS)a Args: s: [*, C_hidden] single embedding s_initial: [*, C_hidden] single embedding as of the start of the StructureModule Returns: [*, no_angles, 2] predicted angles Nrn)rnrrT)rPZkeepdim)r) rrrrrrr[rBrrrr epsilon)rrrlZunnormalized_sZ norm_denomrDrDrErbs&        zEsmFoldAngleResnet.forward r>r?r@rArrBrbr rrrDrDrrErMs rc sReZdZdZfddZd ejeejeeje ee ejejdddZ Z S) r0z" Implements Algorithm 22. cs t||_|j}|j}|j|_|j|_|j |_ |j |_ |j|j}t |||_ t |d||_ |j|j d}t |||_|j|j |j d}t |||_t ||j|_tt|j|_|j||j|j d}t ||dd|_tjdd|_t|_dS)Nrr r^rrrnrO)rrr r pairwise_dimZipa_dim hidden_dimZ num_heads_iparE num_qk_points num_v_pointsrr linear_kvlinear_q_pointslinear_kv_pointslinear_brrrBrr1rZSoftmaxrZSoftplussoftplus)rr c_sc_zZhcZhpqZhpkvZconcat_out_dimrrDrErs*     z'EsmFoldInvariantPointAttention.__init__FN)rrrQr_offload_inference_z_reference_listrNc Cs<|g}||}||}||jdd|jdf}||jdd|jdf}tj||jdd\} } ||} tj| | jdddd} tj | dd} |d | } | | jdd|j|j df} | |} tj| | jdddd} tj | dd} |d | } | | jdd|jddf} tj| |j |j gdd\} }||d}|rt|ddksvt|d|d<trtjjjd d &tt|d t| d }W5QRXntt|d t| d }|td d|j9}|tdt|d7}| d| d}|d}ttj|dd}||j jdt!|jddd}|td d|j dd}||}tj|ddd}|d|d}|j"j#|d}t|d}||}||d}|$|}t|| %ddj&|j'd%dd}t(|d}tj|dddddddft|dddddddfdd}t|d}|d)|}t(ttj|ddd|j"j*d}|j+|jddd}|r|d&|j,|d<t|%dd|dj&|j'd}t(|d}|-tj.|ftj|dd||fddj&|dj'd}|S)ab Args: s: [*, N_res, C_s] single representation z: [*, N_res, N_res, C_z] pair representation r: [*, N_res] transformation object mask: [*, N_res] mask Returns: [*, N_res, C_s] single representation update NrnrOr ).NrrrFr)rrrr rgUUUUUU?rr)r)rnrg"@rBrrr.)rr rr)rrr r).NN)rnr )/rrrr[rErBsplitrrstackapplyrrrrsys getrefcountrCcpurGrrrrrvr-rrrrZunbindrr1rar rrrrr\rpZ invert_applyrrorWrrY)rrrrQrrrrkvr_rZq_ptsZkv_ptsZk_ptsZv_ptsrrZpt_attr1Z square_maskrZo_ptZ o_pt_normZo_pairrDrDrErs   "    (   & 6 &" 0z&EsmFoldInvariantPointAttention.forward)FN) r>r?r@rArrBrbrrrrrrrDrDrrEr0s ) r0cs>eZdZdZfddZejeejejfdddZZ S)EsmFoldBackboneUpdatez* Implements part of Algorithm 23. cs tt|jddd|_dS)Nrr)rrrrrrrrDrErXs zEsmFoldBackboneUpdate.__init__)rrNcCs||}|S)z Args: [*, N_res, C_s] single representation Returns: [*, N_res, 6] update vector )r)rrupdaterDrDrEr]s zEsmFoldBackboneUpdate.forwardrrDrDrrErSs rcs$eZdZfddZddZZS)%EsmFoldStructureModuleTransitionLayercsTtt|j|jdd|_t|j|jdd|_t|j|jdd|_t|_ dSr) rrrrrrlinear_3rrbrrrrDrErks  z.EsmFoldStructureModuleTransitionLayer.__init__cCsB|}||}||}||}||}||}||}|SrF)rrrr)rrrrDrDrErts     z-EsmFoldStructureModuleTransitionLayer.forwardrrDrDrrErjs rcs$eZdZfddZddZZS) EsmFoldStructureModuleTransitioncs\t||_t|_t|jD]}t|}|j |q$t |j |_ t |j|_dSrF)rrr rrrrhZnum_transition_layersrrrS dropout_raterTr rr)rr rrrrDrErs  z)EsmFoldStructureModuleTransition.__init__cCs,|jD] }||}q||}||}|SrF)rrTr)rrrrDrDrErs     z(EsmFoldStructureModuleTransition.forwardrrDrDrrErs rcs>eZdZfddZd ddZddZd d Zd d ZZS)EsmFoldStructureModulecs~t||_t|j|_t|j|_t|j|j|_ t ||_ t |j|_t|j|_t||_t||_t||_dSrF)rrr r r layer_norm_sr layer_norm_zrrr0iparrSr ipa_dropoutlayer_norm_ipar transitionr bb_updater angle_resnetrrrDrErs       zEsmFoldStructureModule.__init__NFc Cs|d}|dkr$||jdd}||}||d}d}|rtt|ddksZt|d|d<|g}d}|}||}t j |jdd|j |j |j dd} g} t|jjD]} ||j||| |||d}||}||}||}| ||} t t| dd | } | |jj} |||\} }|| ||}|||}| |jj}| |!| |||d }| "|| #} q~~|r|d$|j |d<t%t&j'| } || d<| S) a Args: evoformer_output_dict: Dictionary containing: "single": [*, N_res, C_s] single representation "pair": [*, N_res, N_res, C_z] pair representation aatype: [*, N_res] amino acid indices mask: Optional [*, N_res] sequence mask Returns: A dictionary of outputs singleNrnrrZquat)fmt)rr)Zrot_matsZquats)r'r(r)r*r+r,)(rr[rrrrrCrrridentityr\rWZtrainingrhr  num_blocksrrrrZcompose_q_update_vecrrget_rotsZ get_rot_matsZ get_transZscale_translationZtrans_scale_factorrr%r!Z to_tensor_7Z to_tensor_4x4rZstop_rot_gradientrr|rBr)rZevoformer_output_dictr1rrrrZz_reference_listrZrigidsoutputsrjZbackb_to_globalr)r*Zall_frames_to_globalZpred_xyzZ scaled_rigidspredsrDrDrErst          zEsmFoldStructureModule.forwardcCst|ds*|jdtjtj||ddddt|dsR|jdtjtj|ddddt|ds||jdtjtj||ddddt|ds|jdtjtj||dddddS) Ndefault_framesF)r\rW requires_grad) persistent group_idx)rWr atom_mask lit_positions) hasattrregister_bufferrBrqr#Z!restype_rigid_group_default_frameZrestype_atom14_to_rigid_groupZrestype_atom14_maskZ$restype_atom14_rigid_group_positions)rZ float_dtyperWrDrDrE_init_residue_constants"sV    z.EsmFoldStructureModule._init_residue_constantscCs ||j|jt||||jSrF)rr\rWr%r)rrQalphafrDrDrEr%Osz/EsmFoldStructureModule.torsion_angles_to_framescCs2||j|jt|||j|j|j|jSrF) rrr\rWr!rrrr)rrQrrDrDrEr!UszDEsmFoldStructureModule.frames_and_literature_positions_to_atom14_pos)NF) r>r?r@rrrr%r!rrDrDrrErs  q-rcs8eZdZfddZddZddZeddZZS) EsmFoldingTrunkcst|_j}j}t|_tfddt j D|_ d|_ t ||_t ||_t|j ||_|jjdtj|_t|jj|_t|jj|_j|_dS)Ncsg|] }tqSrD)r2rr rDrErUlsz,EsmFoldingTrunk.__init__..r)rrr rWrrpairwise_positional_embeddingrrrhrblocks recycle_binsr recycle_s_normrecycle_z_normr recycle_distordetachZzero_rstructure_modulerGr trunk2sm_sr trunk2sm_zr)rr rrrrrErcs      zEsmFoldingTrunk.__init__cCs ||_dSrF)r)rrrDrDrEset_chunk_sizezszEsmFoldingTrunk.set_chunk_sizec s|j}|}|} |dkr jj}n|dkr0td|d7}fdd} |} | } t| } t| }tj| jdd|tjd}t |D]}t ||dkrgnt g | |} | |}|| |7}| || | |||\} } | | d ||}| } | }t|d ddddddd fd d j}W5QRXq| |d<| |d<|S)a~ Inputs: seq_feats: B x L x C tensor of sequence features pair_feats: B x L x L x C tensor of pair features residx: B x L long tensor giving the position in the sequence mask: B x L boolean tensor indicating valid residues Output: predicted_structure: B x L x (num_atoms_per_residue * 3) tensor wrapped in a Coordinates object Nrz(Number of recycles must not be negative.rcs>|j||d}jD]}|||||jd\}}q||fS)Nrk)rr6r)rrr)rrresidxrblockrrDrE trunk_iters z+EsmFoldingTrunk.forward..trunk_iterrnro)rrr+r g @g`5@r-r.)rWr Z max_recyclesrrBZ zeros_likerr[Zint64rhrrrrrrrrrrr-r distogramr)rZ seq_featsZ pair_featsZtrue_aarr no_recyclesrWs_s_0s_z_0rr-r.Z recycle_sZ recycle_zrZ recycle_idx structurerDrrErsH        zEsmFoldingTrunk.forwardcCstj|||d|jd}|d}dd|jdddD\}}}||}||} |j| d d} d | d |d | |} | d dddddf| d dddddfdjd dd} tj| |kd d} | S)NrrVrcSsg|]}|dqS)r)rwrQrDrDrErUsz-EsmFoldingTrunk.distogram..r rrOrngƠvl¥gO[I-?g:4M?.T)rPZkeepdims)rBrqrWrLcrosspowr)ZcoordsZmin_binmax_binZnum_binsZ boundariesNCACrrrZCBdistsrtrDrDrErs@zEsmFoldingTrunk.distogram) r>r?r@rrr staticmethodrrrDrDrrErbs  Ara} ESMForProteinFolding is the HuggingFace port of the original ESMFold model. It consists of an ESM-2 "stem" followed by a protein folding "head", although unlike most other output heads, this "head" is similar in size and runtime to the rest of the model combined! It outputs a dictionary containing predicted structural information about the input protein(s). c seZdZddgZfddZeeeej dddZ e e deeed d!ej eej eej eej eeed d d ZddZej ej dddZddZed"eeeefdddZeeeedddZedddZeeeeddd ZZS)#EsmForProteinFoldingrr2c st|||_d|_t|dd|_|jd|jjjrF|j |jj |_ |jj |jj |_|jj |_|d||jtt|jd|_|jjj}|j}|j}tt|j t|j |tt|||_t j!d|_"d|_#|j"d|_$|j"d|_%|jj&d |_'|jj&d |_(|jj&d |_)|jj&d |_*|jjj+rftj,|j"|dd |_-t.||_t||j|_/t||j|_0t||j"|_1d|_2|j3}tt|j4t|j4|jjj5t|jjj5|jjj5t|jjj5d|j2|_6dS)N@F)Zadd_pooling_layer af2_to_esmrr rrzzz)Z padding_idxrn%)7rrr distogram_binsresmZrequires_grad_esmfold_configZfp16_esmZhalfZ hidden_size esm_featsZnum_hidden_layersZnum_attention_headsZ esm_attnsZ esm_layersr_af2_to_esm_from_vocab_list vocab_listrrrBr esm_s_combinetrunkrWrrar rGrb esm_s_mlpr#Z restype_numZn_tokens_embedZpad_idxZunk_idxmask_idxindexesm_dict_cls_idxesm_dict_mask_idxesm_dict_eos_idxesm_dict_padding_idxembed_aarrrdistogram_headptm_headlm_head lddt_binsrrZlddt_head_hid_dimr7)rr Z trunk_configrrZstructure_module_configrrDrErsV               zEsmForProteinFolding.__init__)rrNcs*dgfddtjD}t|S)Nrcsg|]}|qSrD)r)rRrrrDrErU!szDEsmForProteinFolding._af2_to_esm_from_vocab_list..)rr#Zrestypes_with_xrBrq)rZ esm_reorderrDrrErs z0EsmForProteinFolding._af2_to_esm_from_vocab_listzbatch_size, sequence_length) output_typeZ config_classN) input_idsattention_mask position_idsmasking_pattern num_recyclesrNcCsR|jj}|}|jd}|jd} |j} |dkr>> from transformers import AutoTokenizer, EsmForProteinFolding >>> model = EsmForProteinFolding.from_pretrained("facebook/esmfold_v1") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/esmfold_v1") >>> inputs = tokenizer(["MLKNVQVQLV"], return_tensors="pt", add_special_tokens=False) # A tiny random peptide >>> outputs = model(**inputs) >>> folded_positions = outputs.positions ``` rrNrVr)rcSsi|]\}}|dkr||qS))r.r-r'r(r)r*r+r,rDrRr_rrDrDrE ns z0EsmForProteinFolding.forward.. mlm_targetsr.r/r-r0r1)r2r5rnr6r,r7r{r8r9)rZno_binsr:)(r rr[rWrBZ ones_likearangeZ expand_asaf2_idx_to_esm_idx bert_mask&compute_language_model_representationsrrr\Zesm_ablate_sequencerrrrwrrrrrrryrrrr"r7rorr|rr rrrr&)rrr r r r cfgaaBLrWesmaaZ masked_aaresm_srrrZ disto_logitsr0r_r7r8r9rDrDrEr$sb      (zEsmForProteinFolding.forwardcCs<|jj|jkr|j|j|_|d|dkd}|j|SNrr)rrWrrM)rrrrDrDrErsz'EsmForProteinFolding.af2_idx_to_esm_idx)rrNc Cst|j}|j\}}|jjjrFtj|||j j dd|j |d}|S|j |j }}||df|}||df|j} tj||| gdd}||t||dkdf<|j||dkddd} tj| d d}|ddddf}|S) NrrnrVrrOT)r Zoutput_hidden_statesZ hidden_statesr)next parametersrWr[r rZ bypass_lmrBrrrrrrZnew_fullrrYrhrrr) rrrWrrrZbosiZeosiZbosZeosZesm_hidden_statesrDrDrErs   z;EsmForProteinFolding.compute_language_model_representationscCsJ|}|}|}|j||dk<d||dk<|j||dk<|||fSr)r(rr)rrrrpatternZnew_aatargetZ new_esmaarDrDrErs zEsmForProteinFolding.bert_mask)seqscst|tkr|g}n|}t|jtfdd|Dtfdd|D}|dkrztjjdd t |dn| }|j dkr| d}|j||dS) Nc s2g|]*}ttj|tjddjddqS)T)sequencemappingZmap_unknown_to_xrrO)rBZ from_numpyr#Zsequence_to_onehotZrestype_order_with_xrZargmaxrRseqrVrDrErUs z.EsmForProteinFolding.infer..csg|]}t|qSrD)rrar#)r1rDrErUsrrVrnr)r )rzrrrrWrlrBrr[r}rarrurr)rr r rYrrD)r1rWrEinfers*  "  zEsmForProteinFolding.infer)rrNc Csdd|D}g}t|dd|}|d}t|djdD]X}|d|}||}||}|d|d }t|||||d |d } |t| qB|S) zDReturns the pbd (file) string from the model given the model output.cSs i|]\}}||dqS)r)rnumpyr rDrDrErsz6EsmForProteinFolding.output_to_pdb..r+rnr5r1rr6rr8)r1Zatom_positionsrr6Z b_factors)ryrrhr[rrr$) rZpdbsZfinal_atom_positionsZfinal_atom_maskrjrZpred_posrZresidpredrDrDrE output_to_pdbs$  z"EsmForProteinFolding.output_to_pdb)rNcOs0t|tkst|j|f||}||dS)EReturns the pdb (file) string from the model given an input sequence.r)rzrrCr%r(rr argskwargsrrDrDrE infer_pdb szEsmForProteinFolding.infer_pdb)r rNcOs|j|f||}||S)r))r%r(r*rDrDrE infer_pdbs szEsmForProteinFolding.infer_pdbs)NNNN)N)r>r?r@Z_no_split_modulesrrrrrBrbrrESMFOLD_INPUTS_DOCSTRINGformatrr&rrrrrrrrr r%rr(r-r.rrDrDrrErs<  7  | 'r)r)rr)rn)rn)brr dataclassesr functoolsrtypingrrrrrr r r&rrBZtorch.nnrr Zintegrations.deepspeedr Zmodeling_outputsrrIrrrrrrZconfiguration_esmrZ modeling_esmrrrZopenfold_utilsrrrrrrr r!r"r#r$r%Z get_loggerr>rZ_CHECKPOINT_FOR_DOCZ_CONFIG_FOR_DOCr&r/rGrKrbr-rlrrprvr|rrrGrModulerrrrrrrr.rArPrUr]r`r2rmr|rrrrr0rrrrrrrDrDrDrEs  $      8 P!  .nK4>&h $AFKy