Vh \ddlZddlmZddlmZddlZddlmZejeZ dZ dZ Gdde Z d Zd eeej eej d ffd eeej eej d fffd Z ddedededeeeffdZeGddZy)N) dataclass)Union)fxcZgfd}tjj||}|fS)z\ Flatten the args into a list form and detach the tensors from computational graph. ct|tjr<|jj |j }j ||Sj ||SN) isinstancetorchTensordetachrequires_grad_ requires_gradappend)avalflat_detached_argss S/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/pipelining/_utils.pyextract_tensor_argsz0flatten_args_detach..extract_tensor_argssQ a &((*++AOO.extract_tensor_args,srr)rrrs @r flatten_argsr&s1I GG  rceZdZdZy)PipeliningShapeErrorz5Shape mismatch between configured and runtime values.N)__name__ __module__ __qualname____doc__rrr!r!9s?rr!c|j|jk(s't|d|jd|j|j|jk(s't|d|jd|j|j|jk(s/t|d|jd|jy)Nz has a shape mismatch: expected z actual z has a dtype mismatch: expected z! has a stride mismatch: expected )shaper!dtypestride)descexpectedgivens rvalidate_tensor_metadatar.=s >>U[[ ("f4X^^4DHU[[M Z   >>U[[ ("f4X^^4DHU[[M Z   ??   ."f5hoo6G5HQVQ]Q]Q_P` a   /rexpected_tensors.actual_tensorsc t|t|k7r&t|dt|dt|dtt|D]}t|d|||||y)Nz: Number of values (z") does not match expected number ()z: value )lenr!ranger.)r+r/r0is rvalidate_tensors_metadatar6Ls  N 33"f(^)<(==_`cdt`u_vvw x  3'( )  fHQC "21"5~a7H  rpp_size num_stagesstylereturnci}|dk(rt|D] }||z||< |S|dk(rU||zdk7rtd|d|dd}t|D])}|||<|dz|zdk(r||zdzdk(r|dz }%|dz}+|Std |d ) z Compute the stage id to rank mapping for either a looped or V-style schedule. Most commonly num_stages == pp_size * 2, but this function can be used to compute the mapping for any number of stages per rank. loopvrz num_stages z% must be evenly divisible by pp_size z for V scheduleszStyle z is not supported.)r4 ValueError)r7r8r9mapping stage_index rank_indexs rgenerate_stage_to_rank_mappingrD[sG  , 9K#.#8GK  9( N% #  1 $j\)NwiWgh   , K#-GK a7*a/w&!+q0a a   N6%(:;< Captures information for a pipeline (`Pipe` object). graphr8has_loss_and_backwardN) r"r#r$r%rGraph__annotations__intboolr&rrrFrF}s 88OOrrF)r<)logging dataclassesrtypingrr r getLoggerr"loggerrr RuntimeErrorr!r.listr tupler6rKstrdictrDrFr&rrrWs!    8 $(0&@<@    D.ellC6G0HHI  $u||,eELL#4E.FFG  17 !*- #s(^D     r