Vh nddlZddlZddlmZmZddlmZddlZddlm Z m Z ddl m Z m Z mZgdZde dee fd Zde d eedee fd Zd e defd Ze j.fd e dedede fdZd e defdZ dd e deede fdZeded e dede fdZd e defdZd e dej<de fdZy)N)AnyOptional) deprecated)_ShardingIterDataPipeSHARDING_PRIORITIES)DataPipe DataPipeGraph traverse_dps)apply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipesgraphreturnc*t|tSN)_get_all_graph_pipes_helperset)rs O/home/dcms/DCMS/lib/python3.12/site-packages/torch/utils/data/graph_settings.pyrrs &uce 44id_cachecg}|jD]J\}\}}||vr|j||j||jt ||L|Sr)itemsaddappendextendr)rrresultsdp_iddatapipe sub_graphs rrrsh!G(- I$$) H   Ux 29hGH I Nrr ct|txs-t|dxrtj|j S)Nr ) isinstancerhasattrinspectismethodr r s r_is_sharding_datapiper()s; h 5 6 *+ 6   X44 5rnum_of_instances instance_idcFt|}dfd ||S)z Apply dynamic sharding over the ``sharding_filter`` DataPipe that has a method ``apply_sharding``. RuntimeError will be raised when multiple ``sharding_filter`` are presented in the same branch. cP|jD]\}}d}t|rs|td|d|tj|j }t |jdkr|j n|j  |}||}||y)NzoSharding twice on a single pipeline is likely unintended and will cause data loss. Sharding already applied to z while trying to apply to )sharding_group)valuesr( RuntimeErrorr% signaturer len parameters) r prev_applieddpr!appliedsig_helperr*r)r.s rr8zapply_sharding.._helper=s"\\^ (MB G$R(+&77CnD^_a^bd ''(9(9:s~~&*%%&6 D%%(+n&& Iw '' (rr)r )r r)r*r.rr8s ``` @rr r 0s'  "E((, EN Orct|dxrNt|dxr@tj|jxrtj|jS)N set_shuffleset_seed)r$r%r&r:r;r's r_is_shuffle_datapiper<XsT-( 0 Hj ) 0   X11 2 0   X.. / rshufflec||St|}t|}|Dcgc]}t|s|}}|s*|r(tjd|j }|g}|D]}|j ||Scc}w)aE Traverse the graph of ``DataPipes`` to find and set shuffle attribute. Apply the method to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) z`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.)r rr<warningswarnr=r:)r r=r all_pipespipe shufflersshufflers rrras  "E#E*I"+J$/CD/IJIJ   [ ##%  &W%& OKs A<A<z`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)categoryrngct||Sr)r )r rFs rr r s Xs ++rc\t|dxrtj|jS)Nr;)r$r%r&r;r's r_is_random_datapiperIs% 8Z ( PW-=-=h>O>O-PPrct|}t|}t}g}|D]G}t||vrt |s|j ||j t|I|D]_}ttjdtjj|j}|j|a|S)a6 Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed``. Then set the random seed based on the provided RNG to those ``DataPipe``. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds )dtype) generator)r rridrIrrinttorchemptyint64random_itemr;)r rFrrAcacherandom_datapipesrB random_seeds rr r s  "E#E*I EE  d8u   t $  # #D ) IIbh   !# KK%++ . 6 6 6 E J J L  k" # Orr) r%r?typingrrtyping_extensionsrrP(torch.utils.data.datapipes.iter.shardingrrtorch.utils.data.graphrr r __all__listrrrOrboolr(DEFAULTr r<r FutureWarningr rI Generatorr rKrrrbsQ ( IH 5}5h5   $'H  (^ H'.. %%%%  %P837  !)$   F .  ,,,,  ,Q(QtQu8r