2Vh dZddlZddlZddlmZddlmZddl m Z ddl m Z ddl m Z ddZd Zd Zd Zd Zd ZdZdZdZdZdZy)z5Utilities for distribution strategy with JAX backend.N)layout) global_state)seed_generator) jax_utils) rng_utilsc|r|jnd}tj|}|Dcgc]}|jd|jc}Scc}w)aReturn all the available devices based on the device type. Note that this should return the global devices in a distributed setting. Args: device_type: string of `"cpu"`, `"gpu"` or `"tpu"`. Defaults to `"gpu"` or `"tpu"` if available when device_type is not provided. Otherwise will return the `"cpu"` devices. Return: List of devices that are available for distribute computation. Nbackend:)lowerjaxdevicesplatformid) device_type jax_devicesdevices V/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/backend/jax/distribution_lib.py list_devicesr sL*5+##%$K++k2K;F Gvq , GG Gs"Act||S)aCCreate a distributed variable for JAX. Since JAX doesn't have a variable class, this will just return a `jax.Array` with the corresponding layout/sharding specified. Note that this function should be used in eager context, not in jitted function. Args: value: the initial value of the variable. layout: `TensorLayout` for the created variable, or a JAX-supported layout instance (e.g. `jax.experimental.layout.Layout`, `jax.sharding.Sharding`). Returns: jax.Array which is the distributed variable. )distribute_tensor)valuers rdistribute_variablers$ UF ++cddlm}t||r |j}t j r t jj||St|t jrt|t jjr2|jj|t|jr|St|tj rt#|dd}||k(r|St j$||S)aDistribute the tensor based on the layout. Note that this function can be used both in eager context, or within a jitted function. Args: tensor: `jax.Array` that need to be distributed. layout: `TensorLayout` for the created variable, or a JAX-supported layout instance (e.g. `jax.experimental.layout.Layout`, `jax.sharding.Sharding`). Returns: Distributed value. r TensorLayout)ndimrN)keras.src.distributionr isinstancebackend_layoutris_in_jax_tracing_scoper laxwith_sharding_constraintArrayshardingShardingis_equivalent_tolenshape jax_layoutLayoutgetattr device_put)tensorrrcurrent_layouts rrr4s 4&,'&&((*ww//??&#))$  CLL)) oo..vC >&& ))rcjddlm}t||r |j}t j ||S)aDistribute the input data with the corresponding layout. Note that the inputs here is a local worker batch. Within the local worker, the data need to be further partitioned to map to each of the devices. Args: inputs: `jax.Array` that is already sharded to a local process size. layout: `TensorLayout` for the distribution information, or a `jax.sharding.Sharding` instance. Returns: A global batch distributed according to `layout`. rr)rrr r!r "make_array_from_process_local_data)per_process_batchrbatch_dim_namers rdistribute_data_inputr5\s04&,'&&  1 1&:K LLrctj}|tjd}tjd}tj j tjg|ztj j}tjdd||jd}tj|tjd}|W|jd }|Atj dtj"||j$|j& yyy) zInitializes the global random number generator across processes. This is required for consistent initialization in multi-host settings. Ncpu)dtypecBtjj|dS)Nall)r r#psum)xs rz initialize_rng..scggll1e,rr:) axis_namerrglobal_seed_generatorseed)r@namer )rget_random_seedr rlocal_device_countnumpyasarrayrmake_default_seeduint32pmapitemset_random_seedrget_global_attribute get_configset_global_attribute SeedGeneratorrAr ) global_seed cpu_devicesnum_local_cpu_devices local_seedr?r@s rinitialize_rngrSss/ ++-Kkk%( # 6 6u =YY&&  - - / 03H H))""'  chh ,   d1g  !!+.)==($//1&9 <  - -',,$.33199  )rc|rHd|vrD|jd}|+|t|k7rtd|dt|d||d}n|}tjj |||t y)N,zThe provided job_addresses z has z jobs, but num_processes is r)coordinator_address num_processes process_id)splitr) ValueErrorr distributed initializerS) job_addressesrWrXrVs rr\r\s -&++C0   $#m:L)L-m_E}%&&B /#  ,A.+OO/#rc*tjS)zDReturn the number of processes for the current distribution setting.)r process_countrrrWrW    rc*tjS)z;Return the current process ID for the distribution setting.)r process_indexr`rrrXrXrarct|tjr|S|jd\}}tj|}|D].}|j |k(s|j t|k(s,|cStd|)Nr r zDevice not found: ) r r DevicerYrrrintrZ) device_namer device_idrrs r_to_backend_deviceris}+szz*(..s3Kkk+.G ??k )fii3y>.IM )+7 88rc4|jj}|jjDcgc] }t|}}t j |j |}tjj||jScc}w)zConvert the DeviceMesh to JAX backend specific Mesh. Args: device_mesh: DeviceMesh instance to convert. Returns: A `jax.sharding.Mesh` instance. ) rr*flattenrinparrayreshaper r&Mesh axis_names) device_meshr*drs r_to_backend_meshrssy    % %E.9.A.A.I.I.KL!!$LGLhhw''.G <<  Wk&<&< ==MsBc|j tdtjj|j }|jj }tjj||S)zConvert the TensorLayout to JAX backend specific Sharding. Args: tensor_layout: TensorLayout instance to convert. Returns: A `jax.sharding.NamedSharding` instance. zDCannot create sharding when device mesh is not set for TensorLayout.)rqrZr r& PartitionSpecaxes backend_mesh NamedSharding) tensor_layoutpartition_specjax_meshs r_to_backend_layoutr|sf  (   \\//1C1CDN((55H << % %h ??r)N)__doc__r rDrljax.experimentalrr+keras.src.backend.commonrkeras.src.randomrkeras.src.utilsrrrrrr5rSr\rWrXrirsr|r`rrrsY; 11+%%H$,*%*PM.+\6  9 >@r