VhRCUddlmZddlZddlZddlZddlZddlZddlZddlm Z m Z m Z ddl m Z ddlmZddlmZmZddlmZmZmZmZddlZddlmZdd lmZmZmZmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,m-Z-dd l.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m6Z6m7Z7ddl8m9Z9ddl:m;Z;er ddlm?Z?da@daAdeBd<ejjeEdZFejeEZHdZIdZJd*dZKd*dZLejdk(ZNejeEZHe9e,ZOd*dZPdZQ ejeQ d+d!ZTe4Gd"d#ZUGd$d%ZVejjd&d'd(k(s7ejjd)d(d(k7se;rejrneVjejePy#eS$rYwxYw),) annotationsN)FutureProcessPoolExecutorThreadPoolExecutor)BrokenProcessPool)partial)timetime_ns)AnyCallableOptional TYPE_CHECKING) get_registered_device_interfaces)counters dynamo_timedget_metrics_contextset_feature_use)config) _load_triton_kernel_from_source code_hashCodeCacheFuture CppCodeCacheCppPythonBindingsCodeCache CUDACodeCacheHalideCodeCache LambdaFuture ROCmCodeCache torch_key)AnyPool SubprocPool)_async_compile_initializer)_set_triton_ptxas_path_worker_compile_triton)clear_on_fresh_inductor_cache) _Faketqdmtqdm) OrderedSet)has_triton_package) HalideMeta)CachingAutotunergzOptional[float]_t0 kernel_codecRt ddlm}|y#t$rYywxYw)zG Setup that must be done prior to forking with a process pool. r triton_keyN)caching_device_propertiestriton.compiler.compilerr/ ImportErrorr.s M/home/dcms/DCMS/lib/python3.12/site-packages/torch/_inductor/async_compile.pypre_fork_setupr4>s)  7    s  &&c~tD]0\}}|js|jj2yN)r is_availableWorkerget_device_properties)_device_interfaces r3r0r0Qs8?A<   ( ( *  # # 9 9 ;<c&t tayyr6)r+r r<r3_compile_startr?Ws {fr<cJtt}t|tz z adayyr6)r+r _cumulative_compile_time)t1s r3 _compile_endrC]s'  V BH, r<win32cNtD]}|jty)z/Shut down all outstanding compile-worker pools.N) _pool_setshutdown after_fork)pools r3shutdown_compile_workersrJns Lr<chtjtjj y)z7Reset pools to initial state without shutting them downN)rFclear AsyncCompile process_pool cache_clearr>r<r3rHrHus OO))+r<)after_in_childc|tjtjt_tjS)z} Temporary for internal rollout. Assign config.compile_threads lazily and return it. TODO: remove after rollout. )rcompile_threadsdecide_compile_threadsr>r<r3get_compile_threadsrTs- %!'!>!>!@  ! !!r<cxeZdZUdZiZded<ed dZed dZed dZ edZ ed dZ y )CompiledTritonKernelsa/ In memory cache for storing compiled triton kernels. Each triton kernel is keyed by the hash of its source code. Each value stored in the cache is a return value of AsyncCompile.triton(). Currently, the cache stores Future objects, but it should be generalizable for any kernels. zdict[str, LambdaFuture]_cachec,t|tS)a Generates a cache key given a triton kernel's full source code. This source includes the inductor meta, compilation metadata, the kernel itself, etc. `kernel_src` should be the exact string passed to async_compile.triton()'s first argument. )extra)rr) kernel_srcs r3keyzCompiledTritonKernels.keys9;77r<cTtj|}|tj|<y)a Saves a compiled triton kernel to the cache. TODO: We store a LambdaFuture as that's the callable returned by async_compile.triton, but the real type we want to return here is actually an abstract triton kernel. TODO: Source code here is not just the kernel's source code, but also includes the inductor preamble, etc. so it could be less strict. NrVr[rW)rZfuturer[s r3savezCompiledTritonKernels.saves%$'' 3,2$$S)r<cltj|}tjj||Sr6)rVr[rWget)rZdefaultr[s r3razCompiledTritonKernels.gets+#'' 3$++//W==r<cit_yr6)rVrWr>r<r3rOz!CompiledTritonKernels.cache_clears ')$r<cvtj|}|tjvrtj|=yyr6r])rZr[s r3 remove_futurez#CompiledTritonKernels.remove_futures4#'' 3 '.. .%,,S1 /r<N)rZstr)rZrfr^r)rZrfrbr returnr)rZrfrgNone) __name__ __module__ __qualname____doc__rW__annotations__ staticmethodr[r_rarOrer>r<r3rVrVsv')F #(88 3 3>>**22r<rVceZdZddZeej dddZedZeej dddZ e ddZ e ddZ dZ ddd Zdd Zdd Zdd Zdd Z ddZddZddZddZy)rMcyr6r>selfs r3__init__zAsyncCompile.__init__s r<cFtdkDsJttSNrt)rTrr>r<r3rIzAsyncCompile.pools$#$q(((!"5"788r<cy)z>No-op function to help mark when the subprocess pool is ready.readyr>r>r<r3 _get_readyzAsyncCompile._get_readysr<c tdkDsJtjdtjttjdk(rt t}ntjdk(rdt jd<ttjtj}tt|ttt j}tjj!d|j"t$j&|j)t*j,|_t0j3||S) Nrtz"Creating '%s' pool with %d workers subprocessspawn0TORCH_WARM_POOL) mp_context initializer) exitpriority)rTloginforworker_start_methodr osenvironr4multiprocessing get_contextrrr!getpidutilFinalizerGsysmaxsizesubmitrMry ready_futurerFadd)rIctxs r3rNzAsyncCompile.process_pools#$q(((  0  & &  !   % % 5245D))W403 ,-  !--f.H.HIC&#%#$> LD  ) )$ CKK ) X!KK (?(?@ d r<chtdkryt|jtyrv)rTr?rNrC)clss r3 warm_poolzAsyncCompile.warm_pools'  A %  r<chtdkr|S|jj|Srv)rTrIr)rtasks r3rzAsyncCompile.submits+  A %6Mxxz  &&r<cptdkDxr(|jjjSrv)rTrNrdonerqs r3use_process_poolzAsyncCompile.use_process_pools/  !A % Q$*;*;*=*J*J*O*O*Q r<c* tjdx}rtddxxdz cc<|Stddxxdz cc<tj dt t jjddd k(r=ttjjjjStjt  |j#}t%d ||rd d g}|Dcic])}|t jvs|t j|+}}|j'j)t* | fd dfd } t-| }tj/||St1dddd5t3} t5 } | j7dt3| z dz} t9j;d| | cdddScc}w#1swYyxYw)a Async_compile.triton is more complicated than the other backends because we're trying to optimize compile time as much as possible for this hot callsite. First of all, the function is cached by CompiledTritonKernels; if there's a kernel already compiled, we grab it directly from the cache and return. Otherwise, if we have multiple compile threads, we kick off triton compilations on each worker process by giving it a kernel and source code to compile. The worker initializes a CachingAutotuner, runs triton compilation, and pickles the kernel back to us. We use TritonCompileResult to represent the objects being pickled back to us by each worker. Some maybe not obvious things that are pickled back to us: - Most of the time, we can avoid sending back CachingAutotuner.fn and other metadata and do not have to pay the cost of loading the triton kernel on the parent. But certain cases, like coordesc tuning and dynamic_scale_rblock, require us to reload the function in the parent lazily when we require it. - The AutotuneCache, if enabled, is constructed on each worker per triton config and pickled by to us via `CachingAutotuner.save_cache_hook`. Ninductorasync_compile_cache_hitrtasync_compile_cache_misszTriton Kernel: %sTRITON_INTERPRETr}1parallel_compile_post_warmupTORCHINDUCTOR_CACHE_DIRTRITON_CACHE_DIRcTtd5cdddS#1swYyxYw)Nreload_kernel_in_parent)r) load_kernelsr3rz4AsyncCompile.triton..reload_kernel_in_parent;s&!";<)&=)))s'cj\}}tj|jdt j d||S)NF)warm_cache_only reload_kerneltriton_kernel_compile_times_us)resultrVre precompiler add_top_n)kernel elapsed_us kernel_namer source_coders r3 get_resultz'AsyncCompile.triton..get_result@s]%)[[]" &33K@!!$)9P"$%//4k: r<)r^zasync_compile.precompileTtriton_compile_time_uslog_pt2_compile_eventdynamo_compile_column_uslog_waitcounterF)rir)rgztuple[CachingAutotuner, int])rVrarkernel_code_logrr?rrgetattrtorch _inductor codecache PyCodeCacheload functoolsrrrrrNrr#rr_rr r"rrr)rrrr device_strr^ is_parallelenv_varsv extra_envrstart_nsrrrrrs `` @@@r3tritonzAsyncCompile.tritons,+..{DA A6 A Z !: ;q @ ;M78A=81;? ::>>,c 2c 9))55::;G  '' +[+ ++- 6 D 23EFH3;OaqBJJBJJqM)OIO$$&--&D  )  "*T:F ! & &{F ;M*&*)A $   #9&($!!%!8%i(2t; #%//4k:  =P<  s H!H AH  Hcddlm}||i|S)Nr)MultiKernelCall)$torch._inductor.codegen.multi_kernelr)rrargskwargsrs r3 multi_kernelzAsyncCompile.multi_kernelasH///r<ctjd|tdkrtj|j Stj ||jtfdS)NzCPP Kernel: %srt submit_fnc&jSr6)r)rsr3z"AsyncCompile.cpp..ms (;(;r<) rrrTrrr load_asyncrr)rrrrs @r3cppzAsyncCompile.cppgsX. <  A %$$[188 8%00 TJ ;< .task|s0%%k37 %%k<@C Cr<rrrrrrrrrs ``` r3cudazAsyncCompile.cudays+/= D{{4  r<cdtjdfd}|j|S)NzROCm Kernel: %scrtjd}tjjrtjd}tj dS)Nr)rexer)rrrrocmgenerate_test_runnerr)r:rrrs r3rzAsyncCompile.rocm..tasksO!))+CH{{//!))+EJ %%k<@C Cr<rrs ``` r3rzAsyncCompile.rocms- /= D{{4  r<ctjd||tdkrtj||Stj |||j }t|S)NzHalide Kernel: %r %srtr)rrrTrgenerate_halidegenerate_halide_asyncrr)rrmetarrs r3halidezAsyncCompile.halides[5t[I  A %"224E E(>>kT[[J + +r<ctdkDr4tdddd5|j|dddtyty#1swYtyxYw)Nrtzasync_compile.waitTrr)rTr _wait_futuresrC)rrscopes r3waitzAsyncCompile.waitsS  1 $$&*)A $  * ""5)  *   * s A  A c |jDcic]\}}t|ttfr|| }}}t t |dt jd}|jD][\}}t jr!t|ts|j| |j||<|jd]ycc}}w#t$r}td|d}~wwxYw)NzInductor Compilationr)totaldescdisabledelayzA compilation subprocess exited unexpectedly. This is likely due to a crash. To facilitate debugging, you can re-run with TORCHINDUCTOR_COMPILE_THREADS=1 to cause compilation to occur in the main process.rt)items isinstancerrr&lenrdisable_progressverbose_progressr%set_postfix_strrr RuntimeErrorupdate)rrrr[valuekernelspbarres r3rzAsyncCompile._wait_futuress$kkm U%&/!:; J  g,'++  #==? KC&&z$ /J$$S) #]]_c  KKN  "% "I   s#C(C C/ C**C/Nrgrh)rgr)rgr)rzCallable[..., Any]rgr )r)rrfrrfrrf)rgr )rrf)rz list[str]rrf)F)rr)rrf)rzdict[str, Any]rgrh)rirjrkrsrnr lru_cacherIryrN classmethodrrrrrrrrrrrrr>r<r3rMrMs Y99Y  D''  Yv0 =, !  !", r<rMTORCH_TNT_IN_USEr}rr~r)rgint)\ __future__ratexitrloggingrrrconcurrent.futuresrrrconcurrent.futures.processrrr r typingr r r rrtorch._dynamo.device_interfacertorch._dynamo.utilsrrrrtorch._inductorrtorch._inductor.codecacherrrrrrrrrr+torch._inductor.compile_worker.subproc_poolrr 'torch._inductor.compile_worker.watchdogr!%torch._inductor.runtime.compile_tasksr"r#torch._inductor.utilsr$ torch.hubr%r&torch.utils._ordered_setr'torch.utils._tritonr(torch._inductor.runtime.hintsr))torch._inductor.runtime.triton_heuristicsr*rAr+rm_logginggetArtifactLoggerrir getLoggerrr4r0r?rCplatform _IS_WINDOWSrFrJrHregister_at_forkAttributeErrorrTrVrMrra is_fbcoderregisterr>r<r3r!s"  NN899 K #   MN@%/28J_..228]Kg! &<  llg% g! Jw  ! ,  Bz2 "020202fFFTJJNN%s+s2 zz~~'-4  v(){   s4GG&%G&