U Ph@sddlmZddlZddlZddlZddlmZddlmZddl m Z m Z ddl Z ddl mZddlmZddlmZdd lmZdd lmZmZdd lmZdd lmZdd lmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ee(dZ)e$d\Z*Z+GdddZ,dS)) annotationsN)deepcopy)sleep)Anycast) BundleGen) DataAnalyzer)EnsembleRunner)NNIGen)export_bundle_algo_historyimport_bundle_algo_history) get_logger)algo_to_pickle) ConfigParser) SaveImage)AlgoKeys has_optionlook_up_optionoptional_import) check_kwargs_exist_in_class_initrun_cmd) module_namennic@sHeZdZUdZded<dEd d d d d d d d d d dd ddddddZddZddZd ddddZdFd dddddZ dGddd d!d"Z dHddd#d$d%Z dId&d'dddd(d)d*Z dJd ddd,d-d.Z ddd/d0d1ZdKddd#d2d3ZdLddd#d4d5ZdMddd#d6d7Zd8dd9d:d;Zdd?d@ZddAdBZdCdDZdS)N AutoRunneru  An interface for handling Auto3Dseg with minimal inputs and understanding of the internal states in Auto3Dseg. The users can run the Auto3Dseg with default settings in one line of code. They can also customize the advanced features Auto3Dseg in a few additional lines. Examples of customization include - change cross-validation folds - change training/prediction parameters - change ensemble methods - automatic hyperparameter optimization. The output of the interface is a directory that contains - data statistics analysis report - algorithm definition files (scripts, configs, pickle objects) and training results (checkpoints, accuracies) - the predictions on the testing datasets from the final algorithm ensemble - a copy of the input arguments in form of YAML - cached intermediate results Args: work_dir: working directory to save the intermediate and final results. input: the configuration dictionary or the file path to the configuration in form of YAML. The configuration should contain datalist, dataroot, modality, multigpu, and class_names info. algos: optionally specify algorithms to use. If a dictionary, must be in the form {"algname": dict(_target_="algname.scripts.algo.AlgnameAlgo", template_path="algname"), ...} If a list or a string, defines a subset of names of the algorithms to use, e.g. 'segresnet' or ['segresnet', 'dints'] out of the full set of algorithm templates provided by templates_path_or_url. Defaults to None, to use all available algorithms. analyze: on/off switch to run DataAnalyzer and generate a datastats report. Defaults to None, to automatically decide based on cache, and run data analysis only if we have not completed this step yet. algo_gen: on/off switch to run AlgoGen and generate templated BundleAlgos. Defaults to None, to automatically decide based on cache, and run algorithm folders generation only if we have not completed this step yet. train: on/off switch to run training and generate algorithm checkpoints. Defaults to None, to automatically decide based on cache, and run training only if we have not completed this step yet. hpo: use hyperparameter optimization (HPO) in the training phase. Users can provide a list of hyper-parameter and a search will be performed to investigate the algorithm performances. hpo_backend: a string that indicates the backend of the HPO. Currently, only NNI Grid-search mode is supported ensemble: on/off switch to run model ensemble and use the ensemble to predict outputs in testing datasets. not_use_cache: if the value is True, it will ignore all cached results in data analysis, algorithm generation, or training, and start the pipeline from scratch. templates_path_or_url: the folder with the algorithm templates or a url. If None provided, the default template zip url will be downloaded and extracted into the work_dir. allow_skip: a switch passed to BundleGen process which determines if some Algo in the default templates can be skipped based on the analysis on the dataset from Auto3DSeg DataAnalyzer. mlflow_tracking_uri: a tracking URI for MLflow server which could be local directory or address of the remote tracking Server; MLflow runs will be recorded locally in algorithms' model folder if the value is None. mlflow_experiment_name: the name of the experiment in MLflow server. kwargs: image writing parameters for the ensemble inference. The kwargs format follows the SaveImage transform. For more information, check https://docs.monai.io/en/stable/transforms.html#saveimage. Examples: - User can use the one-liner to start the Auto3Dseg workflow .. code-block:: bash python -m monai.apps.auto3dseg AutoRunner run --input '{"modality": "ct", "datalist": "dl.json", "dataroot": "/dr", "multigpu": true, "class_names": ["A", "B"]}' - User can also save the input dictionary as a input YAML file and use the following one-liner .. code-block:: bash python -m monai.apps.auto3dseg AutoRunner run --input=./input.yaml - User can specify work_dir and data source config input and run AutoRunner: .. code-block:: python work_dir = "./work_dir" input = "path/to/input_yaml" runner = AutoRunner(work_dir=work_dir, input=input) runner.run() - User can specify a subset of algorithms to use and run AutoRunner: .. code-block:: python work_dir = "./work_dir" input = "path/to/input_yaml" algos = ["segresnet", "dints"] runner = AutoRunner(work_dir=work_dir, input=input, algos=algos) runner.run() - User can specify a local folder with algorithms templates and run AutoRunner: .. code-block:: python work_dir = "./work_dir" input = "path/to/input_yaml" algos = "segresnet" templates_path_or_url = "./local_path_to/algorithm_templates" runner = AutoRunner(work_dir=work_dir, input=input, algos=algos, templates_path_or_url=templates_path_or_url) runner.run() - User can specify training parameters by: .. code-block:: python input = "path/to/input_yaml" runner = AutoRunner(input=input) train_param = { "num_epochs_per_validation": 1, "num_images_per_batch": 2, "num_epochs": 2, } runner.set_training_params(params=train_param) # 2 epochs runner.run() - User can specify the fold number of cross validation .. code-block:: python input = "path/to/input_yaml" runner = AutoRunner(input=input) runner.set_num_fold(n_fold = 2) runner.run() - User can specify the prediction parameters during algo ensemble inference: .. code-block:: python input = "path/to/input_yaml" pred_params = { 'files_slices': slice(0,2), 'mode': "vote", 'sigmoid': True, } runner = AutoRunner(input=input) runner.set_prediction_params(params=pred_params) runner.run() - User can define a grid search space and use the HPO during training. .. code-block:: python input = "path/to/input_yaml" runner = AutoRunner(input=input, hpo=True) runner.set_nni_search_space({"learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}) runner.run() Notes: Expected results in the work_dir as below:: work_dir/ ├── algorithm_templates # bundle algo templates (scripts/configs) ├── cache.yaml # Autorunner will automatically cache results to save time ├── datastats.yaml # datastats of the dataset ├── dints_0 # network scripts/configs/checkpoints and pickle object of the algo ├── ensemble_output # the prediction of testing datasets from the ensemble of the algos ├── input.yaml # copy of the input data source configs ├── segresnet_0 # network scripts/configs/checkpoints and pickle object of the algo ├── segresnet2d_0 # network scripts/configs/checkpoints and pickle object of the algo └── swinunetr_0 # network scripts/configs/checkpoints and pickle object of the algo z dict | Noneanalyze_params ./work_dirNFrTstrzdict[str, Any] | str | Nonezdict | list | str | Nonez bool | Noneboolz str | Noner)work_dirinputalgosanalyzealgo_gentrainhpo hpo_backendensemble not_use_cachetemplates_path_or_url allow_skipmlflow_tracking_urimlflow_experiment_namekwargscKs|dkrLtjtjtj|drLtjtj|d}td|t|_t |trf||_nBt |t rtj|rt ||_td|nt |dd|jkr|jd}tj||_td|jtj|jddtj|jd|_||_| |_| |_| |_tj|jd |_||_||dkrP|jd  n||_|dkrl|jd  n||_||_| |_|ot|_||_| |_ ||_!t"||_#d D]6}||jkrt |j|t$rt%|||j|qd D]$}||jkrt%|||j|qdddh&|j'}t(|dkr>t d|tj)|jdsft d|jdtj|jtj*|jd}||jdkrz&t+,|jd|td|Wnt+j-k rYnX|j.|d|_/d|jkrt0|jd}td|dn|j/}td|d|d||jd<t j1|j|jddd|jd|_2tj|jd|_3||_4|5|6|7|8|9|j:|dd|_;i|_<|j=d krt>d!|jot|_|?i|_@d|_Ad"|j#krd"|jkr|jd"|j#d"<dS)#Nz input.yamlz0Input config is not provided, using the default zLoading input config z is not a valid file or dictrz AutoRunner using work directory T)exist_okz cache.yamlr!r")r!r"r#r$r&r'r))r r%r(r*r+datarootdatalistmodalityrzConfig keys are missing zDatalist file is not found z!Datalist was copied to work_dir: )datalist_filenamenum_foldzSetting num_fold z based on the input config. based on the input datalist .yamlF)configfilepathfmt sort_keyszdatastats.yaml)r2rz HPOGen backend only supports NNIsigmoid)BospathisfilejoinabspathloggerinfodictZ data_src_cfg isinstancerrload_config_file ValueErrorrmakedirsdata_src_cfg_namer r(r)r'cache_filename read_cachecache export_cacher!r"r#r&has_nnir$r%r*r+rr,rsetattr differencekeyslenexistsbasenameshutilcopyfile SameFileErrorinspect_datalist_foldsmax_foldintexport_config_filer.datastats_filenamer1set_training_paramsset_device_infoset_prediction_paramsset_analyze_paramsset_ensemble_method set_num_foldgpu_customizationgpu_customization_specslowerNotImplementedErrorset_hpo_params search_space hpo_tasks)selfrrr r!r"r#r$r%r&r'r(r)r*r+r,param missing_keysr1r2rkU/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/apps/auto3dseg/auto_runner.py__init__s&             zAutoRunner.__init__cCsddddd}|js"tj|js&|St|j}|D]\}}|||q:|drt |dt rvtj|dsd|d<d|d<|drt |j dd}t |dkrd|d<|d rt |j d d}t |dkrd|d <|S) af Check if the intermediate result is cached after each step in the current working directory Returns: a dict of cache results. If not_use_cache is set to True, or there is no cache file in the working directory, the result will be ``empty_cache`` in which all ``has_cache`` keys are set to False. FN)r! datastatsr"r#r!rnr"Z only_trainedrr#T)r'r;r<r=rHrrDitems setdefaultrCrr rrP)rh empty_cacherJkvhistoryZtrained_historyrkrkrlrISs&    zAutoRunner.read_cachecKs(|j|tj|j|jdddddS)zQ Save the cache state as ``cache.yaml`` in the working directory r5NF)r8default_flow_styler9)rJupdaterrYrH)rhr,rkrkrlrKxs zAutoRunner.export_cacherX)r1returnc Cst|}d|kr"tdt|dd|dD}t|dkrt|d}td|d|d tt||krtd |dnbd |krft|d dkrftd |dD] }d|d <q|d D] }d|d <qdd|d D}tdt|d||dD]&}|d|krd|d <||d=q|dt | |d<tj ||dddd}nd}t d|d|dddlm}||ddd}t||dD]*\} \} } | D]} | |d| d <qqtj ||ddd|S)a Returns number of folds in the datalist file, and assigns fold numbers if not provided. Args: datalist_filename: path to the datalist file. Notes: If the fold key is not provided, it auto generates 5 folds assignments in the training key list. If validation key list is available, then it assumes a single fold validation. trainingz#Datalist files has no training key:cSs g|]}d|krt|dqS)fold)rX.0drkrkrl sz5AutoRunner.inspect_datalist_folds..rzFound num_fold r3r4z*Fold numbers are not continuous from 0 to validationzUNo fold numbers provided, attempting to use a single fold based on the validation keyrzcSsi|]}d|kr|d|qS)labelrkr{rkrkrl sz5AutoRunner.inspect_datalist_folds..zFound z8 items in the validation key, saving updated datalist torjson)r8indentz Datalist has no folds specified z...Generating z[ folds randomly.Please consider presaving fold numbers beforehand for repeated experiments.)KFoldT)Zn_splitsshuffle random_state)rrDrErrPmaxr@rAsetlistvaluesrYwarningswarnZsklearn.model_selectionr enumeratesplit) rhr1r/Z fold_listr2r}Z val_labelsrkfi_Z valid_idxvirkrkrlrVsL          z!AutoRunner.inspect_datalist_foldszdict[str, Any] | None)rarbrxcCs||_|dk r||_|S)a Set options for GPU-based parameter customization/optimization. Args: gpu_customization: the switch to determine automatically customize/optimize bundle script/config parameters for each bundleAlgo based on gpus. Custom parameters are obtained through dummy training to simulate the actual model training process and hyperparameter optimization (HPO) experiments. gpu_customization_specs (optional): the dictionary to enable users overwrite the HPO settings. user can overwrite part of variables as follows or all of them. The structure is as follows. .. code-block:: python gpu_customization_specs = { 'ALGO': { 'num_trials': 6, 'range_num_images_per_batch': [1, 20], 'range_num_sw_batch_size': [1, 20] } } ALGO: the name of algorithm. It could be one of algorithm names (e.g., 'dints') or 'universal' which would apply changes to all algorithms. Possible options are - {``"universal"``, ``"dints"``, ``"segresnet"``, ``"segresnet2d"``, ``"swinunetr"``}. num_trials: the number of HPO trials/experiments to run. range_num_images_per_batch: the range of number of images per mini-batch. range_num_sw_batch_size: the range of batch size in sliding-window inferer. N)rarb)rhrarbrkrkrlset_gpu_customizations!z AutoRunner.set_gpu_customizationr)r2rxcCsD|dkrtd|||jkr:td|jd|jd||_|S)z Set the number of cross validation folds for all algos. Args: num_fold: a positive integer to define the number of folds. rzEnum_fold is expected to be an integer greater than zero. Now it gets z1num_fold is greater than the maximum fold number z in r4)rErWr1r2)rhr2rkrkrlr`s zAutoRunner.set_num_fold)paramsrxcCs0|dk rt|ni|_d|jkr,tdt|S)a Set the training params for all algos. Args: params: a dict that defines the overriding key-value pairs during training. The overriding method is defined by the algo class. Examples: For BundleAlgo objects, the training parameter to shorten the training time to a few epochs can be {"num_epochs": 2, "num_epochs_per_validation": 1} NCUDA_VISIBLE_DEVICESz]CUDA_VISIBLE_DEVICES is deprecated from 'set_training_params'. Use 'set_device_info' instead.)r train_paramsrrDeprecationWarningrhrrkrkrlr[s  zAutoRunner.set_training_paramszlist[int] | str | Nonez int | None)cuda_visible_devices num_nodesmn_start_method cmd_prefixrxcCsTi|_|dkrtjd}|dkrXdddttjD|jd<tj|jd<ntt |t r||jd<t | d|jd<nJt |t tfrddd|D|jd<t ||jd<ntd|d |dkrttjd d }||jd <|dkrtjd d }||jd <|dkr*tjdd}||jd<|dk rPtd|d|S)a Set the device related info Args: cuda_visible_devices: define GPU ids for data analyzer, training, and ensembling. List of GPU ids [0,1,2,3] or a string "0,1,2,3". Default using env "CUDA_VISIBLE_DEVICES" or all devices available. num_nodes: number of nodes for training and ensembling. Default using env "NUM_NODES" or 1 if "NUM_NODES" is unset. mn_start_method: multi-node start method. Autorunner will use the method to start multi-node processes. Default using env "MN_START_METHOD" or 'bcprun' if "MN_START_METHOD" is unset. cmd_prefix: command line prefix for subprocess running in BundleAlgo and EnsembleRunner. Default using env "CMD_PREFIX" or None, examples are: - single GPU/CPU or multinode bcprun: "python " or "/opt/conda/bin/python3.8 ", - single node multi-GPU running "torchrun --nnodes=1 --nproc_per_node=2 " If user define this prefix, please make sure --nproc_per_node matches cuda_visible_device or os.env['CUDA_VISIBLE_DEVICES']. Also always set --nnodes=1. Set num_nodes for multi-node. Nr,cSsg|] }t|qSrkrr|xrkrkrlr~4sz.AutoRunner.set_device_info.. n_devicescSsg|] }t|qSrkrrrkrkrlr~:sz%Wrong format of cuda_visible_devices z, devices not setZ NUM_NODESrZMN_START_METHODZbcprunZ CMD_PREFIXz*Using user defined command running prefix z, will override other settings)device_settingr;environgetr>rangetorchcuda device_countrCrrPrrtupler@rrXrA)rhrrrrrkrkrlr\s2 $        zAutoRunner.set_device_infoAlgoEnsembleBestByFold)ensemble_method_namer,rxcKs"t|ddgd|_|j||S)a Set the bundle ensemble method name and parameters for save image transform parameters. Args: ensemble_method_name: the name of the ensemble method. Only two methods are supported "AlgoEnsembleBestN" and "AlgoEnsembleBestByFold". kwargs: the keyword arguments used to define the ensemble method. Currently only ``n_best`` for ``AlgoEnsembleBestN`` is supported. AlgoEnsembleBestNr) supported)rrr,rw)rhrr,rkrkrlr_Ps  zAutoRunner.set_ensemble_method)r,rxcKs2tt|\}}|r |j|nt|d|S)a# Set the ensemble output transform. Args: kwargs: image writing parameters for the ensemble inference. The kwargs format follows SaveImage transform. For more information, check https://docs.monai.io/en/stable/transforms.html#saveimage. z are not supported in monai.transforms.SaveImage,Check https://docs.monai.io/en/stable/transforms.html#saveimage for more information.)rrr,rwrE)rhr,Zare_all_args_present extra_argsrkrkrlset_image_save_transformas z#AutoRunner.set_image_save_transformcCs|dk rt|ni|_|S)a Set the prediction params for all algos. Args: params: a dict that defines the overriding key-value pairs during prediction. The overriding method is defined by the algo class. Examples: For BundleAlgo objects, this set of param will specify the algo ensemble to only inference the first two files in the testing datalist {"file_slices": slice(0, 2)} N)r pred_paramsrrkrkrlr]vsz AutoRunner.set_prediction_paramscCs$|dkrddd|_n t||_|S)z Set the data analysis extra params. Args: params: a dict that defines the overriding key-value pairs during training. The overriding method is defined by the algo class. NFr)Zdo_ccpdevice)rrrrkrkrlr^s  zAutoRunner.set_analyze_paramscCs|dkr|jn||_|S)a Set parameters for the HPO module and the algos before the training. It will attempt to (1) override bundle templates with the key-value pairs in ``params`` (2) change the config of the HPO module (e.g. NNI) if the key is found to be one of: - "trialCodeDirectory" - "trialGpuNumber" - "trialConcurrency" - "maxTrialNumber" - "maxExperimentDuration" - "tuner" - "trainingService" and (3) enable the dry-run mode if the user would generate the NNI configs without starting the NNI service. Args: params: a dict that defines the overriding key-value pairs during instantiation of the algo. For BundleAlgo, it will override the template config filling. Notes: Users can set ``nni_dry_run`` to ``True`` in the ``params`` to enable the dry-run mode for the NNI backend. N)r hpo_paramsrrkrkrlreszAutoRunner.set_hpo_paramszdict[str, Any])rfrxcCsXd}|D]:\}}d|kr6t|d|d|d|t|d9}q ||_||_|S)a$ Set the search space for NNI parameter search. Args: search_space: hyper parameter search space in the form of dict. For more information, please check NNI documentation: https://nni.readthedocs.io/en/v2.2/Tutorial/SearchSpaceSpec.html . r_valuez key z value z has not _value)rprErPrfrg)rhrfZvalue_combinationsrsrtrkrkrlset_nni_search_spaceszAutoRunner.set_nni_search_spacezlist[dict[str, Any]]None)rurxcCsn|D]d}|tj}t|jdr0||j|jn ||j|}ttj|i}t |fd|j i|qdS)a Train the Algos in a sequential scheme. The order of training is randomized. Args: history: the history of generated Algos. It is a list of dicts. Each element has the task name (e.g. "dints_0" for dints network in fold 0) as the key and the algo object as the value. After the training, the algo object with the ``best_metric`` will be saved as a pickle file. Note: The final results of the model training will be written to all the generated algorithm's output folders under the working directory. The results include the model checkpoints, a progress.yaml, accuracies in CSV and a pickle file of the Algo object. r template_pathN) rALGOrr#rr get_scorerSCORErr)rhru algo_dictalgoaccZalgo_meta_datarkrkrl_train_algo_in_sequences   z"AutoRunner._train_algo_in_sequencec Csdtjdddddiddd d }tt|jdd }|jd d }|D]v}|tj }|tj }t ||jd}| } t |} |jD]} | | kr|j| | | <q| d|i| d|jid| d|j} | d| itjtj|j|d} tj| | dddt|jtt|d}d| d}|rJtd|qHt|ddtt|jdd }|||krtdtt|jdd }qld}t|ddtd||}qHdS)a! Train the Algos using HPO. Args: history: the history of generated Algos. It is a list of dicts. Each element has the task name (e.g. "dints_0" for dints network in fold 0) as the key and the algo object as the value. After the training, the algo object with the ``best_metric`` will be saved as a pickle file. Note: The final results of the model training will not be written to all the previously generated algorithm's output folders. Instead, HPO will generate a new algo during the searching, and the new algo will be saved under the working directory with a different format of the name. For example, if the searching space has "learning_rate", the result of HPO will be written to a folder name with original task name and the param (e.g. "dints_0_learning_rate_0.001"). The results include the model checkpoints, a progress.yaml, accuracies in CSV and a pickle file of the Algo object. r4r 1hnameZ GridSearchlocalT)platformZ useActiveGpu)ZtrialCodeDirectoryZtrialGpuNumberZtrialConcurrencymaxTrialNumberZmaxExperimentDurationZtunerZtrainingServiceroZ nni_dry_runF)rrZexperimentNamerfz/python -m monai.apps.auto3dseg NNIGen run_algo  Z trialCommandz_nni_config.yamlr5N)r8rvrznnictl create --config z --port 8088z;AutoRunner HPO is in dry-run mode. Please manually launch: )checkznnictl stop --allzNNI completes HPO on )rrrrPr rrpoprIDrr Zget_obj_filenamerrwrfr;r<r?r>rrYminrgrrXr@rArrr)rhruZdefault_nni_configZlast_total_tasksZ mode_dry_runrrrZnni_genZ obj_filenameZ nni_configkeyZ trial_cmdZnni_config_filenameZ max_trialcmdZ n_trainingsrkrkrl_train_algo_in_nnisN      zAutoRunner._train_algo_in_nnic CsB|jr`|jdk r`tdt|j|jfd|ji|j}|d}t j |j d|jdn td|j rtj|jstd|jdt|j|j|j|j|j|j|jd }|jr|j|j|j|j|j|jd n|j|j|j|jd |}t||j dd n td |j dk}|j s>|r|j!dst"|jdd}t#|dkrltd|jd|rdd|D}|rtd|ddd|D}t#|dkr|j$s|%|n |&||j ddn td|j'r4t(f|j|j|j|j)t*|j+ddkd|j,|j-}|.|j+tddS)z- Run the AutoRunner pipeline NzRunning data analysis...Z output_pathT)r!rnzSkipping data analysis...z"Could not find the datastats file z=. Possibly the required data analysis step was not completed.)r Z algo_pathr(Zdata_stats_filenamerGr*r+)r2rarbr))r2r))r"z Skipping algorithm generation...r#Frorz#Could not find training scripts in zE. Possibly the required algorithms generation step was not completed.cSs g|]}|tjr|tjqSrk)r IS_TRAINEDrr|hrkrkrlr~ds z"AutoRunner.run..zSkipping already trained algos z3.Set option train=True to always retrain all algos.cSsg|]}|tjs|qSrk)rrrrkrkrlr~js )r#zSkipping algorithm training...rr)rGrr2rZmgpuz-Auto3Dseg pipeline is completed successfully.)/r!rr@rArr1r.rZZget_all_case_statsrrrrrKr"r;r<r=rErr rr(rGr*r+ragenerater2rbr)Z get_historyr r#rJr rPr$rrr&r rrXrr,rrun)rhdaZbundle_generatorruZauto_train_choiceZ skip_algosZensemble_runnerrkrkrlr#s             zAutoRunner.run)rNNNNNFrTFNTNN)FN)r)N)NNNN)r)N)N)N)__name__ __module__ __qualname____doc____annotations__rmrIrKrVrr`r[r\r_rr]r^rerrrrrkrkrkrlr(sR , % E';Cr)- __future__rr;rSrcopyrtimertypingrrrZmonai.apps.auto3dseg.bundle_genrZ"monai.apps.auto3dseg.data_analyzerrZ%monai.apps.auto3dseg.ensemble_builderr Zmonai.apps.auto3dseg.hpo_genr Zmonai.apps.auto3dseg.utilsr r monai.apps.utilsr Zmonai.auto3dseg.utilsrZ monai.bundlermonai.transformsr monai.utilsrrrrmonai.utils.miscrrrr@rrLrrkrkrkrl s*