Я некоторое время запускал snakemake на кластере компьютеров, на котором запущен Slurm, просто запустив snakemake как обычную командную строку, как показано ниже:
#!/bin/bash
#SBATCH --account=def-me
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32
#SBATCH --mem=128000M
#SBATCH --time=01-00:00 # time (DD-HH:MM)
#SBATCH --mail-type=ALL
snakemake --snakefile my_file --latency-wait 60
Пока администратор кластера не уведомил меня о том, что мои задания создаютхаос в кластере
Согласно сисадмину Грэма, ваши задания Python снова создают хаос, выделяя огромное количество потоков / процессов: [root @ gra572: ~] uptime 20:35:54 up 70дней, 0 минут, 1 пользователь, средняя загрузка: 978,96, 972,12, 966,96
[root @ gra572: ~] ps -fu my_id | wc -l
41
[root @ gra572: ~] ps -Lfu my_id | wc -l
1069
И, проверяя детали команды ps -Lfu, я вижу, что snakemake генерирует сотни экземпляровПравило, которое должно было использовать только 4 узла процессора.Вот часть результатов команды 'ps -Lfu':
UID PID PPID LWP C NLWP STIME TTY TIME CMD
masih 1989 1984 1989 0 1 00:16 ? 00:00:00 /bin/bash /var/spool/slurmd/job10587690/slurm_script
masih 1997 1989 1997 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2002 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2005 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2008 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2011 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2014 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2017 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2020 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2023 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2027 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 1997 1989 2030 0 11 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python /home/masih/Sim3.6/bin/snakemake --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS -j 32
masih 2003 1997 2003 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/1/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2004 2003 2004 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/1/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2004 2003 2069 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/1/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2006 1997 2006 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/8/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2007 2006 2007 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/8/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2007 2006 2072 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/8/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2009 1997 2009 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/0/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2010 2009 2010 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/0/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2010 2009 2075 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/0/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2012 1997 2012 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/3/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2013 2012 2013 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/3/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2013 2012 2083 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/3/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2015 1997 2015 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/2/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2016 2015 2016 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/2/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2016 2015 2087 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/2/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2018 1997 2018 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/5/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2019 2018 2019 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/5/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2019 2018 2089 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/5/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2021 1997 2021 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/6/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2022 2021 2022 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/6/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2022 2021 2078 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/6/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2024 1997 2024 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/9/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2026 2024 2026 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/9/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2026 2024 2091 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/9/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2028 1997 2028 0 1 00:16 ? 00:00:00 /bin/sh -c cd /home/masih/BacterialSimulator/Pipeline/Modules/pyseer && \ /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/4/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS \ --force -j32 --keep-target-files --keep-remote \ --attempt 1 \ --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \ --latency-wait 5 \ \ \ --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2029 2028 2029 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/4/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
masih 2029 2028 2080 0 2 00:16 ? 00:00:00 /home/masih/Sim3.6/bin/python -m snakemake /project/6006375/masih/3000LGT0.2gLGT0.04/simulations/0/GWAS/pyseer/20/4/pyseerEval.json --snakefile /home/masih/BacterialSimulator/Pipeline/inputs/settingGWAS --force -j32 --keep-target-files --keep-remote --attempt 1 --force-use-threads --wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --notemp --quiet --no-hooks --nolock --mode 1 --allowed-rules PySeer
Snakemake имеет несколько инструкций для работы на кластерах, но не описывает, как установить параметры для slurm.Я нашел страницу github, которая, кажется, является версией sankemake, совместимой с slurm (https://github.com/Snakemake-Profiles/slurm), но я не уверен, является ли это решением этой проблемы или чего-то не хватает в опциях, которые я установил для создания змеи. Я ценюлюбой намек на то, как решить проблему.