Questions tagged [snakemake]
Snakemake is a workflow management system with a Python-style specification language.
1,334
questions
0
votes
0
answers
8
views
Using Checkpoints (Snakemake) with unknown file contents
I am beyond stuck.
I have a long pipeline that uses Snakemake as the workflow manager. There are several steps (calls to rule files) and as I coded each step in the pipeline, I tested it out ...
1
vote
1
answer
21
views
Snakemake using the first argument of a list as a wildcard
I am trying to run the analysis in snakemake where as a proband I take always the first bam file present in the list, i.e NUM_194 and NUM_123. Is there a way to use as a wildcards the first IDENTIFIER ...
0
votes
0
answers
26
views
No values given for wildcard, expand issue
I'm new to python and snakemake. I'm trying to create a bed file from trf output. I'm reusing code from github, but I don't need to do all of the things the github code does. I've pulled the relevant ...
1
vote
2
answers
15
views
How to load files in a notebook when using Snakemake?
In a data processing project with several steps, using Snakemake, there is a Python Jupyter Notebook in a subdirectory that processes some data:
Notebook processing_step_1/process.ipynb contains:
with ...
1
vote
1
answer
39
views
how to add "wildcard-specific wildcard" via snakemake checkpoint
I can't find appropriate words to describe my need, please see the code.
My ideal workflow would do this:
I have known the categories that will be created during the workflow (cates)
I don't know how ...
0
votes
1
answer
25
views
Handling multiple inputs for command in Snakemake
I'm currently working on a project that involves me using snakemake to run svaba, a variant caller, on genome data. svaba run can take multiple sample files but requires a flag in front of each file.
...
0
votes
2
answers
31
views
Missing input files error for expanded input in Snakemake
I have been having a strange error that I cannot fix. I am using snakemake 7.8.2 in a conda environment on an Ubuntu 20.04 machine (have also tried running this on our cluster and the same error ...
1
vote
0
answers
23
views
Using Docker with Snakemake (as motivated by the Snakemake documentation) does not provide a true docker container?
In order to improve the distribution of my snakemake workflow, I am wanting to Docker-ize it. The way I usually deploy my software stack is through conda environments (saved as .yaml files within an ...
0
votes
1
answer
20
views
Snakemake: creating directories and files anywhere while using singularity
Whenever I use snakemake with singularity and would like create a directory or file, I run into read-only errors if I try to write outside of the snakemake directory.
For example, if my rule contains ...
2
votes
1
answer
24
views
snakemake if else statement within rule input
With snakemake, I would like to have a different input (including a wildcard) within a rule according to a boolean
I've tried :
rule quantify:
input:
if trimmed_fastq:
forward =...
1
vote
1
answer
32
views
Set java_opts for baserecalibrator snakemake wrapper
I am running a snakemake pipeline with some conda wrappers and I'm trying to allocate some more memory to a rule since it's taking ages. I tried to allocate more memory and threads through the java ...
0
votes
1
answer
14
views
snakemake: how to sort glob_wildcards()
How can I get files in alphabetical order using glob_wildcards()?
Suppose I have sample1.txt, sample2.txt, sample3.txt, and sample4.txt in the same directory.
When I run this code:
file_pattern = '...
3
votes
0
answers
29
views
How do I set the default python version for conda envs in snakemake
I'm running an NGS pipeline that's supposedly intended to run on Python 3.6. I am getting some errors and warnings if I just run with Python 3.10.
So, I am starting my snakefile from an conda env that ...
1
vote
1
answer
22
views
Snakemake fasterq-dump wrapper AttributeError: 'Wildcards' object has no attribute 'accession'
I am trying to use fasterq-dump wrapper in my snakemake workflow to download paired-end fastq.gz files. Here is my snakefile:
# read a .txt file including many SRR* accession number
import pandas as ...
1
vote
1
answer
17
views
Using expand to concatenate .tab files in subdirectories which are variables themselves
I have two variables and I would like to concatenate all .tab files for "BOB" and "LISA" separately. How to do this in snakemake? With expand like below it is concatenating all tab ...
1
vote
1
answer
32
views
Apply snakemake rule on all generated files
I want to run a simple script "script.py", which will run some caculayions and periodically spit out a step_000n.txt file with n being dependent on the total file execution time. I would ...
1
vote
0
answers
19
views
log notebook as html using snakemake (not only as ipynb)
I tried snakemakes jupyter notebook integration and I like to have a saved notebook for the rule which can be executed easily anywhere! The docs are great for this!
I wonder if there is also an ...
0
votes
0
answers
22
views
snakemake pipeline automatization
I have a .tsv file that looks like this:
genome locus_tag gene substrate
PalbDSM11370 02121 susC pululan
PalbDSM11370 02122 susD pululan
PalbDSM11370 01210 susC arabinan
...
1
vote
0
answers
38
views
Deterministic colors for Snakemake --dag/--rulegraph output
When I run snakemake --dag or snakemake --rulegraph multiple times on the same workflow, I get an identical layout but different, apparently random, colors. Is there a way I can provide a seed to get ...
2
votes
1
answer
34
views
Snakemake expand+zip function unexpected behavior
I am trying to use Snakemake to process calls to the rnaQUAST tool with multiple inputs delineated by two sets of different, but paired keywords. I do not want all combinations of these keywords, only ...
1
vote
1
answer
29
views
Correctly consuming a multiline config file in snakemake as an input
For various reasons I would like to be able to define my inputs in a separate config file. My current version without using a config file looks like:
rule test:
input:
labs = "data/labs....
1
vote
1
answer
34
views
How to limit usage of disk space in Snakemake?
I work with 8 paired-end fastq files with 150 GB each, which need to be processed by a pipeline with space-demanding sub-tasks. I tried several options but I am still running out out disk space:
used ...
2
votes
1
answer
31
views
snakemake: specify file obtained by glob_wildcards
How can I specify the file obtained by glob_wildcards?
Suppose I have sample1.txt, sample2.txt, sample3.txt, and sample4.txt are in the same directory.
The following code is just an example:
FILES = ...
5
votes
1
answer
40
views
Is there a way for Snakemake to evaluate dynamic Snakefile constructs like `eval` does in GNU Make?
I would like to have various dynamic "shortcuts" (rule names) in my Snakemake workflow without needing marker files. The method I have in mind is similar to eval in GNU Make, but it doesn't ...
1
vote
1
answer
32
views
snakemake: list of pathes in input
I am sorry for low level question, I am junior. I try to learn snakemake along with click. Please, help me to understand, for this example, how can I put a list of pathes to input in rule? And
get ...
1
vote
1
answer
41
views
How to ignore Snakemake's "params have changed since last execution"?
Some very late jobs of a workflow crashed due to a conda environment not being active.
Now, when I try to rerun using snakemake deploy_all --ignore-incomplete all jobs are rerun straight from the ...
0
votes
0
answers
32
views
How to suppress verbose list of remaining jobs if Snakemake job failed?
Since about version 7.0.0 of Snakemake, when a Snakemake job fails, a rather verbose list of remaining outputs is printed to the terminal.
This can be useful, but when you have a large workflow with ...
1
vote
1
answer
52
views
How to access Snakemake parameters inside Snakefile?
I'm trying to figure out how to access the Snakemake parameters inside my Snakefile, e.g. the set target rule (either specified by user or first defined rule). I was able to find some settings in the ...
0
votes
0
answers
21
views
Stop Snakemake automatically downloading results to local directory
I'm running snakemake (7.8.0) using kubernetes and s3. When the pipeline finishes running all the resulting data from s3 downloads to the local directory I initiated the run from. The same data is ...
0
votes
0
answers
31
views
snakemake returns a non-zero exit status on a simple rule when executing with --cluster
The following snakemake rule fails when I execute it with snakemake -r -p --jobs 40 --cluster "qsub"
rule raven_assembly:
"""
Assemble reads with Raven v1.5.0
&...
0
votes
1
answer
22
views
Using a function to set log file id with snakemake
I would like to use a function to set output log file names easily.
I tried the following function
# output_path=simulations/sim1/model/test.out
# idx=2 == add <x> after 2nd element
# x=logs
# ...
0
votes
2
answers
35
views
Words count pipeline with SNAKEMAKE
I'm starting to learn how to use Snakemake and I want to do a very simple pipeline to count the number of words in the source file and write the result to the output file. The pipeline should work ...
0
votes
0
answers
24
views
Way to force snakemake to re-evaluate dag before checkpoint with --list-input-changes?
I wonder if anyone might have some ideas for a small problem I am having with checkpoints. I am trying to produce a workflow that is robust to changes in the sample list, so that necessary rules re-...
0
votes
1
answer
46
views
snakemake repeating rules whose output files already exist
I am running snakemake (v7.6.2) and I noticed that, unlike its 'principles', it is attempting to re-run steps of a pipeline whose output files already exist.
In my first run I had the following DAG:
...
1
vote
1
answer
26
views
Snakemake automatization
I am new to snakemake and I tried to do some short workflow.
It looks like this:
rule extractfeat:
input:
"/media/murva/450A-F844/PULs/PspTF13.gbf"
output:
"...
1
vote
2
answers
42
views
Snakemake trouble accessing nested values in config.yaml
So my issue below is partially solved, however now I'm trying to pass a variable as input in rule all and resolve it to get dependent variables as inputs in another rule. My code:
rule all:
[f&...
0
votes
0
answers
22
views
Snakemake fails to produce output
I'm trying to run fastqc on two paired files (1.fq.gz and 2.fq.gz). Running:
snakemake --use-conda -np newenv/1_fastqc.html
...produces what looks like a sensible DAG:
Building DAG of jobs...
Job ...
0
votes
2
answers
36
views
Using modules in Snakemake
I have a Snakefile which I want to use rules from another Snakefile as a module. The problem is, the first rule doesn't get it's input modified and Snakemake throws a MissingInputException
Here's my ...
0
votes
0
answers
25
views
How to update output files without changing them in Snakemake
I have changed an initial rule that adds links to database files. Snakemake typically requires all dependent rules to rerun, even the database change is not really relevant. I tried to avoid this ...
0
votes
1
answer
48
views
How to run an AWK script within the input section of snakemake (in order to retrieve a snakemake input variable)?
I'm trying to build a snakemake pipeline for analysing RNAseq datasets. Some datasets might be (1) single end or paired end, and (2) the strandedness between datasets may vary as well. I want the ...
1
vote
1
answer
30
views
Getting Snakemake to work nicely with datetime-based filenames
I'm working with a large dataset where each file (stored online) has a URL that looks something like
import datetime
def fname(dt: datetime.datetime):
return "dataset-" + dt.strftime(&...
1
vote
2
answers
41
views
How can I reevaluate params of Snakemake rule in every run?
I have a toy Snakefile:
rule DUMMY:
output: "foo.txt"
resources:
mem_mb=lambda wildcards, attempt: 1024 * 2 ** (attempt - 1)
params:
max_mem=lambda wildcards, ...
1
vote
1
answer
40
views
Snakemake runs rule too many times using config.yaml
I'm trying to create this snakemake workflow which would evaluate raw reads quality using FastQc and create a raport using MultiQC. I use 4 input files and get expected results, however I just noticed ...
2
votes
1
answer
32
views
Prevent rules from rerunning when intermediate file is updated
Let's say I have two rules in my snakemake file
The first rule fetches a remote file and makes a temporary local copy
The second rule uses the local file and performs an expensive task
Now lets say ...
0
votes
0
answers
88
views
Why does the Snakemake GSRemoteProvider `stay_on_remote=True` argument cause different behavior across Windows and Linux?
I have the following Snakefile (using Snakemake v7.6.0):
from snakemake.remote.GS import RemoteProvider as GSRemoteProvider
GS = GSRemoteProvider(project="my-gcp-project-id")
rule example2:
...
2
votes
1
answer
35
views
Why is my snakemake rule with wildcards turned into a localrule?
So, I made this simple Snakefile and I am wondering why my rule is interpreted as a localrule. Usually it's not a problem, but for my runs that need be submitted to SLURM it is. The only way I got it ...
0
votes
0
answers
55
views
"Snakemake" call mpirun using multiple nodes and a slurm-clusterprofile
I am writing a snakemake-workflow which calls a process using mpirun in a slurm-job using a docker-image. It is necessary to use multiple computing nodes and lots of processes. I am using a slurm-...
0
votes
1
answer
31
views
Snakemake MissingInputException
We have always run our Snakemake pipelines through Amazon S3.
snakemake --default-remote-provider S3 --default-remote-prefix '$s3' --use-conda
However, we need to run our Snakemake pipeline locally ...
1
vote
1
answer
37
views
WildcardError in Snakefile
I've been trying to run the following bioinformatic script:
configfile: "config.yaml"
WORK_TRIM = config["WORK_TRIM"]
WORK_KALL = config["WORK_KALL"]
rule all:
input: ...
0
votes
0
answers
33
views
Snakemake only execut the first rule of the rulefile
I'm trying to run a snakemake pipeline. It works but only for the first rule which is merged_fastp. Then, the script is still running but doesn't do anything else. I have to run again the script if I ...