0

SNP filtering for missing and redundant markers

I split a large text file (30 GB in size) to small 40 files in the cluster and saved it as RData. Then I am importing these small RData files to R to filter these for missing and redundant SNP markers. But it is giving an error.

I want to split large files to small files, save them as RData, import to R and filtering for missing and redundant markers.

2
  • What's the error?
    – TJ87
    Sep 3, 2019 at 22:06
  • I would do this in Plink; you may not have to split up the files to do these quality control steps.
    – TJ87
    Sep 3, 2019 at 22:19

1 Answer 1

0

RData file is a binary file and your splited files are tab separated text file. So, they cannot be load with load(). I will suggest removing all header information and column names before you split the vcf file. And load the files into r with read.table(sep='\t'). Hope the above can help.

I don't have any idea about how to save text file to RData file outside r environment. But, if you concern the big file size of your vcf file and you want to filter the data before loading, I suggest using read_tsv_chunked which can load every chunk (chunk_size =) of data and do some process on each chunk of data, and finally available in r environment. And beware of skip the number of header line before column names. The following are the script.

library(tidyverse)

filterFun <- function(df, pos) {
  df <- unique(df)
  count.nas <- rowSums(is.na(df))
  df[-which(count.nas>(0.05*ncol(df))),]
}

data <- read_tsv_chunked("xxx.vcf", skip = 22, chunk_size = 10000, callback = DataFrameCallback$new(filterFun))
4
  • Thank you! Is there a way to save the text file as an RData file.
    – Jessica
    Sep 4, 2019 at 1:17
  • Hi Jessica. I edit the answer and add some code. See if the code can solve your problem.
    – raytong
    Sep 4, 2019 at 6:25
  • Thank you! I will try this. Can you share how to save the text file as an RData file in R environment?
    – Jessica
    Sep 4, 2019 at 12:45
  • Hi @Jessica. You have to create or load somethings into R environment and use save.image("filename.RData) to save all objects in the environment. Or use save(list = c("object name", ...), file = "filename.RData") to selectively save objects that you want.
    – raytong
    Sep 4, 2019 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.