I am just a beginner in MySQL, I need to know how much data can be stored in MySQL. I am developing a web crawler, can I store all the data in MySQL, or do I need to use another Database? Which is more faster? What I mean is, which has the highest Writing/Reading Rate? Do I need to reconfigure to add more data?
-
4You mean the entire data of the web? :)– Daniel VassalloCommented Aug 5, 2010 at 16:54
-
You can store as much data in MySQL as you have disk space and your OS can cope with. There's really not a lot of advice anyone can provide, given so vague information.– nosCommented Aug 5, 2010 at 16:55
-
@Daniel :yep @MarkJ:why?– Alex MathewCommented Aug 5, 2010 at 16:58
-
@Tom Gullen sure it is(given the OS can account for all the disk space), you might not be able to store it all in one thable though.– nosCommented Aug 5, 2010 at 16:58
-
@Nos, you edited your comment and it is now correct. What you said previously however was not! :-)– Tom GullenCommented Aug 5, 2010 at 17:05
|
Show 1 more comment
3 Answers
Depends on the operating system.
Operating System File-size Limit
Win32 w/ FAT/FAT32 2GB/4GB
Win32 w/ NTFS 2TB (possibly larger)
Linux 2.2-Intel 32-bit 2GB (LFS: 4GB)
Linux 2.4+ 4TB
Solaris 9/10 16TB
MacOS X w/ HFS+ 2TB
NetWare w/NSS file system 8TB
http://dev.mysql.com/doc/refman/5.0/en/full-table.html
Your write/read rate is of little concern to you; your bottleneck will be your internet connection.
-
i am using a win32/NTFS system,what will i do if the data goes higher than 4tb??its already 3TB Commented Aug 5, 2010 at 16:58
-
Find a different DBS, SQL server on a 64bit machine can have 16TB. msdn.microsoft.com/en-us/library/ms143432.aspx Just shop around and see what tickles your onion jar. Oracle can handle 8589541376 G on a 'BFT' addressing scheme. I don't think MYSql was designed with your needs in mind. Commented Aug 5, 2010 at 17:02
-
@Tom Gullen: So you are suggesting SQL Server,is there any Open Source DB for my needs?what DB did google using? Commented Aug 5, 2010 at 17:04
-
@Alex: google is using homegrown BigTable (stackoverflow.com/questions/362956/…).– WrikkenCommented Aug 5, 2010 at 17:08
-
@Wrikken : Thks for the info,in my case what all DB can use?what are the solutions? what about Hadoop?is it a DB,can i use that for a webcrawler? Commented Aug 5, 2010 at 17:11
https://forums.mysql.com/read.php?22,379547,381106
InnoDB Size Limits
- Max # of tables: 4 G
- Max size of a table: 32TB
- Columns per table: 1000
- Max row size: n*4 GB
- 8 kB if stored on the same page
- n*4 GB with n BLOBs
- Max key length: 3500
- Maximum tablespace size: 64TB
- Max # of concurrent trxs: 1023
Nanda Kishore Toomula
Sr DBA,Nokia India
CMDBA 5.0
This comes from their docs. MyISAM permits data and index files to grow up to 256TB by default, but this limit can be changed up to the maximum permissible size of 65,536TB (2567 − 1 bytes).