I've been looking over this for a while. And I'm still confused. This is about my /quests dir.
-According to "du" its 900. (900kb)
-According to "du -h" its 900kb. (Same)
-According to "du -b" its 13908. (bytes)
-According to FTP, right click->Properties its... 5kb.
Can someone explain to me what is correct? I was afraid it was getting too large for what quests we have in, then thought I was wrong, then looked at it again and began panicking again, then double checked and was relaxed. Then looked again. And repeat. I'm really confused. I had asked before on a *nix forum, but did not understand what they had said. Or at least they were explaining the wrong thing I wanted to understand.
The best way to get the size of a directory is to cd into it, and then type "du -sh ." If I were you, I would not trust the FTP client. The system knows better.
That will give you the size of your directory and its contents.
Recall that du -b prints out the apparent size meaning that it does not take into account such things as cluster fragmentation and filesystem overhead. If your files are highly fragmented, then the actual size taken on disk will be much larger.
"du -sh" gives me 916K. There are 227 files in the quest dir. According to "du -ha" each file is 4.0K which makes no sense. Which is why I'm still confused. I'm really hoping the directory is not almost 1mb. Each file should be a different size because not all players have done the same amount of quests.
227 * 4k = 908k. Makes sense to me that with overhead, that'll be 916k.
As for every file taking up at least 4k, that could be the file system overhead that I was talking about. Recall that if a file needs even just 1 byte, it still has to fill up an entire disk cluster not to mention all the overhead needed to store it in the file system.
The math part made sense but the 4kb file sizes/total size did not.
Yeah I checked -ab before, some were 84, others were 17 etc. So lets say each file is 84b. 227*84=19068. 19068b/1024 =~ 18.6kb. This is where I'm confused. It says 908kb while its it can't be anymore than 18kb.
Unless you are on a distro of Linux that uses Novell's file system, your minimum cluster size will be ~4k regardless of the actual size of the files. When you check du -ab it's reporting back the cluster size of the files, not their actual byte count (du -b). The actual byte count will *always* be signifigantly lower than the cluster usage indicates because the file system is highly ineffecient in its file storage.
du -sh will give you the actual file size, including overhead and everything. In other words, if du -sh says it's 900k, then it's taking up 900k.
du -ba and du -b do the same thing as far as sizes are concerned, btw: report the actual space usage. It's du -B that will report the actual size of the file.
In other words, if you want to be storage efficient, you should group multiple players into one large file. That way, the cluster size penalty won't be nearly as bad. In your system, every 20 bytes is taking up a whopping 4k. You could fit 200 times 20 bytes into 4k...
If you don't feel like writing the code to manage grouped files, it might be worth your time to get a library that does table storage for you, such as BerkeleyDB. Your system might even have BerkeleyDB already installed - a lot of programs use it for its flat-file storage functionality.
I see. So now it doesn't matter how many quests are in the game since it'll be at 4k unless we have so many quests it goes past 4k, right? It just matters how many quest files there are. And there can only be as much quest files as there are player files, which means it'll never go past the total size of the pfile dir. BerkeleyDB eh? What would it do? Store the files into one file? Or something else?
That's right. No matter how big the file is, it'll be 4k. And if you ever have 4k+1 of actual size, the file will take up 8k. This doesn't matter a lot for large files (what is 4k when you already have 10mb?) but it really kills on the smaller files like the ones you have.
BerkeleyDB provides a simple table storage mechanism. You define key-value pairs. It's not a 'database' in the SQL sense of the term. That is, you can't have multiple columns in your tables, etc. You just have unique key to value.
In your case, the key could be the player name, and the value could be the quest data. You could even, conceivably, store the file you're currently writing to disk in string form in the database; you would load that string and parse it using the code you already have.
The advantage to using BerkeleyDB for you would be that you wouldn't get the massive overhead of storing lots of small files. In parallel, however, you'll have to deal with the BerkeleyDB stuff, making sure you open/close the file correctly. I remember from using BerkeleyDBXML (an actual XML database built around BerkeleyDB) that it wasn't very tolerant of crashes; if the database wasn't closed properly you couldn't open it without doing some voodoo on it first. BerkeleyDB is supposed to be fairly lightweight; I've seen it used in many places but have never used it myself. My suggestion would be to check it out and see if it seems reasonable.
If that doesn't work, my suggestion would be to either find another library, or write some code that stores quest data in groups. For example, all players whose names start with 'a' could be in one file. That will help a great deal with your overhead problem.
Well this was the snippet (I think from MudMagic) that has the questcomplete mprogs etc. I don't feel ready to attempt any sort of table storage etc or anything. I think this is best to leave how it is now. Do you have any idea how many lines (one word/line) it would take to go past 4kb?