Mystery of the disappearing disk space

Disappearing hard diskA few days ago, my wife’s iMac started running out of disk space. As in, ‘Zero bytes available’ in the Finder, with a warning dialog. This was quite odd, as it had a good 20GB free last week. I cleared out a few old video podcasts from iTunes to free up a couple of GB and thought nothing more of it.

The next day, I noticed that the free space had suddenly jumped up to 20GB again.

Then, yesterday morning, it ran out of space again. I cleared a few more big files from the drive, then did a ‘df -h’. 780MB free. I did another ‘df -h’. 779MB free. Then 778MB free. It was dropping at a rate of 1-2MB/second.

I knew that iTunes was subscribed to quite a few podcasts and, indeed, it had been downloading a couple of episodes recently. So I shut down iTunes and did another ‘df -h’. 772MB free… 771MB…

Must be the backup, I thought. I do a nightly rsync backup from my iMac to hers, and sometimes it’s still going during the day. But a ‘ps -aux | grep rsync’ on both machines brought up nothing. The backup wasn’t running.

Very strange. Maybe it was some sort of virus, worm, or rootkit that was downloading from the net? I shut down the AirPort network on the iMac to stop all network activity.

745MB free… 744MB… 743MB…

The plot was thickening. I shut down all running apps in the Dock, then did a ‘ps -aux’ and killed off all processes except those that looked like core Mac OS X services. 671MB… 670MB…

I used ‘lsof’ to see what files and folders were being written to. Couldn’t see anything suspicious there.

645MB… 644MB…

I obviously wasn’t going to find the problem, so I just rebooted the iMac to kill everything off. It came back up with around 980MB free, and stayed stable. Phew.

This morning, it now shows 21GB free.

Very odd indeed. What could have caused this weirdness? Some sort of virus? A dodgy hard drive? Some weird Mac OS X oddity? Answers on a postcard please – or, more conveniently, in the comments section below!

Bookmark this post:

33 Responses to “Mystery of the disappearing disk space”

  1. Greg Says:

    didn’t duplicate everything and purge it?

  2. Gideon Says:

    I highly doubt it was a worm/virus etc. I’ve had problems in the past with P2P programs that were meant to be downloading to an external HDD, but instead they downloaded to an invisible folder of the same name as my external HDD but on my internal drive. It also failed to move the files over to the external once they had completed downloading.

    I discovered this using OmniDiskSweeper program. Though you could do the same thing in teminal I though this program was much faster at showing me EVERYTHING on my HDD, including invisible folders and the sizes of each. It’s shareware so you have to pay to get it to delete stuff for you.. but it sounds like you know what you’re doing in Terminal, so you could probably do what I did, and just use OmniDiskSweeper to find the files/folders in question and use Terminal to delete.

    Gideon

  3. Gideon Says:

    Though it’s possible it was Virtual Memory going absolutely nuts. Therefore a restart was going to be pretty much the only remedy. I’ve never heard of this before though.

    Check out Activity Monitor in the Utilities folder to see the virtual memory usage for each process running on your Mac

    Gideon

  4. Matt Says:

    Yes, I was thinking virtual memory too. As I say, I shut down the network and the disk space still went down, which rules out P2P.

    Thanks for the tips re OmniDiskSweeper and Activity Monitor. If it happens again, I’ll check them out! Actually OmniDiskSweeper sounds like a useful bit of kit regardless.

  5. Flameboy Says:

    Ok, this isn´t a quick answer, but I´m having the same problem right now, and it appeared meanwhile running rsync.

    After rebooting, the problem remains. What to do?

  6. Matt Says:

    @Flameboy: One app that might help is What’s Keeping Me?, which I plan to review soon: http://www.hamsoftengineering.com/products/wkm/wkm.html

    This tells you what processes are using what files, which may help to track it down.

    and/or try lsof as I describe in my post above.

    Does that help?

  7. Flameboy Says:

    Matt: Thanks for the hint, but I don´t really se how “What’s Keeping Me?” would help me here?
    The problem seems to be that reported disc use differs from my actual use.

    None of the things above did help me out.

    I ran a utlility called “What size”, and if I summarize the files, I don´t get the same result as Finder do… Weird!

  8. Matt Says:

    @Flameboy: Hmm good point. Trawling through the output of ‘sudo lsof’ might be a better bet than WKM, though it would be a tedious process. You’d need to locate suspect files, then ‘ls’ or ‘tail -f’ them to see if they’re growing. Then you could use lsof again or WKM to identify the process that’s writing to the file.

  9. Flameboy Says:

    I figured it out!

    The problem is caused by a bad mount. When a mount process fails, the system automaticly create a mount and saves the data under /Volumes/NAME_OF_THE _MOUNT.

    So, what I did was only to delete this directory (with ´sudo rm -rf /Volumes/NAME_OF_THE _MOUNT´), and boom, my free space was back again!

    Thanks for taking time helping me, hopefully someone else will be able to use this.

  10. Matt Says:

    @Flameboy: Glad you sorted out the problem! :) Thanks for posting your solution – I hope it helps others.

  11. Will Says:

    Thanks for posting this. Sometime in the last 3 days 120GB of my hard drive just disappeared, finally to be found hiding in the Volumes directory. You’ve saved me a complete erase-and-reinstall.

  12. Matt Says:

    @Will: Wonderful, glad it helped! Thanks again for the solution Flameboy. :)

  13. Tim Says:

    This is no help, but at least you’ll know you’re not alone. I run OS X 10.5 with 80GB HD. I presently have only system and app files, no actual ‘created’ docs or images on the HD and keeps going to zero free space. I haven’t been able find or fix the problem and neither has anyone else. I keep all my work on an external drive, using my laptop virtually as a CPU and screen only.

    Every time I hit zero space and can’t perform simple functions like saving a word file, I trash some little used apps or transfer them to the external hard drive.

    I suspect I have about 60 + gigabytes of space used up somewhere on the machine. It’s like a creeping cancer, literally like an aggressive tumour taking over the hard drive space. I’ve basically given up after so many people have tried and failed to find let alone fix the problem.

    I’m replacing this machine in a few months, so I was going to wait and install all new apps etc.

  14. Tim Says:

    PS I have no file called NAME_OF_THE_MOUNT.

  15. Matt Says:

    @Tim: I think by NAME_OF_THE _MOUNT Flameboy means the actual name of your mount point…

  16. Jeff Says:

    I had similar symptoms but no extra mounts under /Volumes. Using disksweeper I found the problem – .asl log files from yesterday and today that were 3GB and 6GB! Manually deleted these, rebooted and wow, I have 9GB disk space now!

    If you don’t have any extra mounts under /Volumes, check your log files
    /private/var/log/asl/2009.08.12.asl was the name of the large file I found.

    Obviously a huge log file points to another issue, in my case I believe it was caused by copying large amounts of files overnight to a NAS backup box.

  17. Matt Says:

    @Jeff: Ahh, those pesky log files! Good tip – thanks :)

  18. Ryan Says:

    I have had similar problems of disappearing memory driving me up the wall because I keep deleting files left and right sometimes files I actually want. Omnisweeper does show 160GB of asl (know idea what that means). Do I simply erase this? I am really hesitant erasing things thinking my computer won’t run properly.

    Please help.
    thanks.

  19. Matt Says:

    @Ryan: Memory? You mean disk space, right?

    ASL is Apple System Log. If you have 160GB of ASL log files then something is probably not right. See http://forums.macrumors.com/showthread.php?t=641936 .

  20. Jere Says:

    please help.. Im running Mac os x server and i recently upgaded to entropy php 5.3.0 last night and I woke up this morning to NO disk space. its an imac g4 with leopard 10.5 and a messagsaying your startup disk is low.

    what the heck is going on here? where do I find volumes ? pls help.

  21. Matt Says:

    @Jere: My first thought is that your apache or php process is dumping lots of stuff in its log file. Take a look at the files in /var/log/apache2 – are they huge?

    To browse your /Volumes folder, open up a Terminal and type:

    ls -lh /Volumes

    A quick way to find the largest files on your disk (assuming you can actually boot OS X okay) is GrandPerspective:

    http://mac.elated.com/2009/01/26/grandperspective-quickly-find-big-files-on-your-mac-disk/

    Hope that helps!
    Matt

  22. dimitri delcourt Says:

    I got this problem too, the solution was the same as Flameboy: a Share point got somehow duplicated locally.

    Rsync continued to backup to the Volume, !!locally!!, the REAL Sharepoint being renamed Sharepoint-1 (which is unfortunately not visible in the finder!).

    I went to the hidden /Volumes folder, found a folder being my supposed sharepoint, trashed it, got my 80GB! back, things are smooth now.

    Only question, what might have been the cause, how to prevent it?

  23. Matt Says:

    @dimitri: I’m not too sure what causes those extra mount points. I think it might happen if more than 1 user on the same Mac mounts the same volume?

  24. Rowan Says:

    I had the same problem! I couldn’t figure it out. Using the ‘grandperspective’ program: http://grandperspectiv.sourceforge.net/
    it was very useful! It identified a massive log file (12+ GB) – it had the text ‘rbot’ in the filename.
    File: periodic.yvrbotdkvs

    It was ‘recently modified’ up to the minute- and it was growing steadily as my HD space was dropping.

    I couldn’t delete it as it was ‘in use’ and it asked for my admin password to try to delete it. I rebooted, and now- all seems OK. Not sure what happened there.

    Anyone know what this ‘periodic’ file name might be??

    Thanks for everyone’s comments in this thread.. its been really helpful.

    Rowan

  25. Matt Says:

    @Rowan: That is strange – I’ve never heard of such a log file. Mac OS X does have the “periodic” utility that’s used for running cron jobs (try typing “man periodic” in Terminal). Maybe it’s connected with that somehow?

    Rebooting once in a while is always a good idea anyway – it tends to clear out unwanted temp files, log files, memory and so on.

  26. marc meyer Says:

    I had a similar problem. 20 gig disappeared in a matter of minutes on my osx snow leopard. I eventually tracked it down with OmniSweeper, which identified about 100GB which were being used by Apple’s TechTool Pro to backup Directory information for all of my USB disks!

    I used the System Preferences panel for TechTool to disable the directory backup (I already use TimeMachine and SuperDuper for backups), and used OmniSweeper to delete the backup info.

    OmniSweeper is now free, thanks to the talented OmniGroup folks.

  27. Matt Says:

    @marc: Ouch! I like OmniDiskSweeper – nice product.

  28. jason Says:

    thanks so much. had a user at the office this was happening to and it was indeed the mad mount situation. we unmounted some external drives then looked at /Volumes. voila.

    thanks.

  29. Roy Miller Says:

    I had a TIme Machine volume on my OS X Server that was filling up in week – only holding about a week’s worth of backup. The culprit in this case was the TechTool Directory Protection database. It was tracking info on all my data disks directories, bloating and updating the database file on the timescale of minutes.

    So, if you have installed TechTool on your Mac, run TechTool and check the settings for this function. Also look at the Preference Panel for TechTool in System Preferences.

    BTW: TechTool is *not* an Apple product, although it is provided as part of the AppleCare Protection Plan subscription. TechTool is from a company called Micromat… I suppose that Apple has contracted with Micromat to be able to supply TechTool with the AppleCare packages.

  30. Niels Lantz Says:

    Just leaving a tip here although its a very late reply. My Mail went bananas and saved over 1200 copies of one separate email draft. Found it by using Omnidiskweeper

  31. Joseph Says:

    I recently had the same problem, so I ran OmniDiskSweeper, which quickly crashed but told me that there was 80gb being taken up in the Mail folder in the Library. Same as the guy who posted above mine. It turns out that this was a result of “Recovered Messaged” of which there may have been over a thousand (I have a new-ish email and have about 300 emails in it). After deleting this, my hard drive space returned. The file path is: /Users/Library/Mail/V2/Mailboxes/Recovered Messages(email provider name here, in my case gmail)
    I hope this helped you as much as it helped me.

  32. Juan Says:

    Thanks. It was the wrong mounted volume. With El Capitán it happens.

    I was doing a Gotomeeting session sharing a powerpoint presentation. After 15 minutes or so Finder warned me it had no space in the disk

  33. Matt Says:

    Thanks for your comment Juan. Did trashing the mount point fix the problem?