Uncategorized

A scalable and adaptable standardised user file structure?

I’ve got two more mock exams to cover (both on Monday), some University applications progress has been made, more and more of the ThoughtScore project is inching its way into my WIPUP profile (starting to get to the interesting bit), and I’ve opened Blender again to do a quick animation favour which can be seen in my Uncategorised category.

But this post is nothing to do with those fun and amazing things. No. Today I want to talk about something that warrants 3-4 syllable words being used in the post title.

The ins and outs of operating systems aren’t exactly my speciality but I do know that they have a uniform file structure. In Windows, a lot of system stuff goes in C:\WINDOWS, your user files goes in Documents and Settings, and programs go into Program Files. In Windows inside your user file you automatically get a set of folders such as My Documents, My Pictures, etc. In Linux (and all UNIX I believe) your user’s folder is in /home/username/ and other than having hidden dotfiles to store local application settings it’s basically empty. At most your desktop environment or distro will add a "Desktop" folder.

The question is how should I archive and organise my files such that they’re easy to find, easy to transport, and easy to manage. Sure, the Microsoft approach works for some people, but most of the people I know completely ignore that structure. I’ve been thinking about it for a while and have isolated a few ways people normally try to manage it.

By file extension. It’s easy to find and easy to archive. However it’s difficult to manage once you start getting rare file extensions or files with no extensions. It also becomes a pain when your program output or save files require several files of different types to be grouped together. Files can become strewn all around the computer and projects which require hardlinking to files will easily get out of hand.

By file type. This one is only slightly different to file extension in that people group things by what files are, not their extensions. For example "Images" instead of "jpg", "gif", etc. Pros and cons are similar to above.

By file purpose. This is a project based approach. Files are grouped into their uses, such as "Homework", or "Movies", or "Project X". This is rather commonly used but often clashes occur when used in parallel with others such as File extension or file type, especially when the nature of the project requires hardlinking to file locations.

By file attribute. This is often a temporarily used file structure when people are sorting out files. Such examples include "bobs photos", "to be sorted", "jazz music", etc. The main use of this structure is to make it easier to transport the files or manage a bulk collection of files. The sad story is that these directories persist waaaay past their useful lifetime and prove to be effective cloggers.

By time. Useful for archiving, completely regardless of the content. Used for management and for finding, but rarely useful for transporting files.

By referenced location. Some files are put in locations regardless of semantic value and instead for technical convenience – for example dotfiles, plaintext, logs, tmp, backup files, and referenced files from other apps (scripts, programs, graphic projects).

By organised chaos. This is probably how almost every single Windows user’s desktop looks like. A complete mess of random rubbish used to dump stuff. Files are strewn regardless of any attribute and found through searching, indexed searches, and file manager filters.

Now of course searching for the zen of a users’ file structure will only end when a structure is able to accomodate a large number of files, a project-based workflow, technical restraints, archives, and miscellaneous files. In this case it’s useful to define what "accomodate" means, which is an ability to easily find a desired file, to sort and prune undesired or irrelevant files, to prevent duplication, to transport similar files easily (eg: all within one directory), to quickly break down a collection into manageable chunks, and to allow any newcomer to intuitively adapt to your filesystem.

I currently run a structure where my homefolder is where I dump active files, I have a primarily "file type" structure, archives are done through "file purpose" (I despise time), and projects have their own substructures which are completely dependent on referenced locations. For me the biggest inconvenience is referenced locations, where I find myself unable to bulk manage files simply because of the inconvenience of have to re-reference their location. The rest is chaos. All in all, miles away from my personal zen.

Anybody who’s achieved a personal zen are welcome to share.

Dion Moult

I've been developing software for well over 10 years, work as an architect (not the computer kind, the regular sort), and am classically trained as a pianist. I try to do the right thing when I get the chance in my field, such as through contributing to open-source communities and promoting sustainable living.

More Posts

Spread the love
Tagged: , , , , , ,

10 Comments

  1. I store my files according to their major file type.

    E.g.

    Documents -> store all kinds of text and formatted documents, including e-books.

    Images -> a store for images, further sub-classified according to the purpose/project.

    Videos -> video files each movie/tv show in their own subfolders.

    Photo Album -> A separate directory structure for my photos

    Music -> All the music files

    Projects -> My programming projects each in their own subfolder.

    public_html -> my local development web folder.

    That’s about it. It’s a simple system and it works for me. :)

  2. I totally hear you! I’ve been having problems categorizing and locating my files since my very first computer. Though, I recently got a free and legit copy of Windows 7 from my Uni and I must say Microsoft has actually managed to create a pedagogical OS. On the start menu you have a search field which is very fast. And when you search in specific folders it high-lights your search phrase on the results, also very fast. Since the search function is so quick you don’t really have to bother categorizing files and folders anymore =) I’m descovering new handy things on MSW7 every day. Wow, I’m really advertising here. Well, I can recommend you trying it =)

    Erik,
    Sweden.

  3. Hello hari, I guess you don’t have projects of the nature that are quite fussy about referencing files (like 3D projects).

    Erik, nice to see you here again, I’d thought you had disappeared! Anyways it does actually seem that the Windows and Mac folks (especially the Mac folks) tend not to bother so much about file organisation now especially with indexing and things like the spotlight search (which I hear is very effective). However do you run into problems when having to transport or archive files – where such searches don’t work so well?

  4. I manage pritymuch everything within a single, large, SVN repository. within the root of this repository files are organised as follows:

    ANIMATIONS – Stores rendered animations.
    BLENDER – Blender related projects
    MISC – Miscellaneous stuff
    MUSIC – Music that I listen to
    PICTURES – Mainly my photographs, contains some non project specific textures
    PROGS – Programs that I am working on which dont have there own repository, mostly small automation scripts.
    WEBDEV – Like progs, but for websites
    WORK – Collage work

    Primary folders are capitalised so they stand out obviously from other files that get put in there, normally by mistake.

    It would be imposable to list the entire file management system here, as I have a LOT of files, however that’s the basics of it. I also have some files which are maintained out of the primary tree which includes things like programs which have there own repository, a check out of the Blender trunk and a folder containing stuff which I have downloaded off the internet, mostly PDF documents and the like.

    Partial SVN checkouts are used to move the needed parts of the repository across different computers, whenever there is ether not enough space for the whole thing (50 + GB) or preforming a complete checkout would take too long.

    The vast majority of the files are plain text (source code or LaTeX) ,images and blender files.

  5. > Hello hari, I guess you don’t have projects of the nature that are quite fussy about referencing files (like 3D projects).

    I don’t understand. If I have a complex project, I put it under a subdirectory “Projects” which stores all the related files.

    Only the individual documents are stored in other folders.

  6. Hessiess! Even longer time no see! I didn’t know you lurked here.

    Question about your setup – with the Blender projects won’t they link into other directories such as within the renders folder (for sequencing) as well as into the images folder (for generic textures?) As for Webdev, don’t these need to be under the public_html folder (assuming apache is in use)?

    Sounds like a realistic setup, although I don’t think I’m that comfortable with using SVN – it’s rather cumbersome. I just access my files remotely through the cloud.

    hari, I think the main problem is that if you have a very long term project which references files, as your structure gradually changes over time to accomodate other files this old project which you’re afraid will break its links means that it’s inconvenient to move these age-old files. I think that’s that scenario I’m in (and honestly I can’t get out of it at all easily until I finish the project).

  7. hari, I think the main problem is that if you have a very long term project which references files, as your structure gradually changes over time to accomodate other files this old project which you’re afraid will break its links means that it’s inconvenient to move these age-old files.

    Yes, but if you store these project files independently and use all references within its own directory structure, it should be independent of external files outside the project dir, no?

  8. Of course, but over time your structure may change to accomodate new files, and if this goes on for 2 years or so you get a bit stuck :(

  9. I’ve never had such a complex project yet… I would never reference any file outside of a project directory in any case. I still probably don’t understand your predicament though.

    How will changing files outside the directory of the project affect the project itself? I would move ALL files related to the project, regardless of file-type within the project directory.

  10. “Hessiess! Even longer time no see! I didn’t know you lurked here.”

    Hi Dion, Bean lurking for quite a while but there wasn’t anything I had to say, before this post that is :) Any more progress on Thoughtscore?

    “Question about your setup – with the Blender projects won’t they link into other directories such as within the renders folder (for sequencing) as well as into the images folder (for generic textures?) As for Webdev, don’t these need to be under the public_html folder (assuming apache is in use)?”

    All my blender projects have there own local textures and render directories, for example the structure of my BWC 08 entry is as flows:

    BLENDER/projects/The_bird_and_prey : Blend file and final renders
    BLENDER/projects/The_bird_and_prey/refs : reference images
    BLENDER/projects/The_bird_and_prey/textures : Project textures

    Texture images usually get pretty mashed up through UV painting and backing, so they are normally not duplicated exactly. Though the textures probably would be better in a subdirectory of the BLENDER directory. Currently I have not worked on a project which is large enough to require a lot of linking so it hasn’t bean too much of a problem.

    “Sounds like a realistic setup, although I don’t think I’m that comfortable with using SVN – it’s rather cumbersome. I just access my files remotely through the cloud.”

    Subversion has its problems, however from my experiments with other tools (Unison, BZR, GIT) its just the best tool for my current situation, where I am off-line a lot of the time.

    I have a script which automatically adds new files to the repository, deletes ones which have bean deleted in the file manager then commits everything. It almost completely removes the manual work required.

    The biggest problem that I have with other version control systems is the lack of partial checkouts, which I use to keep a copy of my collage work on a USB pen drive.

Leave a Reply

Your email address will not be published. Required fields are marked *