Sage Video Directory Optimization

[mike1961]

Hello - so I got a 16TB hard drive and I was wondering what would be the best way to optimize it for Sage. I was first thinking about partitioning the drive with 3 or 4TB for permanent imported videos, photos and music and allowing the other 12TB for temporary videos. We keep a lot of temporary videos and I was thinking that thousands of videos all in one folder might not be the best way to optimize things but I'm not sure. If it's not then I'm trying to decide if I should partition the drive and map various letters to the partitions or if I just maybe just create a few sub-folders for the temp videos all in the same partition. Has anyone had any experience with this?

[MattHelm]

I have my "main" imports all set up this way. Then I use folder view to find the show I wish to watch. Root of my import looks like this:

Anime
AnimeNew
Movie
TV_Shows

Then each directory is split as needed. The Anime and Movie directories have "A" through "Z" directories, and then the movies are place by name in those.

The AnimeNew and TV_Shows are split by series name, as both have fewer listings.

Makes it very fast to find anything!

[JustFred]

I've found that creating folders, rather than drive letters, works best.

Disadvantages of creating drive letters:
1. Unnecessarily imposes size restrictions.. a pain when one category grows larger than initially expected.
2. Harder to search using Windows search tools. Have to search each drive letter independently.
3. Moving files between drive letters is slower (has to actually copy the data between logical drives, instead of a near-instantaneous move between folders).

[mike1961]

When formatting the disk do you just set the cluster size to the default or 64K or some other size? Thanks,

[JustFred]

Sage **used** to always recommend setting the cluster size to 64KB to reduce/eliminate pauses/stuttering, but that recommendation seems to have gone away years ago. As a practical matter, for a drive that contains mostly large files (such as video files), 64K still makes most sense (more efficient use of the directory structure). On the other hand, for an OS drive, let Windows pick the optimum default cluster size.

[mike1961]

So you think the NTFS default is okay to use or best to still use 64K for video drives?

[JustFred]

For video drives, I prefer and continue to use 64K cluster size.

[mike1961]

That makes sense. I was thinking even though sage.properties files for the videos are only 1K or 2K even with 5000 files in a directory that would still only be around 320 megs with 64K clusters and then one will have great savings with fragmentation for the really large video files. Thanks

[NetworkGuy]

Originally Posted by JustFred

I've found that creating folders, rather than drive letters, works best.

Absolutely agree. I have multiple import directories and different sub-directories:

• Movies - by genre
• TV - by show name
• Music - by recording artist
• Photos - by event

[mike1961]

I was thinking though that it might be a good idea to at least partition the imported stuff (videos, music and photos) from the temporary video directories. My thought being that the imported stuff is not likely to ever be deleted and only added to so fragmentation would be minor compared to the temporary video folder(s) which always have stuff added and deleted and could need a defrag every so often. Otheriwse, the imported stuff could just be on a completely separate drive if not a separate partition.

[mike1961]

You know I just realized that everyone replying was referring to creating folders for imported files. But, what about the temporary video directories? I'm not sure how Sage decides which one to use but I think if there are several drive letters then it always uses the one with the most space for temporary recordings (recordings from the program guide). So, I'm not sure what would happen if I created separate folders on the same drive if it then looks at how many recordings there are in each folder or space available. Does anyone know the answer to this? Thanks,

Edit: from reading the manual it seems that sage looks at which drive has the most space and might spread two recordings at the same time over two drives. The thing I'm trying to avoid is having tens of thousands of files in one folder so I'm not sure if partitions might be better for a really large drive like 16tb for the temp recordings. Maybe I should partition it for 4tb for imported and the other 12TB maybe have two 6TB partitions for the temp recordings.

[graywolf]

Not really sure that is buying you anything. it is still 1 physical disk so although you have 2 logical partitions/drive letters, when it actually comes to writing, going to be using the same heads, so the arm would have to traverse between the disk locations.

What I have is record to a local drive, then daily I have a SJQ task run a groovy script that moves the files in the "temp location" to my NAS and uses ShowName for the folder and Season # as subfolder to it.

find_tv_to_move

import com.google.code.sagetvaddons.sjq.network.ServerClient;
import com.google.code.sagetvaddons.metadata.Factory;
import com.google.code.sagetvaddons.groovy.api.MediaFileHelpers ;

// What directory are we moving archives to?
File dest = new File("\\\\tower\\");

// Queued tasks are marked with this metadata value
String SJQ4_QUEUED = "SJQ4_ARCHIVED";

// Are we running on Windows?  File names are not case sensitive on Windows so we need to handle that
boolean isWindows = System.getProperty("os.name").contains("Windows");

ServerClient clnt = new ServerClient();
int i = 0;
for(Object mediaFile : MediaFileAPI.GetMediaFiles("T")) {
    String filePath = MediaFileAPI.GetFileForSegment(mediaFile, 0).getAbsolutePath();
    String ShowName = SJQ4_METADATA.get("SJQ4_SEGMENT_0");

    if(isWindows)  
        filePath = filePath.toLowerCase();
    
    
    // If the media file is already stored on the NAS, it's already been queued or it's actively recording then just skip it
    if(filePath.startsWith(isWindows ? dest.toString().toLowerCase() : dest.toString()) ||
//        MediaFileAPI.GetMediaFileMetadata(mediaFile, SJQ4_QUEUED).equals("1") ||
        MediaFileAPI.IsFileCurrentlyRecording(mediaFile) ||
        MediaFileHelpers.isBeingViewed(mediaFile)) 
        continue;
    
//    if ( ShowTitle ^= "The" ) {
//       println  "${ShowName} to be moved ... "
       println  (filePath + " to be moved ... ") ;
	   clnt.addTask("MVMEDIALR", Factory.getMap(mediaFile));
       MediaFileAPI.SetMediaFileMetadata(mediaFile, SJQ4_QUEUED, "1");
//    }
}
clnt.close();
return 0;

/******************************************************************************
     Move a SageTV Media File

     Last Modified: 16 Feb 2011
            Author: Derek Battams <derek AT battams DOT ca>

    Use this groovy script to move a SageTV media file object from one location
    to another.
    
    The media file to be moved is defined by the SJQ4_ID environment variable;
    the SJQ4_TYPE variable must also be "MediaFile".  The destination is given
    as the first command line argument to this script.  The destination is a 
    directory.

    This script will trigger a rescan of your media after a successful move; the
    rescan is necessary in order for SageTV to recognize the media file in its
    new location.
******************************************************************************/

boolean testMode = false; // Don't actually move any files, just print out what would be done
String flagsFile="\\\\PWM1\\Users\\PWM\\Documents\\SageTV\\Move_Recordings_Output\\HTPC-LR\\Move_Recordings_Output.txt" ;


/******************* DO NOT EDIT BELOW THIS LINE ************************/

// But if you do then send me your bug fix patches! ;)

import org.apache.commons.io.FileUtils;
import static groovy.io.FileType.*;
import com.google.code.sagetvaddons.sjq.network.ServerClient;
import java.io.*;
import java.util.*;
import java.text.SimpleDateFormat;
import java.text.DateFormat;

public WriteToFile(String outputFileName, String  content) {
    try {
         File outputFile = new File(outputFileName);
// if file doesnt exists, then create it
         if (!outputFile.exists()) {  
              outputFile.createNewFile();
         }
// true = append to file
         FileWriter fw = new FileWriter(outputFile.getAbsoluteFile(),true);
         BufferedWriter bw = new BufferedWriter(fw);
         bw.write(content);
// \n = new line feed         
         bw.write("\n");
         bw.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
}

DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd_HHmm") ;
Date runDate = new Date();
//string runDT = dateFormat.format(runDate)

String ShowName = SJQ4_METADATA.get("SJQ4_SEGMENT_0");
String type = SJQ4_METADATA.get("SJQ4_TYPE");
String id   = SJQ4_METADATA.get("SJQ4_ID");
File dest   = new File("${SJQ4_ARGS[0]}");
String NewShow = "${SJQ4_ARGS[0]}/${ShowName}";
String SJQ4_QUEUED = "SJQ4_ARCHIVED";

String flagSettings ;

println ("Parm passed = " + SJQ4_ARGS[0] );
println ("New File = ${NewShow}");
boolean RC ;


if(!dest.isDirectory() && !dest.mkdirs()) {
    println("Destination directory is invalid! [" + dest.getAbsolutePath() + "]");
    return 1;
}

if(!"MediaFile".equals(type) || id == null || !id.matches("\\d+")) {
    println("No media file attributes provided!");
    return 1;
}

Object mediaFile = MediaFileAPI.GetMediaFileForID(id.toInteger());
if(mediaFile == null) {
    println("No media file for id " + id);
    return 1;
}

if(MediaFileAPI.GetNumberOfSegments(mediaFile) == 0) {
    println("Zero file segments for media file; nothing to move!");
    return 1;
}

boolean WatchedFlag = AiringAPI.IsWatched(mediaFile) ;
boolean ArchivedFlag = MediaFileAPI.IsLibraryFile(mediaFile) ;
boolean LikeFlag = AiringAPI.IsDontLike(mediaFile)
int origAiringID = AiringAPI.GetAiringID(mediaFile);
String showExternalID = ShowAPI.GetShowExternalID(mediaFile);

println("Moving " + Arrays.toString(MediaFileAPI.GetSegmentFiles(mediaFile)) + " to destination: " + dest.getAbsolutePath());
// Copy all segments then delete the originals if more than one segment otherwise just do a filesystem move op
if(MediaFileAPI.GetNumberOfSegments(mediaFile) == 1) {
    try {
        if(!testMode)
            FileUtils.moveFileToDirectory(MediaFileAPI.GetFileForSegment(mediaFile, 0), dest, false);
        else
            println("Would move: " + MediaFileAPI.GetFileForSegment(mediaFile, 0));
    } catch(IOException e) {
        println("Failed to move file to destination!");
        e.printStackTrace();
        return 1;
    }
} else {
    def copied = [];
    // Copy the files to their new home
    for(File segment : MediaFileAPI.GetSegmentFiles(MediaFileAPI.GetMediaFileForID(id.toInteger()))) {
        try {
            if(!testMode) {
                FileUtils.copyFileToDirectory(segment, dest, true);
                copied.add(new File(dest, segment.getName()));
                println("\tCopy of '" + segment.getAbsolutePath() + "' completed...");
            } else {
                println("\tWould move: " + segment.getAbsolutePath());
            }
        } catch(IOException e) {
            println("Error copying segment '" + segment.getAbsolutePath() + "' to destination!");
            e.printStackTrace();
            copied.each {
                if(!it.delete())
                    println("Failed to delete '" + it.getAbsolutePath() + "'");
            }
            return 1;
        }
    }
}
 
println("Moving artifacts...");
mvArtifacts(id.toInteger(), dest, testMode);

if(!testMode) {
    RC=MediaFileAPI.DeleteFile(MediaFileAPI.GetMediaFileForID(id.toInteger()));
    if ( !RC ) {
       println ("DeleteFile failed");
    } else {
       println ("DeleteFile success");
	}
	//    ServerClient sc = new ServerClient();
    File NewFile   = new File(NewShow);
    println ("NewFile string = " + NewFile.toString() );
	RC=MediaFileAPI.AddMediaFile(NewFile, null);
    if ( !RC ) {
	   println ("AddMediaFile failed");
   } else {
	   println ("AddMediaFile success");
	}

    Object mfNew = MediaFileAPI.GetMediaFileForFilePath(NewFile);
	if (mfNew == null ) {
	   println ("mfNew is null") ;
	}

	Object origAiring = AiringAPI.GetAiringForID(origAiringID);
	RC=MediaFileAPI.SetMediaFileAiring(mfNew,origAiring ) ;
    if ( !RC ) {
	   println ("SetMediaFileShow failed");
    } else {
	   println ("SetMediaFileShow success");
	}
    if ( WatchedFlag ) {
        AiringAPI.SetWatched(mfNew);
        println("${NewShow} Should be Watched...");
        isWatched = "Watched" ;
    } else {
        AiringAPI.ClearWatched(mfNew);
        println("${NewShow} Should be Unwatched...");
        isWatched = "Unwatched" ;
    }

    if ( LikeFlag ) {
        AiringAPI.SetDontLike(mfNew);
        println("${NewShow} Should be Don't Like...");
        dontLike = "Don't Like" ;
    } else {
        AiringAPI.ClearDontLike(mfNew);
        println("${NewShow} Should be Like...");
        dontLike = "Like" ;
    }

    if ( ArchivedFlag ) {
        MediaFileAPI.MoveFileToLibrary(mfNew);
        println("${NewShow} Should be Archived...");
       isArchived = "Archived" ;
    } else {
        MediaFileAPI.MoveTVFileOutOfLibrary(mfNew);
        println("${NewShow} Should be UnArchived...");
       isArchived = "UnArchived" ;
    }
    AiringAPI.Record(MediaFileAPI.GetMediaFileAiring(mfNew));

    flagSettings= dateFormat.format(runDate) + "|" + NewFile.toString() + "|${origAiringID}|${isArchived}|${isWatched}|${dontLike}" ; 
	WriteToFile(flagsFile, flagSettings) ;
	
}

//MediaFileAPI.SetMediaFileMetadata(mediaFile, SJQ4_QUEUED, "1");

println ("Move completed") ;
 
return 0;

def mvArtifacts(int mfId, File dest, boolean testMode) {
    for(File segment : MediaFileAPI.GetSegmentFiles(MediaFileAPI.GetMediaFileForID(mfId))) {
        String prefix = segment.getName().substring(0, segment.getName().lastIndexOf('.'));
        File baseDir = new File(segment.getParent());
        baseDir.eachFileMatch FILES, {!new File(baseDir, it.toString()).equals(segment) &&  it.toString().startsWith(prefix)}, {
            File artifact = new File(baseDir, it.getName());
            if(!testMode) {
                FileUtils.moveFileToDirectory artifact, dest, false;
            } else
                println("Would move artifact: " + artifact.getAbsolutePath());
        }
    }
}

[mike1961]

I was thinking the way it might help is rather than 20,000 files in a single folder (as an example) I would only have half that number and also fragmentation would be less of an issue. But, as you mentioned, I'm not sure how much that would help.

[Tiki]

I had no idea what you meant by “temp video files”. It sounds like you are talking about the Sage Recording Directories.

I really don’t think it matters whether you partition the drive or not as far as Sage is concerned. It’s simply a matter of personal preference. Years ago there were compelling reasons for partitioning a drive (such as the fact that older versions of Windows couldn’t handle partitions larger than a particular size). But these days I think it’s usually better to just leave one big partition.

One thing you don’t want to do is to create two Sage recording directories on the same partition. This will confuse Sage’s logic for how much free space is available and how much free space it needs to keep.

[mike1961]

I thought it might matter but perhaps not. Figure 16TB with one folder for a Sage Recording Directory and figure an average of 2GB per video is 8,000 files then double that since each could have a sage properties file which is 16,000 files in one folder. Maybe that's not a problem but at some point I would think the number of files in a folder could be an issue but I'm not sure how many one would have to have.

[xjim1]

I move stuff around quite a bit and prefer folders, a lot quicker than moving between partitions. You can still move your recorded files into different folders. Sage will still see them as long as they're under either a recording directory or imported videos folder. I do what others have mentioned and organize by series and seasons, or whatever. You also won't have to be concerned as much about defrag'g either.

[UgaData]

Windows has a limit to the number of files in the ROOT Directory of a Drive(:\, 250 files or so) but there is no limit within Folders/Directories.

I always format DATA drives with 64K blocks. And defrags can be scheduled with the OS to run in the background (for me once a week).

I also set the drive letter to be something unlikely to be a local drive letter (something greater than E: - usually R: thru U: ), this way there are no conflicts when accessing the Drive from other clients (i.e. both client and server will see U: as the share - for import directories it helps if the server and client see the same share. Of course if you only use extenders this may not matter)

[nyplayer]

Originally Posted by UgaData

Windows has a limit to the number of files in the ROOT Directory of a Drive(:\, 250 files or so) but there is no limit within Folders/Directories.

I always format DATA drives with 64K blocks. And defrags can be scheduled with the OS to run in the background (for me once a week).

I also set the drive letter to be something unlikely to be a local drive letter (something greater than E: - usually R: thru U: ), this way there are no conflicts when accessing the Drive from other clients (i.e. both client and server will see U: as the share - for import directories it helps if the server and client see the same share. Of course if you only use extenders this may not matter)

I find that 64K blocks is no longer needed and uses more space when you store EDL files and other small files.