Thank you

[joonas-fi]

I just want to say, that I've seen and produced a truckload of Go code, and the overall code quality of this library is very, very good!

A few highlights:

• The code is self-documenting,
• The package structure is sensible, e.g. boot sector stuff in own package
• I would've had to build a "fragment reader" myself, but you had it ready. It was a joy. :)
• Where you saw fit to add comments, they were truly for need like this line

  gomft/mft/mft.go

  Line 378 in f64df39

  // fragment.Reader). Note that data will probably not align to a cluster exactly so there could be some padding at the

  (when I saw that I immediately realized what the bug in my software was)

Good job! You rock.

Sorry to spam issue tracker, I couldn't find other contact details :)

[t9t]

Wow, thank you so much for the kind words! I'm speechless....

I wrote this library last year for a personal pet project and decided to put it on Github without any expectation that anyone would ever find it, much less use it. I figured oh well, if anyone needs some reference code or whatever, it's there to check out.

You really made my day!

[joonas-fi]

Glad I could bring some positivity, it was the least I could do when your code helped me so much :)

If I may ask, what was your pet project about?

I can explain mine.. I'm doing something totally crazy. First I need to learn how to read NTFS partition so I can do writes (I can't be good at doing writes if I don't even know how the data structures look like).

I'll probably end up using your project as reference code, because in my ideal usage the parsed structure goes in both directions. Your library excels at reading bytes into structs. But I also need to be able to marshal structs into bytes if I want to do writes to NTFS partitions.

I know I sound crazy by trying to write NTFS partitions (efficient cluster allocation, journaling..).. But my use case is somewhat easy because all I'm doing is generate a partition with file content once. So I can just lay out all files sequentially at partition's 50 % mark without any fragmentation (a single data run per file) and then write out the MFT based on those file locations.

After I get that working, then I can move to my actual use case. Now that I can generate static partitions that Windows can boot from, with very little code changes I can on-the-fly project an entire partition from immutable file tree without consuming any more disk space than the file tree! All I've to keep in RAM is the boot sector, MFT, NTFS system files and the cluster locations of the files. Then when a cluster from the partition is read, I know if it hits an actual file or if it's empty space, and the file content I can actually read from the backing file tree.

I.e. projecting a 20 GB disk to the VM doesn't use any disk space and very little RAM.

Why this? To efficiently run Windows applications in sandboxed VMs. Think Docker layers. Docker lets you install applications on top of one base image (OS and application file trees are separated). With overlay file system I can merge the OS and specific Windows application tree into one, then project it to the VM as a regular-looking filesystem.

Separating the OS and application file trees means that I can maintain the base image separately: fresh installation of Windows with weekly Windows updates applied. This layer doesn't have any other applications on top.

Then I have the layer with an application (say, a game on Windows) that needs to be installed only once (we get its file tree by diffing the base and diff disk images after installation). Its VM can receive always-up-to-date Windows VM by just re-parenting its base layer to the newest base Windows 10 image with help of Linux's OverlayFS (also powers Docker).

All nice and efficient: OS file tree stored only once, and we get security benefits by running each app in own VM without anything else installed.

Windows needs writable disk though, but on the host side I can just redirect writes to a copy-on-write diff disk whose changes get deleted once you shut down the VM. The trick is to keep the Windows VM side as free as state as possible. I'll be writing a blog post on these ideas, but the ideas align with how I run my Linux

[t9t]

Wow, that is a very ambitious and interesting use case! I do feel like I should warn you though... when investigating the NTFS file system, I found so many weird things, a lot of things that I found in my own MFT that seemed undocumented, and in general it's all just very complex (well at least to me, as it was the first time I was really diving into a file system). So writing raw data to an NTFS file system is probably very risky!

My use case wasn't nearly as interesting as that. Basically I was a bit piqued that I couldn't find a cross-platform disk analyzer tool, a la QDirStat and WinDirStat. There are a bunch of different apps for different OSes, but nothing really cross-platform. In particular I was looking at an app with a command-line mode so I could create an index on a headless machine and inspect that on another machine using the same app but with a UI (QDirStat provides such a script, but it's not cross-platform).

These apps typically work by just walking the file system (though with some optimizations here and there as far as I recall). That can take quite some time, especially on volumes with a lot of files. But one particular app, WizTree, is really fast: it can scan an NTFS volume on an SSD in a matter of seconds. It does this by just reading the Master File Table.

So my plan was to build a cross-platform disk analyzer tool, which could really quickly scan NTFS volumes by reading the MFT (regardless of the OS of course), and be able to operate in a headless environment, basically filling all of my needs.

However, this project unfortunately failed on two accounts.

The first is actually that I was unable to find a good UI toolkit for Go which is mature, convenient to use, light-weight, and doesn't look too alien to user (preferably using native widgets, but that just seems like a pipe dream after my investigations).

The other was that when reading the MFT of my own disks, I got a lot of conflicting information. Basically, I compared the contents of the MFT that I could read with the same information that I could get from walking the file tree and doing a stat on each file. Sometimes some of the "real" information matched what was found in some places of a record, while in other cases it was matching data from another place (I can't remember the exact details but I recall it had something to do with conflicting information in resident and non-resident attributes).

So basically after I spent a long time researching NTFS and the MFT and building the library, when I wanted to actually create an app using it I got stuck on the above two points. 😂

[joonas-fi]

Thanks for writing and your insights!

I acknowledge your warning, thank you for it, and I have to say that if I knew what I was getting into, I wouldn't do it again. The last week I've been pretty annoyed, running on sunken cost fallacy.. 😂

But my ignorance-fuelled expedition seems to have paid off. In two weeks I've managed to build high-level reading of NTFS volumes (directory scanning, MFT consistency scanner) and most importantly: I just generated my first NTFS filesystem that I was able to successfully mount on Linux (haven't tested Windows yet) and I was able to read the file from it.

I even have tests where I generate NTFS partition and use the high-level read side to validate that I can read what I tried to write. It found quite many issues that would've cost more to debug later on.

I think I'm not very far away from using NBD to synthetize a file tree as a NTFS partition which I can boot with a Windows VM. My NTFS code interfaces are already prepared for block-level read/write so I can generate the NTFS structures in-memory and mark which cluster ranges belong to actual files so they can be transparently read not-from-RAM but from Linux host-side's file tree.

I'm open sourcing this library, once I clean it up a little.

You're right on the money as far as undocumented things goes. https://flatcap.org/linux-ntfs/ntfs/ has been really valuable for me, along with your code, but some details I've had to dig into ntfs-3g code (Linux's NTFS implementation). One such example was that File Record's attributes have to be sorted in a very specific order: first by type, then by name (unnamed attributes first). I had an attribute that ntfs-3g didn't find, but that was because it expects to find the attribute in that exact order.. And same goes for Directory Indices, where files and subdirectories have to be sorted.

You can get quite far with the Flatcap docs, reading others' example code (like yours), parsing your own Windows's filesystem and looking at the rest from ntfs-3g source code.

WinDirStat / TreeSize are my favourites for visually seeing where the free space has gone! On Linux I've been pretty happy with ncdu.

So basically after I spent a long time researching NTFS and the MFT and building the library, when I wanted to actually create an app using it I got stuck on the above two points.

I very much feel that! :D Too often programming looks like this: https://www.youtube.com/watch?v=AbSehcT19u0

I've also done some research on Go's cross-platform UI toolkit space. I've used Fyne to some success, but it's not mature and it feels native only within Fyne (all Fyne apps look the same on all platforms).

I don't have any recommendations for your needs, but I'm thinking if web technologies could be the way forward even for locally-run apps. Something like Cordova.. You need a browser anyway, and there isn't a "native" concept as far as cross-platform goes, unless it's built on massive abstractions and that seems like a non-starter to me. Fyne and Redox's orbtk seem like fresh ways forward.

With Fyne I appreciate embracing the fact that native abstractions are hard, so the abstraction they built is basically each platform only needs to support a canvas, and what's written to that canvas doesn't concern the concrete platform like Windows / Linux. You lose the native feel though, both in good and bad.

[joonas-fi]

BTW do you want me to file any issues I've found (so far I've found 1-2) or is this repo mainly as an snapshot example code for people to learn from?

An example issue is when offsetLength is zero, it has special semantic meaning (= sparse file):

gomft/mft/mft.go

Line 365 in f64df39

offsetBytes := dataRunData.Read(lengthLength, offsetLength)

(source, see "Example 4 - Sparse, Unfragmented File")

This code decodes a sparse offset as 0, which would incorrectly read a fragment beginning from VBR.

I patched my code to mark sparse runs as having offset -1.

I think the other issue I had in mind is directly related to the above, where DataRunsToFragments needs to take sparse runs into account. Data run offsets are relative previous non-sparse run, not previous run:

gomft/mft/mft.go

Line 383 in f64df39

previousOffsetCluster := int64(0)

Natural consequence of sparse run support is supporting sparse fragments, which I again marked with fragment.Offset = -1. I had to re-write fragment.Reader because alternating between actual disk-backed fragments vs sparse fragments would've been hackish to add without a re-write.

[t9t]

Thanks. I might try Fyne again one day, it seems promising to an extent the last time I looked at it, but unfortunately I was running into some performance issues back then and decided to look into other options first.

I would appreciate it if you could file separate issues for your findings! :) That way I can track and properly address them when I get some time.

I must say that I found sparse files quite mystifying when building gomft initially, so it doesn't surprise me that I made some mistakes in that area. It might even explain some of the odd things I saw when testing gomft own my file system.

One big problem is the lack of good test cases, to see what happens in the real world and how for example the Windows or Linux NTFS drivers handle them. That way I can properly reproduce the issue in gomft, see what the proper solution is, implement that, and then see that the test case is fine.

[joonas-fi]

I filed an issue about the sparse runs with some inaccuracies fixed from my previous comment. :)

BTW my NTFS driver has partial write support (I can generate NTFS volumes with contents I want, but not edit them after the generation phase) - to such an extent that I can boot Windows XP, Windows 7 and Windows 10 from hard drive images 100 % generated from my code. I'll publish my repo soon-ish.

One thing I could mention which has helped me immensely is restruct. Like I said before, your library is great for reading NTFS structures, but I need to go in both directions.

Here's an example of code that can encode/decode $STANDARD_INFORMATION in both directions:

type StandardInformation struct {
	StandardInformationRequired
	// 48 bytes up to here (which is specified as minimum for this attribute)
	Extended *StandardInformationExtended
}

type StandardInformationRequired struct {
	TimeCreation    Time
	TimeAlteration  Time
	TimeMFTChange   Time
	TimeRead        Time
	Permissions     Flags
	MaximumVersions int32
	VersionNumber   int32
	ClassId         int32 // "Class Id from bidirectional Class Id index"
}

type StandardInformationExtended struct {
	OwnerId              int32
	SecurityId           int32
	QuotaCharged         int64
	UpdateSequenceNumber int64
}

func ParseStandardInformation(b []byte) (*StandardInformation, error) {
	if len(b) < 48 {
		return nil, fmt.Errorf("ParseStandardInformation: need at least 48 bytes; got %d", len(b))
	}

	info, err := func() (*StandardInformation, error) {
		switch len(b) {
		case 48: // without extended
			required := StandardInformationRequired{}
			if err := restruct.Unpack(b, binary.LittleEndian, &required); err != nil {
				return nil, fmt.Errorf("ParseStandardInformation: %w: %x", err, b)
			}

			return &StandardInformation{StandardInformationRequired: required}, nil
		case 48 + 24: // with extended
			info := &StandardInformation{}
			if err := restruct.Unpack(b, binary.LittleEndian, info); err != nil {
				return nil, fmt.Errorf("ParseStandardInformation: %w: %x", err, b)
			}

			return info, nil
		default:
			return nil, fmt.Errorf("ParseStandardInformation: weird length: %d", len(b))
		}
	}()
	if err != nil {
		return nil, err
	}

	if err := info.Permissions.AssertNoUnrecognizedFlags(); err != nil {
		return nil, fmt.Errorf("ParseStandardInformation: %w", err)
	}

	return info, nil
}

func (s *StandardInformation) Bytes() []byte {
	if s.Extended == nil { // restruct doesn't understand nil structs
		b, err := restruct.Pack(binary.LittleEndian, &s.StandardInformationRequired)
		if err != nil {
			panic(err)
		}

		return b
	} else {
		b, err := restruct.Pack(binary.LittleEndian, s)
		if err != nil {
			panic(err)
		}

		return b
	}
}

Some structs are simpler. The above was a bit more complex because there was shorter and loger variant. Here's $FILE_NAME with restruct:

type FileName struct {
	ParentDirectoryReference FileReference
	TimeCreation             Time
	TimeAlteration           Time
	TimeMFTChange            Time
	TimeRead                 Time
	SizeAllocated            int64 // [B] (of unnamed data stream) usually but not necessarily aligned to cluster size (resident data 64bit-aligned)
	SizeReal                 int64 // [B] (of unnamed data stream) on disk, can be 0 if it is resident data
	Flags                    Flags
	EAAndReparse             int32 // "Used by EAs and Reparse", although I didn't witness a symlink have data in this field
	FilenameLength           uint8 // [codepoint]
	FilenameNamespace        FilenameNamespace
	FilenameRaw              []byte `struct-while:"!_eof"`
}

func (f *FileName) Filename() string {
	return utf16le.Decode(f.FilenameRaw)
}

func (f *FileName) Bytes() []byte {
	b, err := restruct.Pack(binary.LittleEndian, f)
	if err != nil {
		panic(err)
	}

	return b
}

func ParseFileName(b []byte) (*FileName, error) {
	attr := &FileName{}
	if err := restruct.Unpack(b, binary.LittleEndian, attr); err != nil {
		return nil, fmt.Errorf("ParseFileName: %w", err)
	}

	// covers the case where there is extended data in FILE_NAME attr and
	// our struct marshaling mis-interpreted that
	if len(attr.FilenameRaw) != int(attr.FilenameLength)*2 {
		return nil, errors.New("ParseFileName: unexpected file name length")
	}

	if err := attr.FilenameNamespace.AssertKnownValue(); err != nil {
		return nil, fmt.Errorf("ParseFileName: %w", err)
	}

	if err := attr.Flags.AssertNoUnrecognizedFlags(); err != nil {
		return nil, fmt.Errorf("ParseFileName: %w", err)
	}

	return attr, nil
}