libsmdev_handle_read_buffer: offset exceeds media size.

[devgc]

I think this is the right repository for this issue. I was having an issue in DFVFS where volume shadows are not being opened for some logical volumes and think I have narrowed it down to this library. Here is the exact pysmdev error as shown in the following example (Python 2.7 and 3.6 both produce it):

pyvshadow_volume_open_file_object: unable to open volume. pyvshadow_file_object_read_buffer: unable to read from file object with error: 'pysmdev_handle_read_buffer: unable to read data. libsmdev_handle_read_buffer: offset exceeds media size.'.
 pyvshadow_file_object_io_handle_read: unable to read from file object.
 libbfio_handle_read_buffer: unable to read from handle.
 libvshadow_io_handle_read_ntfs_volume_header: unable to read NTFS backup volume header data.
 libvshadow_volume_open_read: unable to read NTFS volume header.
 libvshadow_volume_open_file_io_handle: unable to read from file IO handle.

with libsmdev_handle_read_buffer: offset exceeds media size. being the issue.

This is the same error I am seeing when trying the open the VShadow object in dfvfs.

I understand this will be difficult to reproduce because it is specific to my logical volume and I have not found a test image where the resulting error is reproduced. Maybe you can formulate a hypothesis by seeing what I did, and what the output was?

Whats weird is it works on my D: but not C:.
C: is on a SSD with bitlocker turned on.
D: is on a HD with bitlocker turned on.
More fs and drive info can be found at the bottom of this comment.

Below is the code I used to produce the error followed by the results of it being ran on my system:

import pysmdev
import pyvshadow


def main():
    ######################################################
    # using normal file handle
    ######################################################
    print("Test 1: VShadow from C as file handle")
    with open('\\\\.\\C:', "rb") as volume_handle:
        vshadow_volume = pyvshadow.volume()
        try:
            vshadow_volume.open_file_object(
                volume_handle
            )
            print("VSS Found for C: Store count -> {}".format(
                vshadow_volume.get_number_of_stores()
            ))
        except Exception as error:
            error_str = str(error).replace("\r", "\n")
            print("Error Opening VShadow using <file> on [C:]")
            print(error_str)
    print("\n")

    print("Test 2: VShadow from D as file handle")
    # As regular file.
    with open('\\\\.\\D:', "rb") as volume_handle:
        vshadow_volume = pyvshadow.volume()
        try:
            vshadow_volume.open_file_object(
                volume_handle
            )
            print("VSS Found for D: Store count -> {}".format(
                vshadow_volume.get_number_of_stores()
            ))
        except Exception as error:
            error_str = str(error).replace("\r", "\n")
            print("Error Opening VShadow using <file> on [D:]")
            print(error_str)
    print("\n")

    ######################################################
    # using pysmdev handle
    ######################################################
    print("Test 3: VShadow from C as pymsdev.handle")
    smdev_handle = pysmdev.handle()
    smdev_handle.open("\\\\.\\C:")

    vshadow_volume = pyvshadow.volume()
    try:
        vshadow_volume.open_file_object(
            smdev_handle
        )
        print("VSS Found for C: Store count -> {}".format(
            vshadow_volume.get_number_of_stores()
        ))
    except Exception as error:
        error_str = str(error).replace("\r", "\n")
        print("Error Opening VShadow using <pysmdev.handle> on [C:]")
        print(error_str)
    print("\n")

    print("Test 4: VShadow from D as pymsdev.handle")
    smdev_handle = pysmdev.handle()
    smdev_handle.open("\\\\.\\D:")

    vshadow_volume = pyvshadow.volume()
    try:
        vshadow_volume.open_file_object(
            smdev_handle
        )
        print("VSS Found for D: Store count -> {}".format(
            vshadow_volume.get_number_of_stores()
        ))
    except Exception as error:
        error_str = str(error).replace("\r", "\n")
        print("Error Opening VShadow using <pysmdev.handle> on [D:]")
        print(error_str)
    print("\n")


if __name__ == "__main__":
    main()

Test 1: VShadow from C as file handle
VSS Found for C: Store count -> 11


Test 2: VShadow from D as file handle
VSS Found for D: Store count -> 11


Test 3: VShadow from C as pymsdev.handle
Error Opening VShadow using <pysmdev.handle> on [C:]
pyvshadow_volume_open_file_object: unable to open volume. pyvshadow_file_object_read_buffer: unable to read from file object with error: 'pysmdev_handle_read_buffer: unable to read data. libsmdev_handle_read_buffer: offset exceeds media size.'.
 pyvshadow_file_object_io_handle_read: unable to read from file object.
 libbfio_handle_read_buffer: unable to read from handle.
 libvshadow_io_handle_read_ntfs_volume_header: unable to read NTFS backup volume header data.
 libvshadow_volume_open_read: unable to read NTFS volume header.
 libvshadow_volume_open_file_io_handle: unable to read from file IO handle.


Test 4: VShadow from D as pymsdev.handle
VSS Found for D: Store count -> 11

Python 2.6 x64 - Library Versions:

>>> import pysmdev
>>> pysmdev.get_version()
u'20171112'
>>> import pyvshadow
>>> pyvshadow.get_version()
u'20170902'

Python 3.6 x64 - Library Versions:

>>> import pysmdev
>>> pysmdev.get_version()
'20171112'
>>> import pyvshadow
>>> pyvshadow.get_version()
'20170902'

Not sure if the following helps or not. But I thought I would include it just in case.

C: Volume info:

NTFS Version   :                   3.1
LFS Version    :                   2.0
Number Sectors :                   0x0000000037c60f44
Total Clusters :                   0x0000000006f8c1e8
Free Clusters  :                   0x00000000050b7fa9
Total Reserved :                   0x000000000000233f
Bytes Per Sector  :                512
Bytes Per Physical Sector :        512
Bytes Per Cluster :                4096
Bytes Per FileRecord Segment    :  1024
Clusters Per FileRecord Segment :  0
Mft Valid Data Length :            0x000000005b140000
Mft Start Lcn  :                   0x00000000000c0000
Mft2 Start Lcn :                   0x0000000000000002
Mft Zone Start :                   0x0000000003729ce0
Mft Zone End   :                   0x0000000003736500
Max Device Trim Extent Count :     0
Max Device Trim Byte Count :       0x0
Max Volume Trim Extent Count :     62
Max Volume Trim Byte Count :       0x40000000

---DRIVE INFO---
PSComputerName              : DESKTOP
ConfigManagerErrorCode      : 0
LastErrorCode               :
NeedsCleaning               :
Status                      : OK
DeviceID                    : \\.\PHYSICALDRIVE0
StatusInfo                  :
Partitions                  : 3
BytesPerSector              : 512
ConfigManagerUserConfig     : False
DefaultBlockSize            :
Index                       : 0
InstallDate                 :
InterfaceType               : SCSI
MaxBlockSize                :
MaxMediaSize                :
MinBlockSize                :
NumberOfMediaSupported      :
SectorsPerTrack             : 63
Size                        : 480101368320
TotalCylinders              : 58369
TotalHeads                  : 255
TotalSectors                : 937697985
TotalTracks                 : 14884095
TracksPerCylinder           : 255
__GENUS                     : 2
__CLASS                     : Win32_DiskDrive
__SUPERCLASS                : CIM_DiskDrive
__DYNASTY                   : CIM_ManagedSystemElement
__RELPATH                   : Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE0"
__PROPERTY_COUNT            : 51
__DERIVATION                : {CIM_DiskDrive, CIM_MediaAccessDevice, CIM_LogicalDevice, CIM_LogicalElement...}
__SERVER                    : DESKTOP
__NAMESPACE                 : root\cimv2
__PATH                      : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE0"
Availability                :
Capabilities                : {3, 4}
CapabilityDescriptions      : {Random Access, Supports Writing}
Caption                     : ATA INTEL SSDSC2BP48 SCSI Disk Device
CompressionMethod           :
CreationClassName           : Win32_DiskDrive
Description                 : Disk drive
ErrorCleared                :
ErrorDescription            :
ErrorMethodology            :
FirmwareRevision            : 0420
Manufacturer                : (Standard disk drives)
MediaLoaded                 : True
MediaType                   : Fixed hard disk media
Model                       : ATA INTEL SSDSC2BP48 SCSI Disk Device
Name                        : \\.\PHYSICALDRIVE0
PNPDeviceID                 : SCSI\DISK&VEN_ATA&PROD_INTEL_SSDSC2BP48\5&ADA8C50&0&000000
PowerManagementCapabilities :
PowerManagementSupported    :
SCSIBus                     : 0
SCSILogicalUnit             : 0
SCSIPort                    : 0
SCSITargetId                : 0
SerialNumber                : x
Signature                   : 809290414
SystemCreationClassName     : Win32_ComputerSystem
SystemName                  : DESKTOP
Scope                       : System.Management.ManagementScope
Path                        : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE0"
Options                     : System.Management.ObjectGetOptions
ClassPath                   : \\DESKTOP\root\cimv2:Win32_DiskDrive
Properties                  : {Availability, BytesPerSector, Capabilities, CapabilityDescriptions...}
SystemProperties            : {__GENUS, __CLASS, __SUPERCLASS, __DYNASTY...}
Qualifiers                  : {dynamic, Locale, provider, UUID}
Site                        :
Container                   :

D: Volume info:

NTFS Volume Serial Number :        0xaad2d584d2d55563
NTFS Version   :                   3.1
LFS Version    :                   2.0
Number Sectors :                   0x00000000e8e077ff
Total Clusters :                   0x000000001d1c0eff
Free Clusters  :                   0x000000001a528d1d
Total Reserved :                   0x00000000000013ef
Bytes Per Sector  :                512
Bytes Per Physical Sector :        512
Bytes Per Cluster :                4096
Bytes Per FileRecord Segment    :  1024
Clusters Per FileRecord Segment :  0
Mft Valid Data Length :            0x0000000008d80000
Mft Start Lcn  :                   0x00000000000c0000
Mft2 Start Lcn :                   0x0000000000000002
Mft Zone Start :                   0x00000000000c8d80
Mft Zone End   :                   0x00000000000cc840
Max Device Trim Extent Count :     0
Max Device Trim Byte Count :       0x0
Max Volume Trim Extent Count :     62
Max Volume Trim Byte Count :       0x40000000

---DRIVE INFO---
PSComputerName              : DESKTOP
ConfigManagerErrorCode      : 0
LastErrorCode               :
NeedsCleaning               :
Status                      : OK
DeviceID                    : \\.\PHYSICALDRIVE1
StatusInfo                  :
Partitions                  : 1
BytesPerSector              : 512
ConfigManagerUserConfig     : False
DefaultBlockSize            :
Index                       : 1
InstallDate                 :
InterfaceType               : SCSI
MaxBlockSize                :
MaxMediaSize                :
MinBlockSize                :
NumberOfMediaSupported      :
SectorsPerTrack             : 63
Size                        : 2000396321280
TotalCylinders              : 243201
TotalHeads                  : 255
TotalSectors                : 3907024065
TotalTracks                 : 62016255
TracksPerCylinder           : 255
__GENUS                     : 2
__CLASS                     : Win32_DiskDrive
__SUPERCLASS                : CIM_DiskDrive
__DYNASTY                   : CIM_ManagedSystemElement
__RELPATH                   : Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE1"
__PROPERTY_COUNT            : 51
__DERIVATION                : {CIM_DiskDrive, CIM_MediaAccessDevice, CIM_LogicalDevice, CIM_LogicalElement...}
__SERVER                    : DESKTOP
__NAMESPACE                 : root\cimv2
__PATH                      : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE1"
Availability                :
Capabilities                : {3, 4}
CapabilityDescriptions      : {Random Access, Supports Writing}
Caption                     : ATA WDC WD2003FZEX-0 SCSI Disk Device
CompressionMethod           :
CreationClassName           : Win32_DiskDrive
Description                 : Disk drive
ErrorCleared                :
ErrorDescription            :
ErrorMethodology            :
FirmwareRevision            : 1A01
Manufacturer                : (Standard disk drives)
MediaLoaded                 : True
MediaType                   : Fixed hard disk media
Model                       : ATA WDC WD2003FZEX-0 SCSI Disk Device
Name                        : \\.\PHYSICALDRIVE1
PNPDeviceID                 : SCSI\DISK&VEN_ATA&PROD_WDC_WD2003FZEX-0\5&ADA8C50&0&000100
PowerManagementCapabilities :
PowerManagementSupported    :
SCSIBus                     : 0
SCSILogicalUnit             : 0
SCSIPort                    : 0
SCSITargetId                : 1
SerialNumber                :      xxxxxxxxxxxxxxxx
Signature                   : 1619285509
SystemCreationClassName     : Win32_ComputerSystem
SystemName                  : DESKTOP
Scope                       : System.Management.ManagementScope
Path                        : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE1"
Options                     : System.Management.ObjectGetOptions
ClassPath                   : \\DESKTOP\root\cimv2:Win32_DiskDrive
Properties                  : {Availability, BytesPerSector, Capabilities, CapabilityDescriptions...}
SystemProperties            : {__GENUS, __CLASS, __SUPERCLASS, __DYNASTY...}
Qualifiers                  : {dynamic, Locale, provider, UUID}
Site                        :
Container                   👍 

[joachimmetz]

Could this be related to: libyal/libcfile#2

is the offset of the NTFS backup header pointing beyond the volume device size reported by pysmdev?

Does the same happen when reading the full disk with pysmdev?

[joachimmetz]

I'll have a closer look as soon as time permits.

[devgc]

I created an image of the logical volume C: and I was able to open the volume shadow object. Maybe this gives more support for this issue having to do with libyal/libcfile#2?

Otherwise, here are some more offsets. The backup header is within the size that the smdev handle sees (smdev_handle.media_size).

Here are the offsets of my C:

Partition Size:          479092181504
smdev.handle.media_size: 479092181504
File System Size:        479092178944
Backup Header location:  479092180992 [479092178944 + 2048]

Here are the offsets of my D:

Partition Size:          2000396746752
smdev.handle.media_size: 2000396746752
File System Size:        2000396742656
Backup Header:           2000396746240 [2000396742656 + 3584]

In either case, the backup volume is the last block in partition.

I tried reading the last block using pysmdev but had no success.

>>> smdev_handle = pysmdev.handle()
>>> smdev_handle.open("\\\\.\\C:")
>>> backup_location = smdev_handle.media_size - 512
>>> print("backup_location: " + str(backup_location))
backup_location: 479092180992
>>> smdev_handle.seek_offset(backup_location)
>>> backup_raw = smdev_handle.read(512)
>>> print(len(backup_raw))
0

>>> smdev_handle = pysmdev.handle()
>>> smdev_handle.open("\\\\.\\D:")
>>> backup_location = smdev_handle.media_size - 512
>>> print("backup_location: " + str(backup_location))
backup_location: 2000396746240
>>> smdev_handle.seek(backup_location)
>>> backup_raw = smdev_handle.read(512)
>>> print(len(backup_raw))
0

I cant seem to read outside the file system size with pysmdev. But, this cant be the issue though because I am still able to open the vshadow object for D: and its backup is outside the file system size too.