Fragmented video recovery challenges
This page is about a MP4, MOV recovery tool I am developing, and which is in it’s infant stage. No need to ask me for the tool, I will release a beta once it is ready to be released as a beta. This may be on one month or in 6 months, I simply can’t tell.
Modern cameras don’t just record video and write it sequentially to a SD or CF memory card. At the same time we see that these cards rely on exFAT file system and that in many data loss scenarios we can not rely on FAT chains being present to determine what clusters were assigned to the video file we’re trying to recover.
We used to “carve” such files: We’d find the start of the file and depending on the IQ of the carver we’d sort of guess file size (“Oh, found new file start, we should close previous file then”) or some slightly smarter tools interpret box headers, add them up, and determine size this way. With modern cameras this no longer is a viable method and I’ll try explain why.
1 – Fragmentation guaranteed
Modern cameras often “interleave” hi-res video data with lo-res video data. As it records video is saves a hi and a lo res video stream simultaneously and since it does not know in advance how much data it needs to save and buffers are finite, it needs to flush bursts of data to the memory card. So what get is blocks <hi-res><lo-res><hi-res><lo-res><hi-res><lo-res> etc. where <hi-res> blocks belong to one file, and the <lo-res> blocks to another.
As long as there’s some form of FAT or file Bitmap available this fragmentation is not an issue, but what common data loss scenarios (file deletion, accidental format) have in common is that we can not rely on such on-disk structures.
When I initially started look into this, I thought this was the issue I’d have to tackle but I was wrong..
2 – Out of order boxes / atoms
MP4 video files are divided into sections we call boxes or atoms.
If we look at “normal” MP4 video file see either [ftyp][moov][mdat] or [ftyp][mdat][moov] boxes.
[ftyp]we could refer to as a header of sorts[moov]is an index, a table of offsets to video data samples[mdat]is the actual video data
Theoretically it does not really matter in what order these are but usually we first find ftyp and then the mdat and moov boxes that are part of one and the same file.
And eventually video files created by modern cameras follow this convention IF and as long we can rely on some sort of file bitmap to assemble the file in the correct order. Without this bitmap we often see something different on-disk: <header-less and interleaved video data>[ftyp][moov][truncated mdat box]
So we can see how convectional carving is bound to fail. Assuming next file follows our on-disk, out-of-order file, a conventional carver is likely to combine the [ftyp][moov][truncated mdat] with the <header-less and interleaved video data> of the next file. But even more likely it will be confused by the [ftyp] of the lo-res video.
Summary
As demonstrated a carver needs to become smarter to to be able recover video recorded with modern action cameras, but also DSLR and System or Compact cameras. Not only does it need to be able to recover video data that’s fragmented due to interleaved <hi-res> and <lo-res> video “streams”, it also has to link these video streams to the corresponding [ftyp] and [moov] boxes and write the recovered file out in the correct order so that offsets within the [moov] box reference the correct data in the [mdat] box.
This is a video based on same ftyp detected at same LBA (look at filenames which are based on LBA address) and was recovered from an SD Card used in a “DJI 4 Pro”:
The left file icon is file recovered by a simple carver (I used DMDE because the filename reflects the LBA at which the file was found). Not only is it 10x too large, it does not play. The file on the right is recovered with my experimental smart MP4 carver. Because I try to make the tool as generic as possible, it does not have to be “trained” to support specific camera models unlike we see in “Disk Drill’s Advanced Camera Recovery” or “GoProRecovery”. In fact, today was the first time I tested against this DJI disk image and it simple just worked.
About the recovery tool (Atom-Forge?)
Currently tools goes through 3 scan phases:
- Cluster size scan: Because cluster or block size is sort of vital as cluster > sector size greatly reduces workload it’s worth to sacrifice some time determining this accurately. It is also vital as fragmentation happens at cluster level: if we find incorrect data at LBA x, then entire cluster in which LBA x is, is likely incorrect and other way around too.
- Scan for ftyp boxes + quick scan for remaining boxes. Quick scan for remaining boxes looks for boxes until it reached value as set by proximity setting. Default is 1%, depending on drive and cluster size this seems a nice balance between succes-rate and speed.
- Final scan for missing boxes but doubles value as set in proximity setting.
So, once scan finished we know clustersize and we have connected all boxes to specific files. Also scan determined whether boxes are out of order on-disk (example: <header-less and interleaved video data>[ftyp][moov][truncated mdat box]).
Only if “defrag mdat’ is set, the tool will make an effort to defrag mdat data. So some of the heavy lifting is done during the actual file recover phase. I am still tweaking this as it spends to much time on hopeless situations at this point.
What the tool can’t do
It’s not a repair tool! To produce a playable video it must be able to locate all required boxes. If for whatever reason the moov box or the ftyp box can’t be found, it will not recover a playable video unlike a tool that does so called “header-stealing” and/or generates a moov box in an untrunc manner (Klennet Carver employs these techniques). Perhaps this can be added as a future enhancement.
What it also can’t do is reassemble .insv files produced by insta360 cameras unlike Disk Drill’s Advanced Camera Recovery. My tool may be able to recover the lost video as MP4 and you may be able to reconstruct .insv files using the free Insta360 File Repair software.