The support forum

Windows Insider: Cache manager BSOD

AndyMidd :

May 11, 2018

Since installing the most recent Windows Insider builds (17661, and 17666), Bvckup triggers a Stop error (BSOD) for Cache_Manager.
Every time.
BlueScreenView reports the crash is within the ntfs.sys driver.
Bvckup 78.18: Trying to test v79.
If Bvckup is auto started, there follows an auto BSOD.
Manually started, instant BSOD.
WIndows 10 Pro Insider preview 64bit.


AndyMidd :

May 11, 2018

same for v79. Pity!

Alex Pankratov :

May 11, 2018

Do you happen to have a stack trace or a minidump? That way we could at least work around this while Microsoft is fixing it properly.

AndyMidd :

May 11, 2018

yes. I have three Minidumps of the BSOD. How can I send you one/more of them? Can't attach here.
Andy


==================================================
Dump File         : 051118-12937-01.dmp
Crash Time        : 11/05/2018 12:48:01
Bug Check String  : CACHE_MANAGER
Bug Check Code    : 0x00000034
Parameter 1       : 00000000`00000273
Parameter 2       : ffffffff`c0000420
Parameter 3       : 00000000`00000000
Parameter 4       : 00000000`00000000
Caused By Driver  : Ntfs.sys
Caused By Address : Ntfs.sys+102543
File Description  : NT File System Driver
Product Name      : Microsoft® Windows® Operating System
Company           : Microsoft Corporation
File Version      : 10.0.17666.1000 (WinBuild.160101.0800)
Processor         : x64
Crash Address     : ntoskrnl.exe+19e120
Stack Address 1   :
Stack Address 2   :
Stack Address 3   :
Computer Name     :
Full Path         : C:\WINDOWS\Minidump\051118-12937-01.dmp
Processors Count  : 4
Major Version     : 15
Minor Version     : 17666
Dump File Size    : 197,125
Dump File Time    : 11/05/2018 12:48:48
==================================================

AndyMidd :

May 11, 2018

Minidump analysis: (winDbg) [ Any use? ]
Andy

BugCheck 34, {273, ffffffffc0000420, 0, 0}

Probably caused by : ntkrnlmp.exe ( nt!CcCopyReadEx+16169d )

Followup:     MachineOwner

---


nt!KeBugCheckEx:
fffff800`30e36120 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:fffffa05`af4df4e0=0000000000000034
0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

CACHE_MANAGER (34)
    See the comment for FAT_FILE_SYSTEM (0x23)
Arguments:
Arg1: 0000000000000273
Arg2: ffffffffc0000420
Arg3: 0000000000000000
Arg4: 0000000000000000

Debugging Details:

------------



KEY_VALUES_STRING: 1


TIMELINE_ANALYSIS: 1


DUMP_CLASS: 1

DUMP_QUALIFIER: 400

BUILD_VERSION_STRING:  10.0.17666.1000 (WinBuild.160101.0800)

SYSTEM_MANUFACTURER:  MSI

SYSTEM_PRODUCT_NAME:  MS-7816

SYSTEM_SKU:  To be filled by O.E.M.

SYSTEM_VERSION:  1.0

BIOS_VENDOR:  American Megatrends Inc.

BIOS_VERSION:  V1.11

BIOS_DATE:  05/09/2015

BASEBOARD_MANUFACTURER:  MSI

BASEBOARD_PRODUCT:  Z87-G43 (MS-7816)

BASEBOARD_VERSION:  1.0

DUMP_TYPE:  2

DUMP_FILE_ATTRIBUTES: 0xc
  Insufficient Dumpfile Size
  Kernel Generated Triage Dump

BUGCHECK_P1: 273

BUGCHECK_P2: ffffffffc0000420

BUGCHECK_P3: 0

BUGCHECK_P4: 0

EXCEPTION_RECORD:  ffffffffc0000420 -- (.exr 0xffffffffc0000420)
Cannot read Exception record @ ffffffffc0000420

CPU_COUNT: 4

CPU_MHZ: d48

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3c

CPU_STEPPING: 3

CPU_MICROCODE: 6,3c,3,0 (F,M,S,R)  SIG: 24'00000000 (cache) 24'00000000 (init)

BLACKBOXBSD: 1 (!blackboxbsd)


BLACKBOXPNP: 1 (!blackboxpnp)


CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  0x34

PROCESS_NAME:  bvckup2.exe

CURRENT_IRQL:  0

ANALYSIS_SESSION_HOST:  WIN10MSI

ANALYSIS_SESSION_TIME:  05-11-2018 20:20:00.0911

ANALYSIS_VERSION: 10.0.17074.1002 amd64fre

LAST_CONTROL_TRANSFER:  from fffff80030e7d5fd to fffff80030e36120

STACK_TEXT:  
fffffa05`af4df4d8 fffff800`30e7d5fd : 00000000`00000034 00000000`00000273 ffffffff`c0000420 00000000`00000000 : nt!KeBugCheckEx
fffffa05`af4df4e0 fffff800`31b02543 : fffffa05`af4df610 00000000`00000000 00000000`0000005c 00000000`0000fff0 : nt!CcCopyReadEx+0x16169d
fffffa05`af4df570 fffff80b`b6124de9 : 00000000`00000000 00000000`000002b0 ffff878e`ec38fef0 00000282`f0c672f0 : Ntfs!NtfsCopyReadA+0x273
fffffa05`af4df830 fffff80b`b6125407 : fffffa05`af4df920 ffff878e`ec38fe00 ffff878e`f15448d0 ffff878e`f15447d0 : FLTMGR!FltpPerformFastIoCall+0xd9
fffffa05`af4df890 fffff80b`b615aae9 : 00000000`00000001 00000000`00000000 000000c1`0ecff198 00000000`00000001 : FLTMGR!FltpPassThroughFastIo+0xc7
fffffa05`af4df8f0 fffff800`31141aad : ffff878e`ec38fef0 000000c1`0ecff200 00000282`f0c76000 ffff878e`ec38fef0 : FLTMGR!FltpFastIoRead+0x159
fffffa05`af4df990 fffff800`30e466b3 : 00000000`00000000 00000000`00000000 00000000`00000000 000000c1`0ecff230 : nt!NtReadFile+0x43d
fffffa05`af4dfa90 00007ff9`eb639764 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
000000c1`0ecff178 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`eb639764


THREAD_SHA1_HASH_MOD_FUNC:  1983e1027d36de55a08bb89d5c014b2e733b3e6e

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  d5638053cebebc467e0b2d52355257d7b8f77ca8

THREAD_SHA1_HASH_MOD:  a2cae9a944c77203b78068fe0901f9a0f77d72a7

FOLLOWUP_IP:
nt!CcCopyReadEx+16169d
fffff800`30e7d5fd 90              nop

FAULT_INSTR_CODE:  c0334590

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  nt!CcCopyReadEx+16169d

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

IMAGE_NAME:  ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  0

IMAGE_VERSION:  10.0.17666.1000

STACK_COMMAND:  .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET:  16169d

FAILURE_BUCKET_ID:  0x34_nt!CcCopyReadEx

BUCKET_ID:  0x34_nt!CcCopyReadEx

PRIMARY_PROBLEM_CLASS:  0x34_nt!CcCopyReadEx

TARGET_TIME:  2018-05-11T11:48:01.000Z

OSBUILD:  17666

OSSERVICEPACK:  1000

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  unknown_date

BUILDDATESTAMP_STR:  160101.0800

BUILDLAB_STR:  WinBuild

BUILDOSVER_STR:  10.0.17666.1000

ANALYSIS_SESSION_ELAPSED_TIME:  b16

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x34_nt!cccopyreadex

FAILURE_ID_HASH:  {c9ff34a1-ed3e-29ff-4c5e-e293cb19d05e}

Followup:     MachineOwner

---


stack:
nt!KeBugCheckEx
nt!CcCopyReadEx + 0x16169d
Ntfs!NtfsCopyReadA + 0x273
FLTMGR!FltpPerformFastIoCall + 0xd9
FLTMGR!FltpPassThroughFastIo + 0xc7
FLTMGR!FltpFastIoRead + 0x159
nt!NtReadFile + 0x43d
nt!KiSystemServiceCopyEnd + 0x13
0x7ff9eb639764

WiseOldElf :

May 12, 2018

it's looking like the same problem here too.  win 10 pro.  i can get win 10 to start in safe mode and bvckup2 doesn't run there as it's set up to run as a service.  to help me exclude bvckup2 as (part of) the problem, how can i stop it running as a service - is there a file that i can tweak when i'm in safe mode ?  then i could start normally and run it manually to see if i still bsod.

Alex Pankratov :

May 12, 2018

@AndyMidd -

I have three Minidumps of the BSOD. How can I send you one/more of them?


Zip'em up please and forward to support@pipemetrics.com

However I got a minidump from another person and there's not much actionable info in it. In both cases (theirs and yours) the crash point is the same and the call stack doesn't extend far enough into bvckup2.exe itself to provide any insights. They both are for AMD systems though, so there's that.

@WiseOldElf - once in safe mode you should be able to start services.msc (press Windows-R, enter services.msc, press Enter) and disable the bvckup2 service there.

WiseOldElf :

May 13, 2018

sorry, i wasn't clear - i knew how to disable the service, but i should've asked, 'is there a file that i can tweak when i'm in safe mode, so that next time i restart, bvckup2 is running in desktop mode, not service mode' ?

Alex Pankratov :

May 13, 2018

Ah, OK. Do this:

Once in safe mode, open Admin command prompt (via Windows-X, A). Go into the folder where bvckup2.exe is and then run:

    bvckup2.exe --switch-to-app -f --console %LocalAppData%\Bvckup2

You should see a separate console window open with the activity log that should conclude with "Completed with 0".

---

If you are to experiment with launching bvckup2 after that, please take a note if the BSOD is triggered by merely launching the program or if it happens only when a backup is run. Also if it's the latter, then please stare at the UI very hard and see what't the last thing the engine appears to be doing before BSOD happens.

At this point I don't even know what it is that ultimately ends up calling KiSystemServiceCopyEnd, because it's certainly not something that any of the code does explicitly :-|

WiseOldElf :

May 13, 2018

some problems with this:
1. got a message 'bvckup2.exe: the term 'bvckup2.exe is not recognized as the name of a cmdlet, function, script file, or operable program..."
2. ran $env:Path = $env:Path+"."
then ran as:
".\bvckup2.exe --switch-to-app -f --console %LocalAppData%\Bvckup2"
3. got a message to say that it couldn't find the folder Bvckup2, so changed it to:
bvckup2.exe --switch-to-app -f --console "c:\program files\bvckup 2"
4. that ran, stopped the service, said it was moving the engine configuration, then said 'fixing up security descriptors...' and it's been stuck there ever since

Alex Pankratov :

May 13, 2018

Oh, my bad. On Windows 10 they changed "Win-X, A" to start the PowerShell instead of the regular command prompt... so it needed a ("cmd.exe" + Enter) right after opening shell window.

3. got a message to say that it couldn't find the folder Bvckup2, so changed it to: bvckup2.exe --switch-to-app -f --console "c:\program files\bvckup 2"


It should've been  "c:\users\wiseoldelf\AppData\Local\bvckup 2" or similar, but the %LocalAppData% notation works fine in the plain shell.

WiseOldElf :

May 14, 2018

hmm, this was interesting.
1. ran the script from command prompt; all good.  messages indicated it had been changed to app mode
2. went back to start menu to run bvckup2 manually, *but* it told me it was still set up as a service, and did i want to start that service. i rebooted the pc instead.
3. went to start menu to run bvckup2 manually.  i had to re-licence it (odd ?) and all of my backup plans had gone.  that's strange, because they have persisted when i have switched service->app and app-> service previously.
4. added a c:\temp -> c:\temp2 test and ran it manually - all good
5. changed bvckup2 back to run as a service - all good
6. selected a backup plan and ran that manually - all good

so, it appears that there was some issue with bvckup2 being a service at the time that one of the previous windows 10 updates was applied, but that by switching to app mode and then back to service mode, that problem's overcome.  i'll monitor it, of course, as i'll need to re-define all of my backup plans anyway, but so far, so good.  
not sure if that helps you, Alex !

AndyMidd :

May 14, 2018

Could the non-paged pool memory be eaten up?
See: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/bug-check-0x34--cache-manager

not sure why that would be...!

AndyMidd :

May 14, 2018

Using Process Explorer, I see my Non-paged pool (normal conditions) is about 8GB, of which about 25% is used. So I doubt if Bvckup can leak that much, so quick. (I have been reading this: https://blogs.technet.microsoft.com/markrussinovich/2009/03/10/pushing-the-limits-of-windows-paged-and-nonpaged-pool/)
Without starting Bvckup, is it possible to temporarily amend the "tasks" (to see if it the core program, or the actions involved in writing to a NAS that trigger the fault). e.g. registry. Or a startup option e.g. "command unload *"?

Alex Pankratov :

May 14, 2018

Could the non-paged pool memory be eaten up?


Very unlikely. At least not by bvckup2.exe itself. R78 releases don't do any explicit memory locking whatsoever. R79 does (see last section in https://bvckup2.com/wip/19042018), but it's still in a tune of megabytes at most _and_ these locks are removed by Windows once files are closed after copying.

Without starting Bvckup, is it possible to temporarily amend the "tasks"


Yep, good idea and a good thing to check.

Try this - go into %LocalAppData%\Bvckup2\engine, then descend into each backup-00xx folder, open settings.ini and change "conf.disabled" from 0 to 1. This will, as you can probably guess, disables backup jobs, so none of them runs at launch.

Alex Pankratov :

May 14, 2018

@WiseOldElf - thanks for the follow-up.

Your original config is still on the disk, it just got stashed in a backup folder when you did you original --switch-to-app with a ProgramData path. Check in %ProgramData%\Bvckup2 for engine_xxx folders and also check for the same in %LocalAppData%\Bvckup2. If you find any, that's your original config, so shut down the app (including stopping the service), move aside 'engine' folder and rename this engine_xxx to 'engine'. This should help, but there are still some edge cases...

AndyMidd :

May 14, 2018

I disabled both tasks as described. But it still continues to BSOD for cache_manager: looks like the core.

WiseOldElf :

May 15, 2018

didn't find any backed-up config, so set it up again (bvckup2 in app mode).  the backup is from a local drive to an internal hard drive.  nb: it shows as one drive, but it's actually two that are: spanned, dynamic, ntfs, 8TB each.  so, it started.  and BSOD.  

i changed to backup to an external hard drive - all fine.

i will do more investigation tonight, but have microsoft changed the way they work with spanned drives ?

WiseOldElf :

May 15, 2018

still in app mode, but something weird now - the first time, it ran without issues copying to an external drive (although i pressed the 'stop' button after about 4 or 5 minutes of copying).  then i 'disabled' the backup, deleted everything in the target folder, 'enabled' it and ran it again  - BSOD.  restarted the machine,  bvckup2 ui popped up, but then immediate BSOD.

Alex Pankratov :

May 15, 2018



Gentlemen, at the risk of stating the obvious - it's a bug in the cache manager and the only proper fix belongs there. However we can try and find a workaround.

To that end, can you please give this version a try?
https://bvckup2.com/files/bvckup2-setup-1.79.0.4-upxless.exe


AndyMidd :

May 15, 2018

I disabled both tasks as described. But it still continues to BSOD for cache_manager: looks like the core.

AndyMidd :

May 15, 2018

79.0.4 also caused the same BSOD.

Alex Pankratov :

May 15, 2018

Yes, I figured as much. Can you please try the "upxless" build from my previous post? I want to check if it's related to bvckup2.exe being packed with UPX.

Alex Pankratov :

May 15, 2018

OK. Not much we can do then, unfortunately, but to wait for MS to fix it.

AndyMidd :

May 15, 2018

sorry for misunderstanding: That was the upxless version I tried, from the link.

AndyMidd :

May 15, 2018

I have raised an insider problem report ("GSOD cache manager"). Maybe the sun will shine....

WiseOldElf :

May 15, 2018

well here's a coincidence for you; strangely recent too:
https://support.microsoft.com/en-us/help/304101/backup-program-is-unsuccessful-when-you-back-up-a-large-system-volume

Alex Pankratov :

May 15, 2018

That's for Windows Server 2003 series though.

WiseOldElf :

May 21, 2018

yeah, saw that - just thought it was quite a coincidence.

any insight into this problem or when they might resolve it ?

Alex Pankratov :

May 21, 2018

No idea, sorry. We have no insight into this whatsoever.

WiseOldElf :

May 26, 2018

i am puzzled though, that i don't see  any other reporting of BSOD issues with cache manager.  i'm assuming that you're unable to replicate it, alex - is there anything further that we're able to do to help ?

New topic

Create
Made by Pipemetrics in Switzerland
Support


Follow
Twitter
Blog / RSS
Miscellanea Press resources
Testimonials
On robocopy
Company
Imprint

Legal Terms
Privacy