Free version: Failed to allocate memory

[SarahC]

I'm getting the following error making the textured mesh from a high detail point cloud - produced by a Canon 7D mark II with an 18mm lens, at f/11. 26 images.

My GTX 970 GPU meter shows my free card memory is at 3.5GB, using only 500MB. (14%) It's running cold.
My RAM usage is 18GB with 14GB free.


I've included the models (800MB, 1.2GB unzipped, 20 mins download):

http://untamed.co.uk/miscFolder/Bins2.zip

http://untamed.co.uk/miscFolder/bin_images.zip

"Textured mesh 3" is the one where the problem occurred - the detail is exquisite, even though there was some wind moving the leaves.
The failed texture loading can be seen in the bushes at the right of the bins, and on the front of the green bin.



[04:04:12] Error in computing Undistorted image: [instance(0,0,0,0,0000000000000000,non-shared)] CImg<float>::CImg(): Failed to allocate memory (34.8 Mio) for image (2500,3648,1,1).
[04:04:12] Unable to load '_89a8812.jpg' file!
[04:04:13] Unable to load '_89a8834.jpg' file!
[04:04:13] Error in computing Undistorted image: [instance(0,0,0,0,0000000000000000,non-shared)] CImg<float>::CImg(): Failed to allocate memory (15.9 Mio) for image (2500,1667,1,1).
[04:04:13] Unable to load '_89a8815.jpg' file!
[04:04:13] Preparing image : '_89a8816.jpg'
[04:04:14] Unable to load '_89a8816.jpg' file!
[04:04:14] Preparing image : '_89a8820.jpg'
[04:04:16] +++ ERROR +++ (computation error)
[04:04:17] +++ ERROR +++ (computation error)
[04:04:17] Done.
[04:04:21] Computing blurriness...
[04:05:31] Done.
[04:05:31] Applying photoconsistency score...
[04:05:31] Done.
[04:05:32] Building connectivity...
[04:05:35] Done.


My detailed system configuration:

Summary
Operating System
Windows 10 Pro 64-bit

CPURAM
32.0GB Dual-Channel DDR3 @ 665MHz (9-9-9-24)

Motherboard
Dell Inc. 0HY9JP (CPU 1)

GraphicsStorageOptical Drives
HL-DT-ST DVD+-RW GH70N
ELBY CLONEDRIVE SCSI CdRom Device

Audio
High Definition Audio Device

Operating System
Windows 10 Pro 64-bit
Computer type: Tablet
Installation Date: 1/2/18 03:40:00
Serial Number: TY4CG............

Windows Security Center
User Account Control (UAC) Enabled
Notify level 2 - Default
Firewall Enabled
Windows Update
AutoUpdate Not configured
Windows Defender
Windows Defender Enabled
Antivirus
Windows Defender
Antivirus Enabled
Virus Signature Database Up to date
Sophos Anti-Virus
Antivirus Disabled
Virus Signature Database Up to date

.NET Frameworks installed
v4.7 Full
v4.7 Client
v3.5 SP1
v3.0 SP2
v2.0 SP2

Internet Explorer
Version 11.248.16299.0

PowerShell
Version 5.1.16299.15

Java
Java Runtime Environment
Path C:\Program Files (x86)\Java\jre1.8.0_161\bin\java.exe
Version 8.0
Update 161
Build 12

Environment Variables
USERPROFILE C:\Users\sarah
SystemRoot C:\WINDOWS

User Variables
OneDrive F:\sarah\OneDrive
Path C:\Users\sarah\AppData\Roaming\npm
C:\Users\sarah\AppData\Local\Microsoft\WindowsApps
TEMP C:\Users\sarah\AppData\Local\Temp
TMP C:\Users\sarah\AppData\Local\Temp

Machine Variables
ADSK_3DSMAX_x64_2017 D:\Program Files\Autodesk\3ds Max 2017\
ADSK_CLM_WPAD_PROXY_CHECK FALSE
ComSpec C:\WINDOWS\system32\cmd.exe
NUMBER_OF_PROCESSORS 8
OS Windows_NT
Path C:\ProgramData\Oracle\Java\javapath
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
D:\Program Files\Terminals\Putty\
D:\Program Files\Terminals\WinSCP\
D:\Program Files\SrcCtl\Git\cmd
C:\Program Files\nodejs\
C:\Program Files (x86)\Common Files\Acronis\VirtualFile\
C:\Program Files (x86)\Common Files\Acronis\VirtualFile64\
C:\Program Files (x86)\Common Files\Acronis\SnapAPI\
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
C:\Program Files (x86)\QuickTime\QTSystem\
C:\Program Files\Common Files\Autodesk Shared\
C:\Program Files (x86)\Autodesk\Backburner\
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files (x86)\Calibre2\
PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;. MSC
PROCESSOR_ARCHITECTURE AMD64
PROCESSOR_IDENTIFIER Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
PROCESSOR_LEVEL 6
PROCESSOR_REVISION 2a07
PSModulePath %ProgramFiles%\WindowsPowerShell\Modules
C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules
C:\Program Files\Intel\Wired Networking\
TEMP C:\WINDOWS\TEMP
TMP C:\WINDOWS\TEMP
USERNAME SYSTEM
windir C:\WINDOWS

Power Profile
Active power scheme High performance
Hibernation Enabled
Turn Off Monitor after: (On AC Power) Never
Turn Off Hard Disk after: (On AC Power) Never
Suspend after: (On AC Power) Never
Screen saver Enabled

Uptime
Current Session
Current Time 23/4/18 04:12:19
Current Uptime 3,011,830 sec (34 d, 20 h, 37 m, 10 s)
Last Boot Time 19/3/18 06:35:09

[Andrea Alessi]

Hi Sarah,

any chance you could share with me the full log please ? It's very odd that with 14GB free memory zephyr couldn't allocate a few MBs. Either something else was going on or it may be an issue with the specific hardware. However before telling you to run diagnostics like memetest and such, i'd like to see the log so i can investigate further if possible

Thank you!

[SarahC]

Sure thing....

Here's the effect of the errors: https://imgur.com/IRoTES3

------------

Textured Mesh Generation
Selected mesh: Mesh 3
Texture size: 4096
Image Resolution percent: 100 %
Maximum number of vertices: Maximum
Use multiband method: yes
Maximum cameras per triangle: 1
Number of selected cameras: 26

[SarahC]

I've just remembered to tell you - I don't actually NEED the models of the bins. I was experimenting, and saw the error.
So I'll provide my services at gaining clues and data as you require. If it's some odd hardware issue for me, no matter.
But if I've been able to help make the software even better - I'm happy to help out. =)

[Andrea Alessi]

Hi Sarah,

thank you for your followup! No worries, we're always happy to help and welcome all feedback

It looks like it's either some hardware issue (system ram) or environment (i.e. some other software running in the background that may have been tampering with the memory). As a test, did you try to re-compute it after a windows reboot?

[SarahC]

>As a test, did you try to re-compute it after a windows reboot?

I haven't.
I will tonight. =)

It's unlikely that anyone would want to use the program like this (apart from me) - I tried opening the program while I was logged in via Remote Desktop. It errors with OpenGL not found.
So, wondering what happens if the program's running already - I RD'd onto the machine - and it crashes Explorer.exe, which then reloads.
I do dev work, and it got me wondering what mechanism was going wrong - several programs I use can open render windows inside a RD desktop. I wonder if it's CUDA? I have had no experience with those libraries so I've never RD'd with a program using CUDA.

[Andrea Alessi]

Keep us updated then

As for the Remote Destkop, the problem is that microsoft's RDP doesn't support OpenGL acceleration. Unfortunately, if you wish to run software that requires 3D acceleration (such as zephyr) you need to use another remote desktop software, for example splashtop or logmein hamachi.

Unfortunately you can not use Microsoft's RDP with Zephyr and the only solution is to use another software.

[SarahC]

I wonder if VNC will work - I've not had chance to work on my project yet, but I plan to restart my machine after some processing completes, and my housemate also has a GTX970 in some workstation they have, so we can test that too. It might be a bug with the GTX drivers...

[Roberto]

I confirm that realVNC, tightVNC and ultraVNC work fine. The problems with the openGL applications is only specific to Windows RDP.

[SarahC]

Thanks Roberto!

Today is the day I reset my machine, and also install the Free version on my housemates Z820 which also uses a GTX970 card:

http://www8.hp.com/uk/en/campaigns/workstations/z820.html

I don't think Andrea was able to re-create the error based on his message. (Is that right Andrea?) Or perhaps he didn't have time.

So....... I'll assume it worked.
So.......
If it crashes on both machines , it may be the nVidia drivers/bug. Beg Andrea for a patch to cover for nVidia.
If it doesn't crash on the z8 but still on mine after the reboot - I'll compare drivers to ensure I've got latest.
I'll also run with as many start up programs disabled as possible.
If mine only crashes after all's disabled - get out Visual Studio, slap in the 3dF thread and JIT debug the swine! (But it's the weekend!)

If I fail at this task - likely, I don't read x86/64 assembly too well.........
I'll pinch my housemates Z820 and build the meshes on it.


I like finding and identifying bugs - I hate finding them and not working them out!

I also like chatting.

[Roberto]

I also made some tests, but I was unable to replicate the bug / issue. From the log I can certainly tell it's not related to the gpu (as the gpu was not running on that part), but it seems due to memory or IO problems. Please let me know if you can replicate the problem on other machines as well.

I also like chatting and correcting bugs

[SarahC]

I was able to reboot this morning! 45 days,wow....

The textured mesh generation causes CUDA to run out of memory.
Texture 4096,
100% resolution,
Vertices: Maximum,
Multiband: yes.


I'm guessing this is expected due to their being: Vertices : 25766103 / Triangles : 8588701, and me not setting the maximum number of vertices?



--- Log started ---
[05:17:37] Computing photoconsistency score...
[05:17:46] Vertices : 25766103 / Triangles : 8588701
[05:17:46] CUDA device: 1/1
[05:17:46] CUDA device name: GeForce GTX 970
[05:17:46] CUDA runtime version: 8.0
[05:17:46] CUDA driver version: 9.1
[05:17:46] CUDA free memory: 3297
[05:17:46] CUDA total memory: 4096
[05:17:46] CUDA blocks properties: 2147483647x65535x65535
[05:17:46] CUDA threads properties: 1024x1024x64 (max 1024)
[05:17:46] CUDA capabilities: 5.2
[05:17:46] CUDA runtime capabilities: 5.0
[05:17:55] Disable CUDA device : GeForce GTX 970 [ID 0] (not enough device memory)
[05:17:55] All CUDA devices are disabled. Computation will run on CPU.
[05:17:55] Preparing neighbors...
[05:17:56] Computing visibilities for camera '_89a8809.jpg'
[05:18:01] Computing visibilities for camera '_89a8810.jpg'
[05:18:06] Computing visibilities for camera '_89a8811.jpg'
[05:18:10] Computing visibilities for camera '_89a8812.jpg'
[05:18:16] Computing visibilities for camera '_89a8813.jpg'
[05:18:21] Computing visibilities for camera '_89a8814.jpg'
[05:18:25] Computing visibilities for camera '_89a8815.jpg'
[05:18:30] Computing visibilities for camera '_89a8816.jpg'
[05:18:34] Computing visibilities for camera '_89a8817.jpg'
[05:18:38] Computing visibilities for camera '_89a8818.jpg'
[05:18:43] Computing visibilities for camera '_89a8819.jpg'
[05:18:47] Computing visibilities for camera '_89a8820.jpg'
[05:18:52] Computing visibilities for camera '_89a8821.jpg'
[05:18:55] Computing visibilities for camera '_89a8822.jpg'
[05:18:59] Computing visibilities for camera '_89a8823.jpg'
[05:19:03] Computing visibilities for camera '_89a8824.jpg'
[05:19:07] Computing visibilities for camera '_89a8825.jpg'
[05:19:11] Computing visibilities for camera '_89a8826.jpg'
[05:19:14] Computing visibilities for camera '_89a8827.jpg'
[05:19:18] Computing visibilities for camera '_89a8828.jpg'
[05:19:22] Computing visibilities for camera '_89a8829.jpg'
[05:19:26] Computing visibilities for camera '_89a8830.jpg'
[05:19:30] Computing visibilities for camera '_89a8831.jpg'
[05:19:33] Computing visibilities for camera '_89a8832.jpg'
[05:19:37] Computing visibilities for camera '_89a8833.jpg'
[05:19:41] Computing visibilities for camera '_89a8834.jpg'
[05:20:37] Preparing image : '_89a8809.jpg'
[05:20:37] Preparing image : '_89a8822.jpg'
[05:20:39] Unable to load '_89a8809.jpg' file!
[05:20:39] Preparing image : '_89a8810.jpg'
[05:20:39] Unable to load '_89a8810.jpg' file!
[05:20:39] Preparing image : '_89a8811.jpg'
[05:20:40] Unable to load '_89a8811.jpg' file!
[05:20:40] Preparing image : '_89a8812.jpg'
[05:20:40] Preparing image : '_89a8823.jpg'
[05:20:41] Unable to load '_89a8823.jpg' file!
[05:20:41] Preparing image : '_89a8824.jpg'
[05:20:42] Preparing image : '_89a8813.jpg'
[05:20:43] Unable to load '_89a8813.jpg' file!
[05:20:43] Preparing image : '_89a8814.jpg'
[05:20:43] Preparing image : '_89a8825.jpg'
[05:20:45] Preparing image : '_89a8815.jpg'
[05:20:45] Preparing image : '_89a8826.jpg'
[05:20:47] Preparing image : '_89a8816.jpg'
[05:20:47] Preparing image : '_89a8827.jpg'
[05:20:49] Preparing image : '_89a8817.jpg'
[05:20:49] Preparing image : '_89a8828.jpg'
[05:20:51] Preparing image : '_89a8818.jpg'
[05:20:51] Preparing image : '_89a8829.jpg'
[05:20:53] Preparing image : '_89a8819.jpg'
[05:20:53] Preparing image : '_89a8830.jpg'
[05:20:55] Preparing image : '_89a8820.jpg'
[05:20:55] Preparing image : '_89a8831.jpg'
[05:20:57] Preparing image : '_89a8821.jpg'
[05:20:57] Preparing image : '_89a8832.jpg'
[05:20:59] Preparing image : '_89a8833.jpg'
[05:21:01] Preparing image : '_89a8834.jpg'
[05:21:03] +++ ERROR +++ (computation error)
[05:21:04] +++ ERROR +++ (computation error)
[05:21:04] Done.
[05:21:35] Computing blurriness...

[SarahC]

It's ALL working now!

After restarting, and rebuilding the bins on "high" settings - I got an issue with memory allocation, which disabled CUDA.
BUT - the meshing and texturing completed fine, just without the CUDA acceleration.

I thought it was a bug - until I checked out the stats:
Points: 6,152,947
Triangles: 8,588,690

GPU usage hits 100% just rotating all those! I'm left with about 0.5 GB video RAM left, 3.4GB used... more than many games!

So yeah - the memory allocation problems were due to something just on my machine - I wonder if "RAM fragmentation" is a thing on GFX cards? I know memory requests in C require contiguous free memory.
My test memory problem was due to the HUGE mesh!

Thanks for listening everyone - I feel a bit awkward knowing now knowing it was my own PC's issue, and not the programs!

[Andrea Alessi]

Hi Sarah,

there may be two different things at play here:

[05:17:55] Disable CUDA device : GeForce GTX 970 [ID 0] (not enough device memory)

This may be due to this issue the GTX970 cards have. I think they partially fixed the issue by releasing newer bios, so although i cannot be 100% sure this is the cause, try updating the bios of your card if your vendor has made an update available, just so you can sleep happily. Seeing you report a ram usage sospiciously close to the 3.5 limit, i'd just thought i'd throw this here.

However, the other issue here

[05:20:37] Preparing image : '_89a8822.jpg'
[05:20:39] Unable to load '_89a8809.jpg' file!

is actually unable to load the image data, either from disk (e.g. image no longer present for some reason) or from ram (e.g. bad ram stick).