Why is linking to Python debug library disabled for Windows?

[Toninoso]

From detail/common.h

/// Include Python header, disable linking to pythonX_d.lib on Windows in debug mode
#if defined(_MSC_VER)
#  if (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 4)
#    define HAVE_ROUND 1
#  endif
#  pragma warning(push)
#  pragma warning(disable: 4510 4610 4512 4005)
#  if defined(_DEBUG)
#    define PYBIND11_DEBUG_MARKER
#    undef _DEBUG
#  endif
#endif

Since I need debug build #undef _DEBUG caused me a lot of grief until I dicovered it.

On top of wanting to be able to debug my own code, I don't mind being able to step through Python for Windows dll code as well. In fact, in a few cases it was very useful.

Could anyone please explain why would one want to #undef _DEBUG?

[wjakob]

Check a bit later in the file. _DEBUG is redefined if PYBIND11_DEBUG_MARKER is set. All of this is done to avoid an annoying import pragma in the Python header files.

[Toninoso]

You mean these pragmas listed below in config.h?

But again, why? Why force both Debug and Release builds to always link to release python36.lib? What's wrong with linking to Debug version python36_d.lib in Debug build?

I am not sure that it is even safe running app in Debug (while debugging, that is) linked to Python36 DLL built for Release. Here is another case that I closed where you crash in DEBUG if you force python36_d on linker command line:

[1239]#1293

Other than that, I agree that the pragma is annoying, but because it doesn't account for python3_d.lib case when one wants to stick to ABI/Limited API. Makes one wonder why do they even build and distribute debug versions and PDB for limited API and then exclude it.

#		if defined(_MSC_VER)
			/* So MSVC users need not specify the .lib file in
			their Makefile (other compilers are generally
			taken care of by distutils.) */
#			if defined(_DEBUG)
#				pragma comment(lib,"python36_d.lib")
#			elif defined(Py_LIMITED_API)
#				pragma comment(lib,"python3.lib")
#			else
#				pragma comment(lib,"python36.lib")
#			endif /* _DEBUG */
#		endif /* _MSC_VER */

[Vigilans]

My pybind11 is installed from Microsoft/vcpkg.

In the beginning, I directed the include and lib folder to where my python is installed, but later I found that vcpkg has done all the dependency issues for me, and setting the dependencies to vcpkg will make my project more portable, so I just did it.

However, vcpkg includes a python36_d.lib in $(VcpkgRoot)debug\lib folder, and when I try to build my project in debug mode, it throws the could not open python36.lib link-error.

Although I can redirect my paths to the release folders, I also wonder whether it will bring some downsides to the developing...

[LeslieGerman]

I encountered similar issues with the "standard" Windows Python install package, which has both the release and debug python libraries/DLLs.

I build my app with CMake and boost-python, and actually both python37.dll (release) and python37_d.dll (debug) are linked to my application somehow! Which is, of-course, NO-NO...

In fact in boost-python, the default behavior is that even when debug boost libraries are created, these libraries are linked to the release pythonX.dll/lib - by intention, according to the docs.

I guess this is the case with pybind, too...

In fact, according to the doc I referred to, boost-python uses the same "undef _DEBUG" trick, like @Toninoso mentioned above.

IMO, this whole debug-release mix is terrible...

Bests!

[geki-yaba]

Well, JFYI, if you are new to python with c++ extension dlls, just like me, here are some steps to windows debugging success with latest VS2017:

Visual Studio Installer: enable native python development tools (name may differ - german setup here); you may install python via VS Installer, I prefer method below.

Read the "The Hitchhiker’s Guide to Python!"[0] and use Chocolatey[1] as package manager for windows; referenced in the hitchhiker's guide. After install of chocolatey, open an administrative shell - I use cygwin. There, install python3 with debug and symbols data like so:

$ choco install python3 -ia "CompileAll=1 Include_debug=1 Include_symbols=1"

$ python -m pip install -U pip # may need to execute twice

$ python -m pip install -U setuptools
$ python -m pip install -U pybind11

Then edit \path\to\python\include\pybind11\detail\common.h. Remove these blocks:

About line 106

#  if defined(_DEBUG)
#    define PYBIND11_DEBUG_MARKER
#    undef _DEBUG
#  endif

About line 131

#  if defined(PYBIND11_DEBUG_MARKER)
#    define _DEBUG
#    undef PYBIND11_DEBUG_MARKER
#  endif

Now then, Microsoft has safe APIs for strdup and sscanf. If you rename these in pybind11 code, the crashes on quit disappear.

Edit pybind11.h and replace strdup with _strdup. Edit detail\common.h and replace sscanf with sscanf_s.

Now, the topic is a really interesting question. I guess, noone of the devs cares for windows and copy-paste from Boost.Python is the simple way out then.

Edit
There are some more caveats to work out:

Project Properties > Configuration Properties > Debugging >
Working directory: $(OutDir)
Environment:
PYTHONPATH=%PYTHONPATH%;$(OutDir)
PATH=%PATH%;$(OutDir)..

I have my python library in a py3 subfolder of Debug64/Release64, therefore the PATH fix to access to my c++ dlls. And if you ever happen to change the Python version used, edit the vcxproj file manually and look for <PythonVersion>3.7</PythonVersion>, since the Python environment setup in my visual studio solution explorer for my python project is missing.

[0] https://docs.python-guide.org/
[1] https://chocolatey.org/install#installing-chocolatey

[lloydrayner]

Im having the same issue. Trying to build in Windows 10 with VS17 and Python3.6. On calling cmake I get the following error

LINK : fatal error LNK1104: cannot open file 'python36_d.lib' [C:\Users\lloyd\PycharmProjects\orbv2\Pangolin\build\src\pypangolin.vcxproj]

I gather than @Toninoso has a solution but wonder if anyone could describe the solution in more detail for me. I don't quite follow.

I have also tried the suggestions from @geki-yaba.

Thank you

[seando-adsk]

We have also run into this problem trying to create a debug version. I think the solution is do what Boost does which is to use a define which allows this #undef behavior to be overridden.

#ifdef _DEBUG
# ifndef BOOST_DEBUG_PYTHON
#  undef _DEBUG // Don't let Python force the debug library just because we're debugging.
#  define DEBUG_UNDEFINED_FROM_WRAP_PYTHON_H
# endif
#endif

[JGamache-autodesk]

The line # if defined(_DEBUG) should probably become # if defined(_DEBUG) && !defined(Py_DEBUG)
This would allow PyBind11 to react correctly to the presence of a debug interpreter when one is provided in the ${PYTHON_EXECUTABLE} CMake parameter.

[ChilledCosmonaut]

FWIW: If you have problems that the vcpkg installation of pybind11 requires the python*_d.lib file when building for a debug target, just add #undef Py_DEBUG above your pybind includes.