On Windows 10 64 bits you need to do both of the following:
1. Delete registry entries
HKEY_CLASSES_ROOT\Directory\background\shell\git_gui
HKEY_CLASSES_ROOT\Directory\background\shell\git_shell
HKEY_CLASSES_ROOT\Directory\Shell\git_gui
HKEY_CLASSES_ROOT\Directory\Shell\git_shell
For convenience or if you want to automate removal (for example to use after choco upgrade) you can run in Powershell (run as Admin):
New-PSDrive -Name HKCR -PSProvider Registry -Root HKEY_CLASSES_ROOT | Out-Null
Remove-Item -Path "HKCR:\Directory\shell\git_gui" -Recurse
Remove-Item -Path "HKCR:\Directory\shell\git_shell" -Recurse
Remove-Item -Path "HKCR:\Directory\Background\shell\git_gui" -Recurse
Remove-Item -Path "HKCR:\Directory\Background\shell\git_shell" -Recurse
2. Unregister the shell extensions dll to remove additional menu entries (GitExt clone..., GitExt Create New Repository, etc.):
regsvr32 /u "C:\Program Files (x86)\GitExtensions\GitExtensionsShellEx64.dll"