You can set for each field separately, see post from Kevin Appelt. In this case you need to remember for any further field in future to add this setting too.
To adapt file upload for all fields globaly, you can use UserTSConfig edit_docModuleUpload. See https://docs.typo3.org/typo3cms/TSconfigReference/singlehtml/Index.html#document-UserTsconfig/Setup/Index
To avoid file upload for all fields simple add to ext_localconf.php of your customer extension (sitepackage):
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addUserTSConfig('setup.override.edit_docModuleUpload = 0');
If you want use a separate configuration file for UserTSConfig, which I recommend add to ext_localconf.php
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addUserTSConfig('<INCLUDE_TYPOSCRIPT: source="FILE:EXT:sitepackage/Configuration/TypoScript/User/Default.ts">');
and in your extension /Configuration/TypoScript/User/Default.ts
setup.override.edit_docModuleUpload = 0
If you also want exclude file upload and folder creation in file element browser (popup window) add following code to your extension /Configuration/TypoScript/User/Default.ts
# Hide 'file upload' and 'create folder' in link- and element browser
options.folderTree.uploadFieldsInLinkBrowser = 0
options.folderTree.hideCreateFolder = 1
options.createFoldersInEB = 1