if you need to test real transactions, the only way is to public at least an alpha version of your application, as mentioned here
Because Google Play does not allow you to use your developer account
to directly purchase in-app products that you have created yourself,
you'll need to create test acccounts under your developer account
profile. To create a test account, simply enter a valid Google email
address. Users with these test accounts will then be able to make
in-app-billing purchases from uploaded, unpublished applications that
you manage.
To test your In-app Billing Version 3 application using your own
product IDs:
In the Play Console, add one or more tester accounts to the developer
account that you are using to publish your application. Login to the
Play Console with your developer account. Click Settings > Account
details, then in the License Testing section, add the Google email
addresses for your tester accounts. Build a signed APK file for your
In-app Billing application. To learn how to build and sign your APK,
see Building Your Application for Release. Make sure that you have
uploaded the signed APK for your application to the Play Console, and
associated one or more in-app products with your application. You
don't need to publish the application on Google Play to test it.
Warning: It may take up to 2-3 hours after uploading the APK for
Google Play to recognize your updated APK version. If you try to test
your application before your uploaded APK is recognized by Google
Play, your application will receive a ‘purchase cancelled’ response
with an error message “This version of the application is not enabled
for In-app Billing.”
Install the APK file to your physical test device by using the adb
tool. To learn how to install the application, see Running on a
Device. Make sure that your test device is running on Android 2.2 (API
level 8) or higher and is installed with Google Play client Version
3.9.16 or higher. Login to the test device by using a tester account. Test your In-app Billing application by purchasing a few items, and
fix any issues that you encounter. To learn more about how you can
perform large-scale testing of your In-app Billing app, see Test
Purchases (In-app Billing Sandbox).
But, if you need just to check if your payment flow is correct you can use static requests, which allows you to test payments without publishing.
We recommend that you first test your In-app Billing implementation
using static responses from Google Play. This enables you to verify
that your application is handling the primary Google Play responses
correctly. You can do this even if the app hasn't been published yet.
Another way of testing was Draft app, but as you can read in the second link,
Draft Apps are No Longer Supported
Previously, you could publish a
"draft" version of your app for testing. This functionality is no
longer supported. Instead, there are two ways you can test how a
pre-release app functions on the Google Play Store:
You can publish an app to the alpha or beta distribution channels.
This makes the app available on the Google Play Store, but only to the
testers you put on a "whitelist".
In a few cases, you can test Google
Play functionality with an unpublished app. For example, you can test
an unpublished app's in-app billing support by using static responses,
special reserved product IDs that always return a specific result
(like "purchased" or "refunded").
So, in short words.. you can set up static payments
all over your application (to test calls) and check this way if they work or not. Once the tests are done, replace static Requests with the final form and use your alpha version to test it. Sadly, no other ways are available (officially) since, as posted above, google doesn't allow developer account to purchase your own products.
Afaik, you can't retrieve the keystore from google play