Start with your customers, not the feature list
The first question is not about the gateway at all. Look at your last hundred orders and ask how those specific people want to pay. A gateway with an impressive dashboard that does not offer the method your customers use is worthless, and a plain one that does is not.
In Kenya this usually means mobile money is not optional. If you also sell across borders, you need a second rail for buyers whose cards get declined or who would rather not use one. Write down the methods your customers need before you read a single provider page.
Questions one to four: does the money arrive correctly
One. Which payment methods does it actually offer, on the platform you actually use? A provider may support a method in general and not through the plugin you would install.
Two. What currency do you receive, and when? Pricing currency and settlement currency are different questions, and the gap between a payment confirming and a balance being withdrawable is where a lot of unpleasant surprises live.
Three. What are the limits? Minimum and maximum per transaction, and per day. A gateway that cannot handle your largest order is a gateway you will bypass manually, which defeats the point.
Four. How is the conversion rate set if a conversion happens? Ask for it in writing, and compare against a rate you can see quoted independently on the same day.
Questions five to eight: does your system find out
Five. Are orders marked paid by a server-to-server webhook, or by the buyer being redirected back? Only the first is proof. Anyone can visit your success URL. If the answer is the redirect, walk away.
Six. Are webhooks signed, timestamped and given a unique event id? You need to be able to verify a delivery came from the provider, reject one that is hours old, and detect a retry so you do not fulfil an order twice. Ask to see the documentation for this specifically. A vague answer here is the answer.
Seven. Can a retry double-charge? Ask whether the API supports idempotency keys. A double-clicked Pay button and a timed-out request that gets retried are both routine, and a gateway without idempotency will occasionally take money twice and make it your problem.
Eight. What happens when your endpoint is down? Look for retries with backoff over a meaningful window, visibility into failed deliveries, and the ability to replay one once you have fixed your side.
Questions nine to twelve: can you operate it
Nine. Can you refund from your own admin, in part as well as in full? If refunding means emailing support, factor the delay into your customer-service promises, because you will be making it.
Ten. Is there a real test mode with separate keys? You should be able to run a complete payment, webhook and refund cycle in your own test suite without moving money. Building against live credentials is how launch weekends go wrong.
Eleven. How are credentials structured? A single all-powerful key that has to reach the browser is a bad design. Look for a split between a publishable key that can do almost nothing and a secret key that never leaves your server, ideally with domain restrictions and scopes.
Twelve. Can you reconcile? Every payment should carry your own order reference, and you should be able to list and filter payments by it. Ask whether errors include a request id you can quote to support, because the difference between a five-minute answer and a five-day one is usually that id.
The question nobody asks: is it actually available
Payment providers publish pages for integrations that are in development. That is not dishonest in itself, but it becomes your problem if you plan a launch around one. Ask directly: is this integration live for merchants today, is it in a pilot, or is it on a roadmap? Then ask for a named contact and a date.
Ask the same question about the parts, not just the product. An API can be live while a specific platform plugin is still in a pilot. Get it in writing, per component, before you build a launch plan on top of it.
How to run the evaluation in an afternoon
Shortlist two providers, get test credentials from both, and run the same three scenarios end to end: a successful payment, an abandoned payment, and a partial refund. Then deliberately break your webhook endpoint and see what each provider does about it.
You will learn more from that afternoon than from a month of comparing feature tables, and the failure paths are where the difference between providers actually lives. Any provider that cannot give you test credentials to do this has answered a different question, and you should take note of it.