AsksOpen AsksPro Asks

Sandbox · nothing here is real money · How does this work?

A pre-launch app review checklist: what to ask a Pro

Choose relevant reviewers, define realistic tasks and ask for evidence you can act on. A pre-launch review checklist with a Pro Ask template.

By Shaun ConwayUpdated 8 min read

Key takeaways

  • Start with the launch decision you need to make, then invite Pros whose experience matches your intended users and review questions.
  • Ask reviewers to attempt realistic tasks and distinguish what they observed from their interpretation and recommendations.
  • Require evidence appropriate to each finding, with its effect on the user and a clear priority. Avoid quotas for problems.
  • Judge the report against the agreed brief. Candid criticism and a well-supported report with no defects can both complete the work.
  • The Verifier service evaluates the submission; in the current Pro review flow, your final approval releases payment.
On this page9 sections

A useful pre-launch review helps you decide what needs to change before you put an app in people's hands. Invite Pros who understand the intended users and their work, ask them to attempt realistic tasks and require a report that explains what happened and why it matters. The result should give you a practical basis for the next product decision.

When I build something, I bring an understanding of how I intended it to work. The people I am building for bring the circumstances in which it will actually be used. Connecting those perspectives early is one of the most useful forms of cooperation a developer can organise.

This checklist helps you turn that intention into a clear Pro Ask: who to invite, what to review, what evidence to request and how to assess the report.

Start with the decision you need to make

Write down what you want to learn before selecting review activities. Are new users able to reach a useful result? Does the core workflow fit their existing work? Is there a failure that should delay launch? Does the product description set an expectation the app can meet?

Choose a scope that can resolve those questions in the time you agree. A short review of one journey and a broad product assessment are different commitments.

Review areaA useful questionWhat to request
First useCan an intended user reach a useful result without help?Observations from the first attempt, where help was needed and whether the task was completed.
Fit with the user's workDoes the product help them do something that matters?A realistic task, comparison with their current method and conditions for using it again.
Main workflowsDoes the agreed behaviour work on the supported platforms?Expected and actual results, reproduction steps and device or browser details.
Product descriptionDoes the listing or landing page make the product understandable and set accurate expectations?What the reviewer understood before using it and where the experience differed.
AccessibilityCan people with relevant access needs complete the important tasks?A defined assessment using appropriate experience, methods and assistive technology.
Specialist concernsWhat needs technical or professional assessment before the intended use?A separate scope matched to the expertise, access and evidence needed.

A store listing matters for an app distributed through an app store; a web app may need a landing-page review instead. Security, code quality and professional requirements need their own consideration according to the product's intended use. A general user review does not establish those specialist conclusions.

Invite the right perspectives

Someone unfamiliar with the app can reveal assumptions in the first-use experience. Someone experienced in the work can tell you whether the product addresses the right problem. You may need both perspectives, and they may come from different reviewers.

For an accounting app, an accountant might examine how the workflow supports professional judgement, while a business owner assesses whether the inputs and outputs make sense to them. For a property-service dashboard, a host might test scheduling and exceptions, while a guest assesses the information they receive. Give each a task relevant to their role.

Choose Pros for that experience. A LinkedIn check confirms an account connection; it does not establish professional qualifications or suitability for your review. Look at their background and record, and establish any credentials the work requires before inviting them.

Checklist for writing the brief

Describe the user and the task

Name the intended user, the situation and the goal. "As a first-time business owner, create and send a sample invoice" gives a reviewer a task to attempt. Specify the starting point, what completing the task means and which version to use.

Let the first attempt show what the product makes understandable. Record any guidance you provide so that assisted completion is not mistaken for independent success.

Provide access privately

Include the product link, supported platform and version in the brief. Explain how the Pro will get access, including any TestFlight or internal-testing invitation. Send test credentials through the private room rather than including them in the brief or recording.

Use sample records and test payment flows where appropriate. Define what is outside the review, and tell the Pro how to contact you if access fails or an activity would affect real users or data.

Require observations and supporting evidence

For each scenario, ask what the Pro attempted, what happened and whether they completed the task. Evidence should fit the observation:

  • A screenshot or recording timestamp can locate a confusing step.
  • Reproduction steps, expected and actual results help explain a bug.
  • An input and output can support feedback on an agent's reply.
  • A comparison with the user's current process can explain a missing practical requirement.

Ask reviewers to separate observations from interpretations and proposed changes. "I could not find the cancellation control" is an observation. "Other users will abandon the product" is a hypothesis requiring further evidence.

Prioritise by the effect on the user

Use a simple scale with definitions agreed before the review. For example:

PriorityMeaning within this review
BlockerPrevents completion of an essential agreed task.
MajorCreates substantial difficulty or a material risk of an incorrect result.
MinorCauses friction but leaves a workable route to completing the task.
PolishImproves clarity or presentation without materially changing task completion.

Ask the Pro to explain the priority in context. A specialist finding may need a different scale appropriate to that discipline.

Avoid a minimum number of defects. A good report may contain one important finding, several small observations or no defects in the scenarios examined. Require coverage and evidence, rather than a quota of problems.

Ask what already works and what remains uncertain

Knowing what helps a user is useful when deciding what to preserve. Ask for positive observations supported by examples, as well as difficulties and suggested changes.

The report should state what could not be assessed and why. An inaccessible scenario is a limitation to explain, rather than a completed test or an invitation to guess.

A brief you can adapt

Title: Pre-launch review of [app or agent] by [intended user]

Decision this review will inform: [what we need to learn]
Intended users and context: [who, doing what]
Relevant experience required: [experience or qualifications]
Product and version: [URL or test distribution link]
Platforms to cover: [devices, browsers or operating systems]
Access: [test accounts and sample data supplied privately]
Review and reporting time: [agree with the Pro]

Scenarios:
1. From [starting point] to [useful result].
2. [A second task or important exception].

Out of scope: [actions, systems and data excluded from this review]
If access fails or a scenario cannot be attempted, explain why and
contact us through [private route].

Requirements for each report:
1. Describe your relevant experience, platform and version reviewed.
2. Address every scenario: what you tried, what happened, whether you
   completed it and any help needed; explain anything not assessed.
3. Support findings with appropriate evidence, without exposing private
   data or credentials.
4. For bugs, include reproduction steps, expected and actual results.
5. Explain the effect on the user and priority using the agreed scale.
6. Suggest concrete improvements or a next question to investigate.
7. State what worked well, limitations and recommended follow-up.

No minimum number of defects is required.
Candid feedback meets the brief whether positive or critical.
A public review or endorsement is not required.

Payment per approved report: [amount the Pro receives]
Reports wanted: [number and relevant user perspectives]
Closing date: [date]

For an agent, supply representative inputs and ask the Pro to assess outputs and the points where they would intervene. For a product-description review, compare the promise made before use with the experience of attempting the task. Adapt the requirements so they reflect the work you are actually commissioning.

Evaluate the report against the agreement

The Verifier is the service that evaluates the submission against the ask's requirements. A Decision Oracle provides that service, establishing facts from the report and its evidence, including confidence levels. An Expert Decision Oracle brings specialised expertise, using AI with expert skills and qualified human judgement where needed.

The IXO Decision Engine executes the rubric, the agreed evaluation rules, deterministically against those facts. The signed result is the Universal Decision and Impact Determination (UDID). What is an ask? explains how these roles fit together.

In the current Pro review flow, Asky Pro Checker provides the Verifier service and the asker gives final approval before payment. Passing evaluation can therefore leave a report awaiting your approval. The report and evidence remain private to the authorised participants and evaluation service; the Pro's public record can show approved work.

When reading the report, check whether it covers the agreed scenarios, supports its observations and explains its limitations. Distinguish a missing requirement from feedback you disagree with or had hoped would be more positive. Base acceptance on the brief the Pro agreed to, rather than adding requirements after the work.

If an agreed requirement is missing, explain the gap. The current flow pays on final approval, so a report that is not approved is not paid. Approval accepts the review work within its scope; it does not establish that the product is ready for every user or use case.

Agree on scope, time and payment together

Set the payment per approved report and agree on the time needed to attempt the scenarios and prepare useful feedback. Access setup, platform coverage and specialist expertise affect the effort. Avoid treating one hour as a standard for every review.

Under the current fee model, funding a $100 payment to a Pro costs $125: $100 for the Pro and $25 in network and hosting fees. The fee is 20% of the total funding, equivalent to 25% of the Pro's payment.

Commissioning the review commits funding before work begins. Payment to the Pro follows approval. Read the total and terms shown for your ask before funding it.

Use the feedback before launching

Bring the reports together and look for patterns, keeping each reviewer's context visible. Decide which observations call for a change, which need more investigation and which reflect a use case you are not serving yet. Then ask relevant users to try important changes again.

The review can also begin a relationship with a prospective customer. Invite interested Pros to continue using the product or try a revised version. If they want to share their experience publicly, request permission for a quote or invite a separate customer review with the paid review context clear. Their decision to endorse or buy the product is separate from payment for their feedback.

FAQ

What should a pre-launch app review cover?

Start with your intended users and the launch decision you need to make. Choose relevant first-use and core-workflow tasks, then add product-description, accessibility or specialist reviews where they address a specific question. Agree on the scope with reviewers who have suitable experience.

How many problems should a reviewer find?

Set requirements for coverage, evidence and explanation rather than a number of defects. A report with no observed defects can still complete the agreed review; it should state what was examined and the limits of that conclusion.

How long does a review take?

Agree on review and reporting time for the actual scenarios, platforms and access requirements. Confirm availability before setting a deadline, and leave time to act on the feedback.

Can AI agents help with the review?

An agent can assist an eligible, invited Pro within the ask's rules. The report should distinguish automated checks from the Pro's own observations and judgement, particularly when the brief requires first-hand user experience. See Asks for agents.

Will my app or report become public?

A private Pro Ask restricts access to its brief and submissions; it does not change the visibility of an app you already host publicly. Approved work can appear in the Pro's public record while the report stays private.

Invite the people you are building for

Define the decision, choose the scenarios and invite relevant Pros through Review my app. Give them a clear agreement and room to be candid. Use what they learn to make the next version more useful.

Shaun Conway

Shaun Conway is a physician and founder of IXO, where he brings his expertise in human judgement and decision-making into building systems for voluntary intelligent cooperation that shapes the world we want.