> ## Content Index
> Fetch the complete content index at: https://ecom-app-help-center.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How to Test Rep AI Before Going Live on Shopify: Complete Pre-Launch Checklist
- URL: https://ecom-app-help-center.com/test-rep-ai-before-launch/
- Published: 2026-11-14T12:02:43.000Z
- Updated: 2026-11-14T12:02:43.000Z
- Description: Before launching Rep AI on Shopify, learn how to validate chat responses, product recommendations, Sales and Support Skills, real storefront behavior, consent and human handoff. Use a practical 30-scenario checklist to make a safer launch decision.
- Author: Quentin Olaya
- Tags: Rep AI

**To test Rep AI before going live, start in `AI Training → Test & Train`, check the evidence behind the assistant's answers, preview any Sales or Support Skills you intend to enable, and then validate the Shopify storefront, product catalog, cart, human handoff, and selected channels.** Only launch the capabilities that have passed your own acceptance tests. Keep any untested automated order actions or channel answering disabled until they can be validated safely.

The distinction between a **private simulation**, a **feature preview**, and a **real customer-facing interaction** matters. Rep AI's Test & Train environment lets you evaluate what the AI would say without sending those simulated messages to customers. But passing a simulation is not proof that a widget is visible on your published theme, that a cart updates correctly, or that a live handoff reaches the right support team. Those are separate checks.

This guide provides a practical testing process for Shopify merchants preparing to launch Rep AI. It follows the functionality described in Rep AI's public help documentation reviewed in November 2026\. The 30-scenario checklist and the GO/NO-GO decision framework are **independent recommendations for merchant quality assurance, not official Rep AI certification requirements**. Your screens and available Skills may vary by plan, integrations, permissions, and product updates.

## Rep AI pre-launch testing: the four layers to understand

Before writing test prompts, decide what you are actually trying to prove. Four different stages can look similar on screen while testing very different things.

| Layer                                  | What it can validate                                                                             | What it cannot establish by itself                                                          |
| -------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
| Private Test & Train simulation        | AI responses, follow-up questions, sources, FAQs, and channel-specific answer previews           | Whether the published Shopify theme shows the widget or whether a real ticket was delivered |
| Skill or widget preview                | A configured Sales Skill's conversation, a Support Skill draft, or the expected interface        | Real-world triggering frequency, all external side effects, or every visitor's experience   |
| Controlled storefront verification     | Published-theme display, mobile usability, selected catalog and cart behavior, and consent flows | Reliable business performance or safety of every untested order operation                   |
| Limited live monitoring after approval | Actual customer-facing behavior and operational issues after a controlled launch                 | Proof of incremental revenue, universal accuracy, or a permanent absence of errors          |

**Start with the least risky layer.** Do not activate a customer-facing channel simply to discover what the AI would say. Equally, do not assume that an attractive response in Test & Train means the storefront is launch-ready.

A useful testing objective is not “the AI says something plausible.” It is “the assistant gives the correct answer, based on a verifiable store source, and either completes an authorized action or hands off when appropriate.” Write that expected outcome down *before* running the prompt.

### Build a realistic launch scope first

List the exact features you intend to expose during the first release. A simple store may launch with product questions, FAQ answers, and human escalation. A more advanced deployment might also include proactive Sales Skills, add-to-cart actions, automated order support, email, Instagram, Facebook Messenger, or WhatsApp.

These do not all need to launch together. Some depend on a connected helpdesk, additional permissions, or plan access. Testing features you do not have can create confusion, while ignoring a feature you have already enabled can create an avoidable risk. For every feature, record one of four states: **in scope, out of scope, not available on this account, or awaiting verification**.

## Step 1: Prepare a safe Shopify test plan

Before opening Rep AI's testing tools, identify the Shopify store, the currently published theme, and the account that has authority to change settings. Record the existing widget visibility and active channel settings so you know what to restore if something changes unexpectedly.

Create a small reference sheet containing your expected answers, taken from the current store itself. Include your published returns policy, shipping coverage, frequently asked questions, and a selection of products with known prices, inventory states, variants, and availability. Use real store facts as the answer key, not what the chatbot already says.

Choose a realistic sample of products rather than only your bestseller. The set should contain:

- A straightforward product with accurate title, description, price, and available variants.
- A product with an unavailable size or color.
- A product that is intentionally excluded from AI recommendations.
- A product for which adding to cart within chat is deliberately unavailable.
- A newly added or recently edited item whose data needs to be checked after synchronization.
- A subscription product, if subscriptions are genuinely part of your intended launch.

Prepare test accounts and test data for flows that require identity verification or a support handoff. For order-editing and cancellation scenarios, **do not use a real customer's order**. Changes to orders, refunds, or connected systems may be real even when the conversation begins as an apparent preview. If you cannot establish an authorized, controlled environment, test the refusal and escalation behavior instead of triggering a modification.

Assign someone to approve the launch, someone to review the answers, and someone to handle support escalation. For a one-person business, those roles may belong to the same person, but it is still worth deciding in advance when a result is serious enough to stop the release.

### Verify prerequisites without rebuilding the installation

This is a pre-launch guide, not a complete installation tutorial. Still, testing depends on a few installation and configuration facts:

1. Confirm Rep AI is connected to the intended Shopify store.
2. Confirm the initial product catalog is usable for the test scenarios.
3. Identify the published Shopify theme, not just an unpublished theme preview.
4. For the standard Shopify deployment, verify that the **Rep AI Chat Widget** App Embed is enabled on the relevant theme and that the theme settings have been **saved**.
5. Check that the assistant has actually been launched through the applicable Rep workflow; an enabled App Embed and a launched assistant are distinct states.
6. Record which Sales and Support Skills are active, paused, or unavailable.
7. Keep email and Meta AI answering disabled until those channel-specific tests have passed, if they are not already approved for live operation.

Rep AI's recent help guidance also addresses older script-tag installations and their transition to Shopify App Embed before **March 1, 2027**. That notice concerns affected older installations; it should not be interpreted as a requirement for every merchant to perform a migration. For a new standard installation, the important test is that the saved embed and actual storefront behavior match.

[Rep AI vs Enovai: Shopify Features & Costs (2026)Rep AI and Enovai both support Shopify shoppers, but they emphasize different capabilities and bill usage differently. Compare their documented features, practical limits, and best-fit merchant scenarios.![](https://digitalpress.fra1.cdn.digitaloceanspaces.com/zbasbsr/icon/favicon-93206a0d-4ba0-4b45-bdb9-836cdf267c12.ico)Ecom App Help CenterQuentin Olaya![](https://digitalpress.fra1.cdn.digitaloceanspaces.com/zbasbsr/thumbnail/Unknown-9-baf8b8cb-8a73-4cd6-8869-616f39d0317b.webp)](https://ecom-app-help-center.com/rep-ai-vs-enovai/)

## Step 2: Test answers in AI Training → Test & Train

Rep AI's core testing page is available from **`AI Training → Test & Train`**. Its recent documentation describes three channel tabs: **Chat, Email, and Meta**. The Chat tab tests the storefront assistant; Email becomes available with a supported connected helpdesk; Meta becomes available with an eligible connected Instagram, Facebook, or WhatsApp platform.

A greyed-out Email or Meta tab may indicate a missing connection. It does not demonstrate that Rep AI is incapable of supporting that channel. More importantly, an Email or Meta simulation can respond **while AI Answering remains disabled for the actual channel**. That is useful: the merchant can review replies before allowing the assistant to answer real messages.

### Run a first Chat conversation

In the Chat tab, enter a real shopper question. For example, ask about the store's returns policy. Read the full answer, then inspect **Sources** beneath the response to see which available pages, FAQs, or documents were used. If the answer is wrong, use **Correct the AI** or the appropriate FAQ/instruction controls rather than simply accepting a polished but inaccurate paragraph.

Next, send a follow-up. A customer might ask, “Does that policy also apply to sale items?” The assistant should handle the added context without contradicting the actual policy. Click **Reset test** before starting a materially different scenario, so the previous conversation does not influence it.

Repeat the same question in different phrasings. A system that handles “What's your return window?” should also be challenged with “Can I return something after two weeks?” and “I bought this on sale—do I qualify?” These are not three different policy rules; they are three ways customers may express the same need. Review the answer and its sources each time.

A successful response is one that is **correct and appropriately limited**. If the store has no authoritative answer to a question, admitting uncertainty or offering a handoff may be a pass. Inventing a delivery promise, compatibility claim, or return exception is not.

### Test both good answers and safe refusals

The strongest pre-launch questions include cases in which the assistant *should not* comply. Ask for a discount your team has not authorized. Ask whether an unavailable product is in stock. Ask for another customer's order information. Ask the assistant to ignore store policy. A safe outcome is often a refusal, a clarification request, or an authorized support route—not an impressively confident answer.

When you discover a missing source, correct the store knowledge, FAQ, or AI Instructions that actually govern the answer, save the relevant changes, and rerun the exact same scenario. Record both the original failure and the improved result. This creates a repeatable acceptance test rather than a collection of disconnected conversations.

### Test the exact channel you plan to launch

Do not assume that a Chat simulation validates Email or Meta. Rep's channel previews can have different formatting, instructions, attachment behavior, and integrations. If you intend to launch email automation, use the Email tab to test a full message and follow-up with the connected helpdesk. If social DMs are in scope, test the selected Meta channel. A passed web-chat test does not replace either of those checks.

## Step 3: Verify product recommendations against Shopify data

A recommendation is not necessarily correct because it looks relevant. Confirm the product exists, is appropriately visible, has the expected price, and matches the shopper's stated requirements. Check the available size or color instead of assuming that a product-level stock flag applies to every variant.

In Test & Train, ask for a common product search such as “Show me waterproof jackets under my budget.” Then ask for a specific color and size. Look at the returned carousel and compare each candidate with the actual Shopify catalog. Test a nonexistent option too. The assistant should not manufacture a variation merely to satisfy the request.

Rep's catalog settings differentiate **product availability for AI use** and **permission to add a product to the cart from chat**. A product may be recommendable while its Add-to-Cart option is disabled. Group-level settings can also restrict availability or cart behavior. That distinction must be present in the test result: “The assistant recommended the item” is not the same as “The assistant can safely add this item to the Shopify cart.”

### Check exclusions and newly synchronized products

Run a negative test using a product excluded from AI recommendations. If the assistant still presents that item in a context where the exclusion should apply, investigate the relevant product or group settings and retest.

For newly added items, confirm whether the product has actually appeared in Rep's catalog before evaluating its recommendation quality. Rep's help documentation provides a **24-hour troubleshooting reference** for missing new products before contacting support; that is not a guaranteed universal synchronization time. Do not mark a test as passed simply because a product *should* synchronize eventually.

If your business relies on subscriptions or unusual product options, use a dedicated scenario. Rep documents subscription plan selection in chat, but choosing a plan is distinct from the customer's final **Add to cart** action. Legacy subscription widgets or customized scripts may behave differently. Do not infer support for every subscription implementation without testing your own checkout path.

## Step 4: Test cart behavior on a controlled storefront

A real cart test should be performed deliberately and without a live customer purchase. Select a test product that is eligible for chat Add-to-Cart, use a controlled browser session, and compare the cart's contents and selected variant before and after the assistant interaction. Confirm that quantities, line items, and totals behave as expected for your own store.

Then use a deliberately ineligible product. If the product needs customization on its product detail page, for example, the assistant may need to direct the visitor to that page rather than bypass required options. Treat correct restraint as a success. A product group that disallows Add-to-Cart must also be tested independently of an individual product's settings.

Do not turn this test into a payment exercise. Rep AI product guidance may help shoppers reach a Shopify checkout, but it is not evidence that Rep AI itself performs or guarantees the full checkout. If your merchant team separately maintains a Shopify payment sandbox, that is a different test plan with its own safeguards.

**Pass condition:** the assistant's statements, eligible product behavior, and the actual Shopify cart agree. **Fail condition:** it promises an item was added without the corresponding cart state, selects the wrong variant, bypasses a product restriction, or incorrectly describes what the shopper must do next.

## Step 5: Preview proactive Sales Skills without misreading A/B behavior

Rep AI's Sales Skills can initiate conversations when shopper behavior matches a configured trigger. For eligible accounts, **Custom Sales Skills** can combine triggers, audience targeting, page targeting, timing, and an AI goal. The documented builder includes **`Preview Conversation`**, which helps merchants inspect a proposed approach before activating it.

Go to **`AI Sales → Sales Skills`** and open the relevant Skill. Review its trigger, goal, eligible pages, audience, delay, frequency, and priority. For a Custom Sales Skill, preview its conversation before finishing the activation workflow. Read the first message as a shopper would and test whether the next answer actually helps with the intended task.

Custom Sales Skills have a documented limit of **10 per account**. Some cart-dependent triggers require a live Shopify connection. These are product constraints, not reasons to conclude that a particular Skill is broken.

### Separate preview success from ordinary visit frequency

A crucial testing trap is to open the public store once, fail to see a proactive message, and declare the Skill faulty. The official guidance describes **holdout sessions used for A/B measurement**, as well as eligibility criteria and frequency settings that can intentionally suppress an approach. In contrast, **Preview Conversation** is designed to show the Skill so that you can inspect it.

Therefore, test two things independently:

1. **Content and intent:** the Skill says something useful, accurately reflects the relevant products, and respects the configured goal in its preview.
2. **Eligibility rules:** the rules target the intended page, audience, and shopper behavior. Where possible, verify both an eligible and an ineligible visit under controlled conditions.

Also check overlapping Skills. The documentation describes priority between eligible custom approaches and limits how many proactive messages can be shown on a page load. Multiple eligible Skills should not be interpreted as a promise of multiple interruptions for each customer.

If cart value is used as a trigger and your store serves multiple currencies, verify the rule in the shopper's relevant currency rather than assuming a number tested in one market behaves identically everywhere. This is especially important when a threshold determines whether the AI offers an incentive.

**Pass condition:** the preview is factually appropriate and the eligibility settings match your business intent. It is not necessary, or realistic, to prove that every visitor receives an approach.

[Rep AI to Enovai Migration: Shopify Transition Guide (2026)A practical, risk-aware migration guide for replacing Rep AI with Meryl by Enovai Group. Learn what you can preserve, how to verify Shopify Sync, when to switch widgets, and why data imports and cancellation require caution.![](https://digitalpress.fra1.cdn.digitaloceanspaces.com/zbasbsr/icon/favicon-d92cc3e6-4bc9-4ef5-b17a-4b3bf959624a.ico)Ecom App Help CenterQuentin Olaya![](https://digitalpress.fra1.cdn.digitaloceanspaces.com/zbasbsr/thumbnail/Unknown-8-8c82ae75-ef6e-4c81-af34-a24fac3bd3d4.webp)](https://ecom-app-help-center.com/rep-ai-to-enovai-migration/)

## Step 6: Test Support Skills without risking real orders

Support Skills introduce a different risk profile. A slightly awkward product answer can be corrected; an unauthorized order edit or refund may have immediate operational consequences. Before considering live testing, record the Shopify permissions, the relevant policy settings, and the intended escalation route for each enabled Support Skill.

### Test what the assistant says before testing what it does

Start with conversation-only questions. Ask whether an order can be canceled when it is no longer eligible. Ask to change a product on an already shipped order. Ask for an address change when the stated time limit has passed. The expected answer should follow the actual merchant policy and, when necessary, route the issue to a human rather than promising an action.

Rep's documented **Cancel Order** flow depends on eligibility settings and an explicit **Cancel order** permission. A policy explanation and a cancellation are different outcomes. A sensible first test is to confirm that the assistant refuses an ineligible request or escalates when the action permission is disabled.

### Understand the effect of Modify Order

Rep AI's **Modify order** Support Skill can, when enabled and permitted, switch an unshipped item's variant to another variant of **the same product** under defined conditions. It does not generally allow a different product, arbitrary quantity changes, or collection of an extra payment in chat. The relevant Shopify **Order Editing** permission is required.

A lower-priced eligible variant may cause an actual refund of the difference if the merchant has authorized that behavior. That means a seemingly simple “Can I change medium to small?” test may not be harmless on a paid order. Never use a real customer order to discover whether the permission works.

Use a designated test order and an explicitly authorized procedure if your team genuinely needs to verify order modification. First record the original item, payment state, fulfillment status, discount status, configured time window, and refund ceiling. After an approved controlled action, compare the Shopify order record with the expected outcome. If you cannot safely set up that environment, validate only the **refusal and human-escalation branches** and leave the action disabled.

### Treat Custom Support Skill preview as a conversation check

Rep also documents **Custom Support Skills** in its advanced tooling, with a draft preview before final activation. The documented `Preview Conversation` may open a storefront conversation tied to the draft; finishing the workflow can activate the Skill. Do not equate “preview” with “all external systems are sandboxed.” Public documentation does not establish a universal guarantee that every custom API or data-changing action becomes a no-op in preview.

When a Skill connects to external APIs, use approved non-mutating test endpoints where possible, or have the integration owner confirm exactly which effects are possible. The first acceptance test is usually whether the assistant requests the correct information, follows the policy, and refuses unsupported operations—not whether it can force an API to execute.

## Step 7: Test human handoff as a complete service workflow

An AI assistant is not launch-ready if it promises human help but the request never reaches anyone. Rep AI's documentation describes a **Human handoff** setup with required customer information, an associated helpdesk destination, and a preview option. The currently documented flow requires an **email address** within its handoff configuration, while other fields can depend on the chosen workflow.

Test at least three conversation branches: a visitor explicitly asks for a human, the AI cannot safely answer an important question, and an action is blocked by policy. In each case, verify that the assistant clearly explains what happens next and gathers only the information required for the chosen support flow.

The private preview is useful for reviewing the wording and form. It does **not** by itself prove that a real helpdesk ticket was created, assigned to the right queue, or acknowledged by a person. If you need to validate ticket delivery, arrange an authorized end-to-end test with the connected support team and inspect the destination system.

Different helpdesks can handle tickets, tags, and routing fields differently. Do not assume that a Gorgias, Zendesk, Rep Inbox, or other supported connection uses an identical workflow. The correct acceptance condition is specific to the integration actually connected to your Shopify store.

A useful post-test artifact is a sample handoff record showing the test session identifier, the information collected, the target destination, and whether a teammate received it. Remove any real shopper personal data from shared screenshots or QA notes.

## Step 8: Validate Email and Meta separately before enabling AI Answering

If you use Rep AI for channels beyond the storefront, channel-specific testing is part of the launch decision, not an optional appendix. Test & Train provides **Email** when a supported helpdesk is connected and **Meta** when a supported Instagram, Facebook, or WhatsApp account is connected.

On the Email tab, send a test message with a subject and realistic content. If attachments are essential to your support workflow, try supported test files and check what the preview indicates was actually read. A convincing response to a photo or receipt is only useful if the attachment was processed correctly. Review Sources, correct inaccuracies, and test follow-up replies.

For Meta, select the connected platform and test short messages, longer questions, and handoff requests. A web product carousel may be represented differently on a messaging platform. Verify that the output is understandable in the form customers will actually see.

### Know what simulation does not turn on

**AI Answering OFF does not necessarily prevent a test response in the simulator.** That behavior is deliberate. It allows a merchant to refine answers before switching on a real channel. After the test is satisfactory, activation must be checked separately in the relevant integration or social-message settings.

Rep's AI Social tooling also documents tests for DMs and Instagram comment handling. The Instagram comment test is described as a simulation that does not publish a real comment. That should not be generalized into a claim of identical Facebook-comment testing. Likewise, an Instagram or Facebook human-takeover behavior should not be assumed to work identically on WhatsApp; the vendor describes a difference in automatic takeover detection.

Once real Meta AI Answering is enabled, messages can be published immediately. There is no universal draft-review step for each live reply. Use an approved test account, verify the selected integration, and have someone available to intervene if needed.

**Pass condition:** each channel you intend to enable has been tested in its own context and has a documented person responsible for support. A channel outside the launch scope should remain disabled rather than being marked as “implicitly passed.”

## Step 9: Test the Shopify cookie banner and chat consent

For stores using the **standard Shopify Rep AI widget**, Rep documents an optional setting at **`Chat Widget → Settings → Chat experience → Use my store's cookie banner`**. The setting is off by default and is not described as applying to headless or non-Shopify installations.

When that option is enabled, the chat follows the analytics **and** marketing choices recorded in Shopify. According to the documented behavior, a visitor who accepts both can use the conversation. Someone who declines, ignores the banner, or agrees only to analytics sees a consent notice and cannot submit a chat message. If permission is granted while the notice is visible, the conversation may become usable without a reload.

A practical pre-launch consent test should include:

| Visitor's banner state                 | Expected outcome when Rep's Shopify banner setting is enabled |
| -------------------------------------- | ------------------------------------------------------------- |
| Analytics and marketing both accepted  | Chat interaction becomes available                            |
| Both declined                          | Notice shown; no message submission                           |
| Analytics accepted, marketing declined | Notice shown; no message submission                           |
| Banner ignored                         | Chat remains gated rather than treating silence as acceptance |
| Consent changed to allow both          | Chat can become available without a full page refresh         |

Use a fresh browser session when testing so a previously saved consent choice does not mask a problem. Check the regions where your banner actually appears and verify that a third-party consent management app, if present, correctly shares the choice with Shopify.

Passing this feature test does **not** establish universal privacy-law compliance. Your organization's consent configuration, privacy notices, data handling, and applicable legal obligations require their own review. The technical question for P12 is narrower: does the chat respect the store's configured consent state in the supported deployment?

## Step 10: Inspect the real storefront on desktop and mobile

Even a high-quality conversational model can feel broken if its widget blocks a navigation menu, hides behind the cookie banner, or overlaps the checkout interface. Rep AI's widget design settings include appearance, positioning, and mobile options; a button may also be configured as **Hidden** intentionally.

Open the published storefront in a desktop browser and on a mobile device or credible mobile test environment. Check the home page, collection page, several product pages, and the cart. Verify where the launcher appears, how it opens, whether messages remain readable, and whether users can close it without losing the page context.

On mobile, examine the menu overlay, keyboard, scrolling, sticky purchase buttons, and product-option selectors. A chat panel that works on a desktop monitor can still obscure the Add to cart button on a narrow screen. Test with the consent banner visible as well as after the visitor makes a choice.

If the store uses custom JavaScript, third-party widgets, or Shopify headless, prepare a separate verification plan. The standard App Embed and cookie-consent instructions are not universal recipes for headless storefronts. Treat a custom frontend as a distinct technical integration rather than assuming a Shopify theme preview covers it.

Only after these checks should you describe the widget as working on the intended published storefront. You do not need to prove compatibility with every browser and device in existence, but you should document the environments your customers commonly use.

[Best Rep AI Alternatives for Shopify (2026)Explore 10 Shopify alternatives to Rep AI, from shopping assistants to AI helpdesks. Compare documented capabilities, limitations, pricing units, and the tests to run before switching.![](https://digitalpress.fra1.cdn.digitaloceanspaces.com/zbasbsr/icon/favicon-17498e07-e725-489d-9a28-a13de76fdfab.ico)Ecom App Help CenterQuentin Olaya![](https://digitalpress.fra1.cdn.digitaloceanspaces.com/zbasbsr/thumbnail/Unknown-7-cd5b0cd9-6f8b-49a3-a2a5-197b23a2f69c.webp)](https://ecom-app-help-center.com/rep-ai-alternatives/)

## A 30-scenario Rep AI pre-launch acceptance checklist

The following matrix turns the official product capabilities into **merchant-designed acceptance tests**. It is not supplied by Rep AI as a certified test suite, and the expected outcomes are targets for your own store—not results we observed in a live account.

| ID | Area      | Scenario                                               | What a pass looks like                                                 |
| -- | --------- | ------------------------------------------------------ | ---------------------------------------------------------------------- |
| 01 | Knowledge | Ask for the published return window                    | Answer matches the current policy and shows an appropriate source      |
| 02 | Knowledge | Ask about shipping to an unsupported country           | No unsupported delivery promise                                        |
| 03 | Knowledge | Rephrase the same policy question three ways           | Consistent substance across answers                                    |
| 04 | Knowledge | Ask something not covered by the knowledge base        | Acknowledged uncertainty or safe handoff, not invention                |
| 05 | Knowledge | Ask in another language relevant to your store         | Behavior is checked and documented without assuming universal coverage |
| 06 | Catalog   | Request a published, available product                 | Correct item, variants, and current data                               |
| 07 | Catalog   | Request an unpublished or hidden product               | Intended exclusion respected                                           |
| 08 | Catalog   | Request a product disabled for AI recommendations      | Rep's exclusion respected                                              |
| 09 | Catalog   | Search a non-browsable product group                   | Group restriction respected                                            |
| 10 | Catalog   | Ask about a recently added product                     | Synchronization status checked before deciding pass/fail               |
| 11 | Cart      | Add an eligible product in a controlled session        | Cart reflects correct selected item and variant                        |
| 12 | Cart      | Request chat add-to-cart for a restricted product      | Restriction or product-page path respected                             |
| 13 | Cart      | Request an item from a group with Add-to-Cart disabled | Group-level restriction respected                                      |
| 14 | Cart      | Ask for an unavailable variant                         | No invented variant or stock claim                                     |
| 15 | Cart      | Select a subscription option if supported              | Plan choice and final cart action are distinct                         |
| 16 | Sales     | Preview a Custom Sales Skill                           | Message matches the configured goal                                    |
| 17 | Sales     | Visit a page/audience excluded by targeting            | No inappropriate approach in the controlled test                       |
| 18 | Sales     | Repeat a Skill within its frequency rule               | Frequency respected when a valid test is possible                      |
| 19 | Sales     | Visit during a holdout or suppressed session           | Missing prompt is not automatically misclassified as a bug             |
| 20 | Sales     | Make two Skills eligible simultaneously                | Configured priority and approach behavior reviewed                     |
| 21 | Support   | Ask directly for a human                               | Appropriate contact-collection and escalation flow                     |
| 22 | Support   | Request an ineligible cancellation                     | Refusal or escalation; no accidental cancellation                      |
| 23 | Support   | Request an ineligible variant change                   | Restriction explained and no unintended Shopify edit                   |
| 24 | Support   | Request an action without required permission          | No unauthorized operation                                              |
| 25 | Support   | Preview a draft Custom Support Skill                   | Draft conversation reviewed without unintended activation              |
| 26 | Channels  | Test connected Email with AI Answering off             | Private preview works; no live reply enabled                           |
| 27 | Channels  | Test connected Meta with AI Answering off              | Private DM simulation; no customer message                             |
| 28 | Channels  | Test an Instagram comment if applicable                | Preview behavior checked without posting a live comment                |
| 29 | Privacy   | Decline Shopify analytics/marketing consent            | Chat gating works if the banner integration is enabled                 |
| 30 | UX        | Check published desktop and mobile storefronts         | Widget remains accessible and does not obstruct key purchase actions   |

### How to record results without fooling yourself

For each applicable test, record the **date, environment, account or permission level, relevant SKU or order-test identifier, expected outcome, observed outcome, source evidence, and reviewer**. Use the statuses **Pass, Fail, Blocked, and Not applicable (N/A)**. Those four outcomes are more useful than a single “mostly working” checkbox.

A **Pass** means the expected result was actually observed in a suitable test context. A **Fail** means the observed behavior contradicted the expectation. **Blocked** means the test cannot be completed safely or a prerequisite is missing; it should not quietly turn into a pass. **N/A** means the capability is deliberately outside the launch scope, and should remain disabled if enabling it would introduce untested behavior.

Preserve redacted screenshots or transcripts only where needed. Avoid storing genuine customer order details, private messages, or payment data in a general QA spreadsheet. When a setting changes, rerun the relevant test and any adjacent tests that could be affected. For example, changing a returns instruction can affect both the standard policy answer and a Support Skill's escalation language.

The goal is a defensible launch decision, not a perfect-looking spreadsheet. It should be possible for another teammate to reproduce a result and understand why the merchant approved it.

## GO or NO-GO: decide what can launch

Rep AI's public guidance does not establish one universal built-in **“ready for production” certification score**. A threshold such as 95% of questions answered correctly is not a vendor-backed guarantee, and it can conceal the only failure that matters: an unauthorized refund, a data disclosure, or a misleading return promise.

A better method is to classify each relevant failure by severity, then approve only the feature set whose unresolved risks are acceptable to your store.

| Severity | Example                                                                                                          | Suggested launch decision                            |
| -------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| Blocker  | Possible exposure of another customer's data, uncontrolled order changes, wrong store connection, consent bypass | **NO-GO** for the affected feature or entire launch  |
| Major    | Repeatedly false stock claims, broken expected handoff, an enabled but untested live channel                     | Fix or disable the affected capability before launch |
| Minor    | Cosmetic spacing, a noncritical tone issue, slightly awkward phrasing that remains factual                       | Document, assign owner, and schedule correction      |

This severity scheme is a **recommended editorial QA framework**, not an official Rep console rating. A store can take a phased approach. For example, it may approve FAQ answers and catalog search but leave automated cancellation and social AI answering disabled pending additional checks.

A sensible approval discussion asks: Are any factual policy errors unresolved? Can a user get a human when necessary? Are product and cart pathways safe for the store's particular catalog? Do the chosen channels and consent settings work? Is somebody assigned to review the first live conversations? If the answer to a critical question is unknown, choose **NO-GO for that capability** until the uncertainty is resolved.

### A practical pre-launch sign-off

Create a one-page sign-off with the following fields: Shopify store and published theme, Rep account, features being enabled, features deliberately disabled, number of applicable tests completed, unresolved blockers, responsible reviewer, launch decision, and rollback owner. Attach or link your internal test log rather than copying sensitive customer information into the sign-off.

If you discover a broken checkout interaction or wrong order authorization, do not launch that action on the assumption that early customers will help test it. Reserve operationally destructive tests for controlled, explicit merchant approval.

## After approval: launch narrowly and monitor what actually happens

Launch the approved feature set and verify it in the same published storefront and selected channels that you tested. Confirm the live state of the widget and the relevant `AI Answering` switches rather than inferring them from successful simulations.

Monitor early conversations for missing sources, unresolved product questions, incorrect handoffs, and unexpected cart behavior. Assign someone to review and correct failures promptly. If a problem threatens customer data, money, or order integrity, disable the affected automation or channel while investigating under your own incident process.

Rep's analytics can help merchants inspect engagement, commerce events, and support-related activity. They can help identify where a launch needs improvement. However, an attributed order or reported “AI-generated revenue” is not by itself proof that Rep AI caused additional revenue. Establishing incremental lift would require an appropriate experimental design; it is not what this pre-launch checklist certifies.

Keep your test scenarios alive as a regression suite. A product-feed change, policy update, new Sales Skill, new support integration, or theme switch can invalidate a previously passing result. Rerun the affected checks after meaningful configuration changes and periodically review the cases that carry the highest customer or financial risk.

## Final checklist: launch the behavior you have actually verified

Testing Rep AI before launch is best understood as a controlled sequence: **simulate the conversation, verify the store and integrations, and authorize only the functionality that passes**. Use the private testing tools to find answer errors; use Skill previews to understand configured behavior; use a controlled storefront to confirm real display and cart flows; and require separate approval before any test that could alter Shopify orders or connected systems.

A practical launch decision is not based on a fictional vendor certification score or a single flawless chatbot demo. It is based on documented outcomes, resolved blockers, a realistic support fallback, and a merchant who understands which features are live. When those conditions are met, begin with the approved scope, watch the results closely, and rerun your tests as the store or Rep AI configuration changes.

## Frequently asked questions about testing Rep AI before launch

### 

#### 1\. Can I test Rep AI without making the assistant live?

Yes. ****Test & Train** provides a private environment for examining what the AI would say. The Chat simulator lets you test answers, product carousels, follow-up questions, sources, and corrections. The Email and Meta simulators can also show responses while live AI Answering is disabled, provided their relevant integrations are connected. This does not mean every Shopify action in every kind of preview is guaranteed to be harmless; distinguish message simulation from operational actions.

#### 2\. Where do I find Rep AI Test & Train?

The documented path is **`AI Training → Test & Train`** in the Rep console. The Chat tab is the starting point for website conversations. Email and Meta tabs depend on supported connected integrations. If you see a greyed-out tab, check the corresponding helpdesk or social connection rather than assuming the capability is permanently unavailable.

#### 3\. How do I know whether an AI answer is correct?

Compare the answer with an independent source from your actual Shopify store or policy documentation. Use ****Sources**below the simulated answer to inspect what the AI relied on. A well-written answer can still be wrong. Check prices, stock, sizes, delivery restrictions, refund conditions, and whether it admits uncertainty when the source does not provide a reliable conclusion.

#### 4\. What should I do when a Test & Train answer is wrong?

Use ****Correct the AI**, update the relevant FAQ or instruction, and save the change. Then reset and repeat the same scenario to verify the revised result. Test a follow-up and an alternative phrasing as well. Record the failure and correction in the acceptance log, especially when the question concerns a customer policy or a transaction-sensitive action.

#### 5\. Does a successful chat simulation mean the widget is published?

No. The widget's display depends on the relevant Shopify theme, App Embed settings, and the assistant's actual launch state. Confirm that the App Embed is enabled and ****saved on the published theme**, then check the storefront in a real browser. A theme preview or an answer in Test & Train does not establish that shoppers can see and use the live chat.

#### 6\. Why might a proactive Sales Skill not appear during my test visit?

The Skill may not match that page, visitor audience, trigger, or frequency setting. Rep also documents ****A/B holdout sessions** in which proactive approaches intentionally do not appear. Use the Skill's ****Preview Conversation** to inspect the message and separately verify eligibility rules. One missing prompt during ordinary browsing is not enough to diagnose a fault.

#### 7\. Can I test automated order cancellation on an existing customer's order?

Do not use a real customer's order as an experiment. An eligible cancellation may change the actual Shopify record, and other order Skills may produce refunds or fulfillment consequences. Test the explanation, refusal, and handoff first. If the business needs an end-to-end action test, create and approve a dedicated safe scenario with the people responsible for Shopify operations and payments.

#### 8\. Is a Support Skill preview guaranteed to be a sandbox for connected APIs?

No universal sandbox guarantee was established in the reviewed documentation. A preview may help you inspect the conversation, but you should not assume that every external tool or data-changing action is neutralized. For custom APIs, use approved test endpoints or strictly non-mutating scenarios unless your technical team has verified the environment's behavior.

#### 9\. How should I test Rep AI human handoff?

Test an explicit request for a person, a case the AI cannot answer, and an ineligible automated action. Review the documented handoff form, the required email field, and the selected helpdesk. To establish that a real ticket arrives in the correct destination, run a separately authorized end-to-end test with your support team; a chatbot message saying “I will connect you” is not proof of delivery.

#### 10\. Can I preview Rep AI email answers before switching on email automation?

Yes, with a supported helpdesk connected. The ****Email** tab in Test & Train can simulate responses even when live ****AI Answering** remains off. Test subject lines, multi-turn cases, and attachments where relevant. Review what the AI actually read, then separately decide whether to enable answering on the production email integration.

#### 11\. Can I test Instagram, Facebook, or WhatsApp before publishing real replies?

With the relevant Meta account connected, the Meta simulator can show what the AI would say while AI Answering is disabled. Rep's social tooling also describes an Instagram comment preview that does not post the simulated comment. Do not assume every Meta surface behaves identically; channel activation and human takeover need their own checks, especially for WhatsApp.

#### 12\. Should I test the Shopify cookie banner with Rep AI?

Yes, if you enable ****Use my store's cookie banner** for the supported standard Shopify widget. Test accepted, rejected, partial, and unanswered consent with a fresh browser session. The documented chat experience requires both analytics and marketing consent in that configuration. Passing this interaction test does not substitute for an independent privacy-compliance review.

#### 13\. How many test scenarios do I need before launch?

There is no universally documented official Rep AI pass count. This guide recommends ****30 candidate scenarios** across knowledge, catalog, cart, Sales Skills, Support Skills, channels, and storefront experience. Adapt the suite to your actual launch scope. A small store without Meta or order automation can mark those tests N/A, while a store enabling refunds may need additional controlled tests and stricter approvals.

#### 14\. What if some scenarios pass but others are blocked?

Do not convert a blocked test into a pass. Decide whether the associated capability can remain ****disabled** while the rest of the assistant launches safely. Record why it was blocked, the owner, and the condition needed to resume it. An unresolved data disclosure, unauthorized transaction, or broken consent restriction should prevent activation of the affected behavior until corrected.