Dify Chatbot Testing Notes

Source: Dev.to

This note will be updated periodically.

No.1 Chatbot Memory Settings

It is the memory setting for the LLM. The setting can be turned on or off, with an additional window‑size sub‑setting.

When the memory is on without a defined window size, the chatbot sends all previous prompts from the session to the LLM in the workflow. This can severely alter test results, especially during unit testing. While memory is essential for providing contextual data to the underlying LLM, it can unintentionally affect behavior.

For example, when testing the DeepSeek model, the history prompt that instructs the model to act as a GPT model influences subsequent role‑based tests, causing DeepSeek to mimic GPT behavior.

Takeaway:
When performing chatbot testing, explicitly set an appropriate memory window size (or disable memory) to avoid unintended side effects. This issue was discovered during black‑box testing through debugging.