The first step in ally’s response process is determining the intent behind the user's question. This is achieved using a fine-tuned reasoning model developed by Envision.

1. Intent Recognition: ally analyzes the structure and context of the question to extract its underlying intent.

‍2. Query Understanding: The model discerns whether the user is inquiring about factual information, requesting a visual description, or seeking a specific action.

3. Examples:
‍"Do I need my umbrella today?" → Identifies intent as a weather inquiry.

‍"What am I holding in my hand?" → Recognizes intent as a request for object recognition using the camera.

This reasoning model is integral to ensuring ally accurately interprets user requests before executing any further actions.

Once Ally determines intent, its agentic model dynamically selects the appropriate tools, ensuring precise and relevant responses.

1. Language Model (LLM): Used for answering general knowledge questions, calculations, and queries requiring text-based reasoning. Example: "What is the capital of the Netherlands?" or "What is the square root of 8?"

2. Camera: Used for visual queries where an image needs to be captured and analyzed. Example: "What is this object?"

3. Visual Language Model (VLM): Interprets the contents of images to provide descriptions or extract relevant details.

4. OCR (Optical Character Recognition): Used for reading and extracting text from images and documents. The OCR model, developed by Envision, can:
- Recognize complex text layouts, including headings and document structures.
- Accurately extract textual information for enhanced readability and accessibility.

5. Weather API: Retrieves real-time weather data for location-based queries.

6. Calendar Integration: Fetches scheduled events and reminders when users ask for their agenda.

7. Web Search: When necessary, ally performs a web search to find information beyond its existing knowledge base.

Once ally has gathered the necessary information using the appropriate tools, it applies context-based personalization to refine the final response.

1. User Preferences: ally takes into account personal details stored in the "About You" section. This allows it to tailor responses based on individual preferences, such as food choices or professional background.

2. Location Awareness: ally integrates location data to enhance responses for queries related to weather, local recommendations, and news updates, ensuring relevance

3. Language and Cultural Adjustments: The assistant is aware of the user’s preferred language and adjusts its responses accordingly, incorporating cultural nuances where necessary

4. Response Style Customization: Users can define how they want ally to communicate by configuring preferences in the "About ally" section. This allows responses to be professional, concise, cheerful, or humor-infused, depending on the user's preference.