{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bfbb9b2d",
   "metadata": {},
   "source": [
    "#### Purpose of the script\n",
    "\n",
    "Metadata can be very useful in ChatLLM projects. One of the most common ways to extract metadata is to look at the `file_path` column on the feature group section and try to find metadata that can be extracted directly from there.\n",
    "\n",
    "This script utilises LLM's to do just that. The process is as follows:\n",
    "1. Loads a FG that is of type \"documentset\"\n",
    "2. Finds some discreet examples of file paths\n",
    "3. Provides a sample SQL code that can be used to extract useful metadata from the file_path.\n",
    "\n",
    "Please note that the model might not do a perfect job in extracting useful metadata. Human sensibility can help iterate and make the response better, or you might just choose to directly alter the SQL that is being generated. Regardless, it can provide a good starting point for extracting metadata from file_paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b427da7",
   "metadata": {},
   "outputs": [],
   "source": [
    "FG_TABLE = 'YOUR_DOCUMENTS_FEATURE_GROUP'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c849eef-4ec9-4cd6-b201-7078c8af1816",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-20T18:08:28.593869Z",
     "iopub.status.busy": "2024-11-20T18:08:28.593102Z",
     "iopub.status.idle": "2024-11-20T18:10:05.111986Z",
     "shell.execute_reply": "2024-11-20T18:10:05.111313Z",
     "shell.execute_reply.started": "2024-11-20T18:08:28.593835Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import abacusai\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cluster import KMeans\n",
    "client = abacusai.ApiClient()\n",
    "\n",
    "df = client.describe_feature_group_by_table_name(FG_TABLE).load_as_pandas()\n",
    "\n",
    "def get_diverse_samples(df, n_samples=100):\n",
    "    # Convert file paths to a more normalized form and remove the filename\n",
    "    paths = df['file_path'].apply(lambda x: x.lower().replace('\\\\', '/'))\n",
    "    # Get just the directory paths by removing the last component\n",
    "    dir_paths = paths.apply(lambda x: '/'.join(x.split('/')[:-1]))\n",
    "    \n",
    "    # Create TF-IDF vectors from the directory paths\n",
    "    vectorizer = TfidfVectorizer(\n",
    "        analyzer='char',\n",
    "        ngram_range=(3, 3),  # Use character trigrams\n",
    "        max_features=2000\n",
    "    )\n",
    "    \n",
    "    path_vectors = vectorizer.fit_transform(dir_paths)\n",
    "    \n",
    "    # Determine number of clusters\n",
    "    n_clusters = min(n_samples, len(df))\n",
    "    \n",
    "    # Perform clustering\n",
    "    kmeans = KMeans(n_clusters=n_clusters, random_state=42)\n",
    "    clusters = kmeans.fit_predict(path_vectors)\n",
    "    \n",
    "    # Create a dictionary to store samples from each cluster\n",
    "    samples = []\n",
    "    \n",
    "    # For each cluster, select one random sample\n",
    "    for cluster_id in range(n_clusters):\n",
    "        cluster_mask = clusters == cluster_id\n",
    "        cluster_indices = np.where(cluster_mask)[0]\n",
    "        \n",
    "        if len(cluster_indices) > 0:\n",
    "            # Randomly select one sample from this cluster\n",
    "            selected_idx = np.random.choice(cluster_indices)\n",
    "            samples.append(df.iloc[selected_idx])\n",
    "    \n",
    "    # Convert to DataFrame\n",
    "    result_df = pd.DataFrame(samples)\n",
    "    \n",
    "    # If we need more samples to reach n_samples, add random ones\n",
    "    if len(result_df) < n_samples:\n",
    "        remaining = n_samples - len(result_df)\n",
    "        additional = df.sample(n=remaining)\n",
    "        \n",
    "        # Instead of using drop_duplicates, we'll use index-based deduplication\n",
    "        combined_indices = pd.Index(result_df.index).union(additional.index)\n",
    "        result_df = df.loc[combined_indices]\n",
    "        \n",
    "        # If we still need more samples after deduplication\n",
    "        if len(result_df) < n_samples:\n",
    "            more_samples = df.drop(result_df.index).sample(n=n_samples - len(result_df))\n",
    "            result_df = pd.concat([result_df, more_samples])\n",
    "    \n",
    "    return result_df\n",
    "\n",
    "# Example usage:\n",
    "diverse_samples = get_diverse_samples(df, n_samples=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d65316d-95b8-467f-a69f-99ef9c165ae4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-20T18:25:58.645002Z",
     "iopub.status.busy": "2024-11-20T18:25:58.644413Z",
     "iopub.status.idle": "2024-11-20T18:25:58.649060Z",
     "shell.execute_reply": "2024-11-20T18:25:58.648350Z",
     "shell.execute_reply.started": "2024-11-20T18:25:58.644974Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "system_message = \"\"\"\n",
    "Here are the revised behavior instructions:\n",
    "\n",
    "---\n",
    "\n",
    "**Objective:**  \n",
    "Your task is to assist in crafting a SQL query to extract metadata from file paths. This metadata will support a RAG-based application by identifying patterns that enhance our approach. We aim to extract metadata that is meaningful but not overly detailed, focusing primarily on the document type.\n",
    "\n",
    "**Instructions:**  \n",
    "1. **Analyze File Paths:**  \n",
    "   Review the provided example file paths to determine the most relevant metadata to extract. The goal is to identify patterns that can inform the RAG application.\n",
    "\n",
    "2. **Create SQL Query:**  \n",
    "   Develop a SQL query that effectively extracts the necessary metadata. Use the example below as a guide, but adapt it based on the file paths you receive. The query should be logical and straightforward, avoiding unnecessary complexity.\n",
    "\n",
    "3. **Example SQL Query:**  \n",
    "\n",
    "```\n",
    "   SELECT *, \n",
    "          CASE WHEN UPPER(split(file_path, '/')[4]) LIKE 'GENERAL SERVICES' THEN UPPER(split(file_path, '/')[5])\n",
    "               WHEN UPPER(split(file_path, '/')[4]) LIKE 'SPECIFIC SERVICES' THEN UPPER(split(file_path, '/')[5])\n",
    "               ELSE UPPER(split(file_path, '/')[4])\n",
    "          END AS services_type,\n",
    "          UPPER(split(file_path, '/')[3]) AS document_type\n",
    "   FROM TABLE\n",
    "```\n",
    "4. **Adaptation:** \n",
    "    In the example, the query extracts metadata based on specific conditions. You may find that a simpler approach is sufficient. Consider creating 3-4 metadata columns if necessary, ensuring they provide valuable insights without being overly granular.\n",
    "\n",
    "5. **Be Sensible:**\n",
    "    Use your judgment to determine the most effective way to extract metadata. The complexity of the query should match the complexity of the file paths and the needs of the application. \n",
    "    If a metadata column has the same values across the board, then we don't care about it. Furthermore, avoid creating metadata columns that would extract the final documents name as metadata. We don't want that at all.\n",
    "\"\"\"\n",
    "r = client.evaluate_prompt(prompt = str(diverse_samples['file_path'].unique()), system_message = system_message)\n",
    "print(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fecc230-c09a-4229-8b8f-90a41961b831",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
