{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f7949b4cd1eb1d5b",
   "metadata": {},
   "source": "# Introduction to Data Transfer (Export)\n\nData transfer in Elevate Data enables you to seamlessly move prepared datasets or imported data tables to Price-It and Underwrite-It. This functionality ensures that the data you have imported, curated and transformed within Elevate Data can be leveraged across the entire Earnix application ecosystem without manual intervention. The transfer process is designed to be secure, efficient, and flexible to meet diverse business needs.\n\nThe data transfer process typically includes the following stages:\n\n\n * Update Type Mapping – Align source data formats with the specific requirements of the target system.\n * Configuration of Target – Specify where Elevate Data should transfer Data to: Destination, location, naming convention.\n * Selection of Dataset/Data table – Choose the specific version of the dataset or data table to be transferred.\n * Follow-up and Completion – Review the status and confirm successful transfer."
  },
  {
   "cell_type": "markdown",
   "id": "b620fc2a-1c6c-43af-9e43-f86bfe459595",
   "metadata": {},
   "source": "## Install the SDK"
  },
  {
   "cell_type": "code",
   "id": "5e9vo2pegoh",
   "source": "!pip install --upgrade earnix-elevate",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b4888073-ffb4-439b-808e-fcb5365f6d31",
   "metadata": {},
   "source": [
    "## Set your Elevate server and credentials\n",
    "\n",
    "In this example we use environment variables to authenticate to Elevate, but you can also inject your credentials using Python arguments to each Service's client, like in the commented example."
   ]
  },
  {
   "cell_type": "code",
   "id": "a33e3871-fbfa-412e-aece-c88bb37a4a4b",
   "metadata": {
    "jupyter": {
     "is_executing": true
    }
   },
   "source": [
    "import os\n",
    "os.environ.setdefault(\"E2_SERVER\", \"YOUR_SERVER\")\n",
    "os.environ.setdefault(\"E2_CLIENT_ID\", \"YOUR_CLIENT_ID\")\n",
    "os.environ.setdefault(\"E2_SECRET_KEY\", \"YOUR_SECRET_KEY\")\n",
    "\n",
    "# Alternatively:\n",
    "# ConnectionService(\n",
    "#     server=\"YOUR_SERVER\",\n",
    "#     client_id=\"YOUR_CLIENT_ID\",\n",
    "#     secret_key=\"YOUR_SECRET_KEY\"\n",
    "# )"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "cd56f9be-5c12-4e8f-ad7d-1838e14aefc1",
   "metadata": {},
   "source": [
    "## Imports and demo preparation"
   ]
  },
  {
   "cell_type": "code",
   "id": "f2140ddd-52ef-4742-96fe-978473da3021",
   "metadata": {},
   "source": "from datetime import datetime\nfrom time import sleep\n\nfrom earnix_elevate import (\n    ConnectionService,\n    CreateExportRequest,\n    CreatePriceItDataTargetRequest,\n    DataTargetService,\n    ExportService,\n    UpdateConnectionTypeMappingRequest,\n)\nfrom earnix_elevate.clients.imx import JobStatus\n\nDEMO_SUFFIX = datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "ef0e93746e697f9b",
   "metadata": {},
   "source": "## Update Price-It Connection Type Mapping\n\nData type mapping is the process of aligning source data formats with the specific requirements of a target system to prevent errors and ensure data integrity.\n\nThe following code demonstrates how to update these mappings to match export data types with Price-It table columns. Once updated, these definitions apply to all future exports using that connection. The process involves fetching a (Price-It) connection by its ID, preparing a request with the new mapping configurations, and executing an update call."
  },
  {
   "cell_type": "code",
   "id": "7ecc9dd8724c448",
   "metadata": {},
   "source": [
    "import os\n",
    "CONNECTION_ID = int(os.environ.get(\"EXPORT_CONNECTION_ID\", \"36602\"))  # use your connection system id\n",
    "\n",
    "connection_service = ConnectionService()\n",
    "connection = connection_service.get_connection(CONNECTION_ID)\n",
    "update_request = UpdateConnectionTypeMappingRequest(\n",
    "    id=connection.id,\n",
    "    lockVersion=connection.lock_version,\n",
    "    typeMapping='{\"AGE\": \"integer\", \"NAME\": \"text\"}',\n",
    "    typeMappingName=\"Simple Mapping\",\n",
    ")\n",
    "connection_service.update_connection_type_mapping(connection.id, update_request)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "28b04b0f-7fcf-435f-b818-9eb9f34ce300",
   "metadata": {},
   "source": "## Create Price-It Data Target\n\nA target is a pre-configured destination for delivering data during the transfer process, defined per connection type. For a Price-It target, it includes parameters such as project path, table path, and table naming conventions. Once created, the target entity is reusable across multiple transfers, ensuring consistency and reducing setup effort.\n\n\nTo create a Price-It target, first create a data target request. Next, initialize a data target service and invoke its creation method with the request. You need to provide an identifier of an existing Price-It connection. The table name may include dynamic fields, that would be replaced with a concrete string during the export. The fields are: source_name, source_version_number and timestamp."
  },
  {
   "cell_type": "code",
   "id": "73e803a6-32f3-49a0-93d2-5acb0dadeda0",
   "metadata": {},
   "source": "data_target_name = os.environ.get(\"EXPORT_DATA_TARGET_NAME\", \"PriceIt Dim Data Target Test \" + DEMO_SUFFIX)\nproject_path = os.environ.get(\"EXPORT_PROJECT_PATH\", \"YOUR_PROJECT_PATH\")\n\ndata_target_request = CreatePriceItDataTargetRequest(\n    name=data_target_name,\n    description=\"PriceIt Dim Desc\",\n    connectionId=connection.id,\n    type=\"price_it\",\n    projectPath=project_path,\n    tablePath=\"Data Tables\\\\\",\n    tableName=\"table_{{source_name}}_{{timestamp}}\",\n)\n\ndata_target_service = DataTargetService()\ndata_target = data_target_service.create_data_target(data_target_request)\n\n# Alternatively, if data target already exists, you can use its ID to fetch it\n# data_target = data_target_service.get_data_target(DATA_TARGET_ID)\n\nprint(f\"\\n{data_target=}\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "a6649983-9984-458f-86a8-5344b101f444",
   "metadata": {},
   "source": "## Export Data Table / Dataset\n\nAfter the data target has been successfully created, the export operation can be performed. To do so, create an export request and use the export service to execute it.\n\nThe export request supports two source types: 'datatable' and 'dataset', representing data tables and datasets, respectively. The source ID should correspond to the identifier of the data table or dataset to be exported."
  },
  {
   "cell_type": "code",
   "id": "1d4acbad-c523-4296-8dbe-cc3b993698fb",
   "metadata": {},
   "source": [
    "DATA_TABLE_ID = int(os.environ.get(\"EXPORT_DATA_TABLE_ID\", \"25852\"))  # use your data table system id\n",
    "\n",
    "export_datatable_request = CreateExportRequest(\n",
    "    targetId=data_target.id,\n",
    "    sourceType=\"datatable\",\n",
    "    sourceId=DATA_TABLE_ID,\n",
    "    sourceVersion=1,\n",
    "    useSample=False,\n",
    "    validateToRelease=True,\n",
    "    castBooleanToInteger=False,\n",
    ")\n",
    "\n",
    "export_service = ExportService()\n",
    "export_response = export_service.export(export_datatable_request)\n",
    "\n",
    "print(f\"\\n{export_response=}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "a8631aa6-cfb5-4fa8-9694-53dde4737428",
   "metadata": {},
   "source": [
    "## Follow-up on the export status of the exported data"
   ]
  },
  {
   "cell_type": "code",
   "id": "6e81557d-a11f-42fb-b43c-fd0a10d0b965",
   "metadata": {},
   "source": "export_id = export_response.export_id\n\nMAX_ATTEMPTS = 180  # Wait max 15 minutes (180 * 5 sec)\nWAIT_INTERVAL = 5  # seconds\nstatus_code = None\n\nfor attempt in range(MAX_ATTEMPTS):\n    try:\n        export_response = export_service.get_export(export_id)\n        status_code = export_response.execution.status if export_response.execution else None\n        if status_code == JobStatus.SUCCEEDED:\n            print(\"\\nThe export was completed successfully\\n\")\n            break\n        elif status_code == JobStatus.FAILED:\n            print(\"\\nThe export operation failed\\n\")\n            break\n    except Exception as ex:\n        print(f\"Exception during status polling: {ex}\")\n    sleep(WAIT_INTERVAL)\n    print(f\"Waiting Export:{export_id}\\n\")\nelse:\n    raise RuntimeError(f\"Timed out waiting for export {export_id} to complete.\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "export-cleanup-md",
   "metadata": {},
   "source": [
    "## Delete the Data Target"
   ]
  },
  {
   "cell_type": "code",
   "id": "export-cleanup-code",
   "metadata": {},
   "outputs": [],
   "execution_count": null,
   "source": [
    "data_target_service.delete_data_target(data_target_id=data_target.id)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}