diff --git a/.github/workflows/deploy-bicep.yml b/.github/workflows/deploy-bicep.yml new file mode 100644 index 0000000..ddd2a77 --- /dev/null +++ b/.github/workflows/deploy-bicep.yml @@ -0,0 +1,26 @@ +name: Deploy Azure Infrastructure with Bicep + +on: + push: + branches: + - main + workflow_dispatch: + +jobs: + deploy-infrastructure: + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up Azure CLI + uses: azure/cli@v2.1.0 + with: + azcliversion: 'latest' + inlineScript: | + # Log in to Azure + az login --service-principal -u ${{ secrets.AZURE_CLIENT_ID }} -p ${{ secrets.AZURE_CLIENT_SECRET }} --tenant ${{ secrets.AZURE_TENANT_ID }} + + # Deploy Bicep Template + az deployment group create --resource-group --template-file infrastructure/azure-resources.bicep diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..83fe30e --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +.vs/CQRS-Pattern-Azure/config/applicationhost.config +.vs/ +app-service-api/bin/Debug/net8.0/app-service-api.pdb +app-service-api/bin/ +app-service-api/obj/Debug/net8.0/app-service-api.csproj.CoreCompileInputs.cache +app-service-api/obj/app-service-api.csproj.nuget.g.targets +app-service-api/obj/Debug/net8.0/.NETCoreApp,Version=v8.0.AssemblyAttributes.cs +app-service-api/obj/ +update-processor/.vscode/ +query-handler/.vscode/extensions.json +command-handler/.vscode/ diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..1523df6 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024 Igor Iric + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..78c596d --- /dev/null +++ b/README.md @@ -0,0 +1,116 @@ +# Index Table Pattern on Azure + +This repository demonstrates the **Index Table Pattern** on Azure, which uses Azure Table Storage to store data and Azure AI Search for efficient indexing and searching of that data. The architecture is designed for scenarios where quick and scalable search capabilities are needed over structured data. + +## πŸ—οΈ Architectural Overview + +The **Index Table Pattern** architecture consists of the following components: + +1. **Azure Function**: Handles data updates and triggers indexing in Azure AI Search. +2. **Azure Table Storage**: Stores the data in a scalable, low-cost NoSQL store. +3. **Azure AI Search**: Indexes the data stored in Azure Table Storage for fast search capabilities. +4. **Application Insights**: Monitors and collects telemetry data from the Azure Function. + +### πŸ“Š Architectural Diagram + +```mermaid +graph TD + Client["Client"] -->|Data Update| AzureFunction["Azure Function"] + AzureFunction -->|Updates| AzureTableStorage["Azure Table Storage"] + AzureFunction -->|Indexes| AzureAISearch["Azure AI Search"] + Client -->|Search Request| AzureAISearch + + subgraph Monitoring + AppInsights["Application Insights"] + end + + AzureFunction -->|Logs & Telemetry| AppInsights +``` + +## πŸ“‚ Repository Structure + +``` +/index-table-pattern +β”‚ +β”œβ”€β”€ README.md # Root README with architecture overview and getting started +β”œβ”€β”€ LICENSE # MIT License +β”‚ +β”œβ”€β”€ infrastructure +β”‚ β”œβ”€β”€ README.md # README for Infrastructure deployment +β”‚ β”œβ”€β”€ azure-resources.bicep # Bicep template for all Azure resources +β”‚ └── .github/workflows/deploy-bicep.yml # GitHub Action to deploy Azure resources +β”‚ +└── azure-functions + β”œβ”€β”€ README.md # README for Index Table Function + β”œβ”€β”€ IndexTableFunction.csproj # C# project file for Index Table Function + β”œβ”€β”€ IndexTableFunction.cs # Main code for Index Table Function + └── .github/workflows/deploy-index-function.yml # GitHub Action to deploy Index Table Function +``` + +## πŸš€ Getting Started + +### Step 1: Deploy the Infrastructure + +1. **Navigate to the `infrastructure` Folder**: + - Go to the **`infrastructure`** folder and follow the instructions in the [Infrastructure README](infrastructure/README.md) to deploy the required Azure resources using the Bicep template and GitHub Actions. + +2. **Run the GitHub Action**: + - The GitHub Actions workflow **`deploy-bicep.yml`** will automatically deploy the Azure resources defined in the Bicep template. + +3. **Verify Deployment**: + - After the deployment completes, verify that the Azure resources (Storage Account, Function App, AI Search, Application Insights) are properly created in your Azure subscription. + +### Step 2: Deploy the Azure Function + +1. **Navigate to the `azure-functions` Folder**: + - Go to the **`azure-functions`** folder and follow the instructions in the [Function README](azure-functions/README.md) to deploy the Azure Function using GitHub Actions. + +2. **Run the GitHub Action**: + - The GitHub Actions workflow **`deploy-index-function.yml`** will automatically build and deploy the Azure Function. + +### Step 3: Test the Azure Function + +1. **Use Postman or Any HTTP Client**: + - Follow the testing instructions in the [Function README](azure-functions/README.md) to send HTTP POST requests to the Azure Function to update data and verify that it is correctly stored in Azure Table Storage and indexed in Azure AI Search. + +## πŸ’‘ How It Works + +1. **Data Update**: + - The **Azure Function** receives HTTP POST requests with JSON data, updates the data in **Azure Table Storage**, and then triggers indexing in **Azure AI Search**. + +2. **Search Capabilities**: + - **Azure AI Search** provides search capabilities over the data stored in Azure Table Storage, enabling quick and efficient searches. + +3. **Monitoring**: + - **Application Insights** collects telemetry data from the Azure Function, allowing you to monitor its performance, track errors, and gain insights into usage patterns. + +## πŸ” Key Points to Remember + +- **Environment Variables**: Ensure all required environment variables are properly set in your Azure Function App: + - **`AzureWebJobsStorage`**: Connection string for the Azure Storage Account. + - **`AzureAISearchEndpoint`**: Endpoint URL for your Azure AI Search service. + - **`AzureAISearchApiKey`**: API key for your Azure AI Search service. +- **Monitor with Application Insights**: Check Application Insights to monitor the function's execution and gather telemetry data. + +## πŸ“Š Monitoring and Logging with Application Insights + +This repository uses **Azure Application Insights** to monitor and collect telemetry data from the Azure Function. Application Insights helps to: + +- Track request rates, response times, and failure rates. +- Monitor the performance of the Azure Function. +- Diagnose failures and exceptions. +- Gain insights into the usage patterns and overall health of the system. + +### How to View Application Insights Data + +1. Go to the **Azure Portal**. +2. Navigate to **Application Insights** and select the **`indexTableAppInsights`** resource. +3. Use the available tools to explore logs, requests, failures, dependencies, and custom metrics. + +## πŸ“„ License + +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. + +## πŸ™Œ Contributing + +Contributions are welcome! Please open an issue or submit a pull request for any improvements or suggestions. diff --git a/azure-functions/.github/workflows/deploy-index-function.yml b/azure-functions/.github/workflows/deploy-index-function.yml new file mode 100644 index 0000000..0ff2731 --- /dev/null +++ b/azure-functions/.github/workflows/deploy-index-function.yml @@ -0,0 +1,33 @@ +name: Deploy Index Table Function + +on: + push: + branches: + - main + workflow_dispatch: + +jobs: + build-and-deploy: + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up .NET + uses: actions/setup-dotnet@v2 + with: + dotnet-version: '8.0.x' + + - name: Build the project + run: dotnet build azure-functions/index-table-function/IndexTableFunction.csproj --configuration Release + + - name: Publish the project + run: dotnet publish azure-functions/index-table-function/IndexTableFunction.csproj --configuration Release --output ./output + + - name: Deploy to Azure Function + uses: Azure/functions-action@v1 + with: + app-name: 'indexTableFunctionApp' + package: './output' + publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }} diff --git a/azure-functions/.gitignore b/azure-functions/.gitignore new file mode 100644 index 0000000..ff5b00c --- /dev/null +++ b/azure-functions/.gitignore @@ -0,0 +1,264 @@ +## Ignore Visual Studio temporary files, build results, and +## files generated by popular Visual Studio add-ons. + +# Azure Functions localsettings file +local.settings.json + +# User-specific files +*.suo +*.user +*.userosscache +*.sln.docstates + +# User-specific files (MonoDevelop/Xamarin Studio) +*.userprefs + +# Build results +[Dd]ebug/ +[Dd]ebugPublic/ +[Rr]elease/ +[Rr]eleases/ +x64/ +x86/ +bld/ +[Bb]in/ +[Oo]bj/ +[Ll]og/ + +# Visual Studio 2015 cache/options directory +.vs/ +# Uncomment if you have tasks that create the project's static files in wwwroot +#wwwroot/ + +# MSTest test Results +[Tt]est[Rr]esult*/ +[Bb]uild[Ll]og.* + +# NUNIT +*.VisualState.xml +TestResult.xml + +# Build Results of an ATL Project +[Dd]ebugPS/ +[Rr]eleasePS/ +dlldata.c + +# DNX +project.lock.json +project.fragment.lock.json +artifacts/ + +*_i.c +*_p.c +*_i.h +*.ilk +*.meta +*.obj +*.pch +*.pdb +*.pgc +*.pgd +*.rsp +*.sbr +*.tlb +*.tli +*.tlh +*.tmp +*.tmp_proj +*.log +*.vspscc +*.vssscc +.builds +*.pidb +*.svclog +*.scc + +# Chutzpah Test files +_Chutzpah* + +# Visual C++ cache files +ipch/ +*.aps +*.ncb +*.opendb +*.opensdf +*.sdf +*.cachefile +*.VC.db +*.VC.VC.opendb + +# Visual Studio profiler +*.psess +*.vsp +*.vspx +*.sap + +# TFS 2012 Local Workspace +$tf/ + +# Guidance Automation Toolkit +*.gpState + +# ReSharper is a .NET coding add-in +_ReSharper*/ +*.[Rr]e[Ss]harper +*.DotSettings.user + +# JustCode is a .NET coding add-in +.JustCode + +# TeamCity is a build add-in +_TeamCity* + +# DotCover is a Code Coverage Tool +*.dotCover + +# NCrunch +_NCrunch_* +.*crunch*.local.xml +nCrunchTemp_* + +# MightyMoose +*.mm.* +AutoTest.Net/ + +# Web workbench (sass) +.sass-cache/ + +# Installshield output folder +[Ee]xpress/ + +# DocProject is a documentation generator add-in +DocProject/buildhelp/ +DocProject/Help/*.HxT +DocProject/Help/*.HxC +DocProject/Help/*.hhc +DocProject/Help/*.hhk +DocProject/Help/*.hhp +DocProject/Help/Html2 +DocProject/Help/html + +# Click-Once directory +publish/ + +# Publish Web Output +*.[Pp]ublish.xml +*.azurePubxml +# TODO: Comment the next line if you want to checkin your web deploy settings +# but database connection strings (with potential passwords) will be unencrypted +#*.pubxml +*.publishproj + +# Microsoft Azure Web App publish settings. Comment the next line if you want to +# checkin your Azure Web App publish settings, but sensitive information contained +# in these scripts will be unencrypted +PublishScripts/ + +# NuGet Packages +*.nupkg +# The packages folder can be ignored because of Package Restore +**/packages/* +# except build/, which is used as an MSBuild target. +!**/packages/build/ +# Uncomment if necessary however generally it will be regenerated when needed +#!**/packages/repositories.config +# NuGet v3's project.json files produces more ignoreable files +*.nuget.props +*.nuget.targets + +# Microsoft Azure Build Output +csx/ +*.build.csdef + +# Microsoft Azure Emulator +ecf/ +rcf/ + +# Windows Store app package directories and files +AppPackages/ +BundleArtifacts/ +Package.StoreAssociation.xml +_pkginfo.txt + +# Visual Studio cache files +# files ending in .cache can be ignored +*.[Cc]ache +# but keep track of directories ending in .cache +!*.[Cc]ache/ + +# Others +ClientBin/ +~$* +*~ +*.dbmdl +*.dbproj.schemaview +*.jfm +*.pfx +*.publishsettings +node_modules/ +orleans.codegen.cs + +# Since there are multiple workflows, uncomment next line to ignore bower_components +# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) +#bower_components/ + +# RIA/Silverlight projects +Generated_Code/ + +# Backup & report files from converting an old project file +# to a newer Visual Studio version. Backup files are not needed, +# because we have git ;-) +_UpgradeReport_Files/ +Backup*/ +UpgradeLog*.XML +UpgradeLog*.htm + +# SQL Server files +*.mdf +*.ldf + +# Business Intelligence projects +*.rdl.data +*.bim.layout +*.bim_*.settings + +# Microsoft Fakes +FakesAssemblies/ + +# GhostDoc plugin setting file +*.GhostDoc.xml + +# Node.js Tools for Visual Studio +.ntvs_analysis.dat + +# Visual Studio 6 build log +*.plg + +# Visual Studio 6 workspace options file +*.opt + +# Visual Studio LightSwitch build output +**/*.HTMLClient/GeneratedArtifacts +**/*.DesktopClient/GeneratedArtifacts +**/*.DesktopClient/ModelManifest.xml +**/*.Server/GeneratedArtifacts +**/*.Server/ModelManifest.xml +_Pvt_Extensions + +# Paket dependency manager +.paket/paket.exe +paket-files/ + +# FAKE - F# Make +.fake/ + +# JetBrains Rider +.idea/ +*.sln.iml + +# CodeRush +.cr/ + +# Python Tools for Visual Studio (PTVS) +__pycache__/ +*.pyc \ No newline at end of file diff --git a/azure-functions/README.md b/azure-functions/README.md new file mode 100644 index 0000000..b4375a5 --- /dev/null +++ b/azure-functions/README.md @@ -0,0 +1,115 @@ +# Index Table Function + +This Azure Function is part of the **Index Table Pattern** architecture on Azure. It is designed to handle data updates by storing data in **Azure Table Storage** and indexing the data in **Azure Cognitive Search** for efficient search capabilities. + +## πŸ“‘ Overview + +The **Index Table Function** performs the following actions: + +1. **Receives Data Update Requests**: The function receives HTTP POST requests containing JSON data. +2. **Updates Azure Table Storage**: The function updates the data in Azure Table Storage, using it as a scalable NoSQL store. +3. **Indexes Data in Azure Cognitive Search**: After updating Azure Table Storage, the function indexes the data in Azure Cognitive Search to enable fast and efficient search capabilities. +4. **Performs Search Requests**: The function receives HTTP GET requests with search queries and returns matching results from Azure Cognitive Search. + +## πŸ› οΈ Configuration + +Ensure the following environment variables are set in your Azure Function App configuration: + +- **`AzureWebJobsStorage`**: Connection string for the Azure Storage Account used for Table Storage. +- **`AzureCognitiveSearchEndpoint`**: The endpoint URL for your Azure Cognitive Search service (e.g., `https://.search.windows.net`). +- **`AzureCognitiveSearchApiKey`**: The API key for your Azure Cognitive Search service. + +## πŸ”§ Prerequisites + +- Azure Function Core Tools (if running locally) +- .NET 8 SDK +- Azure CLI (for deployment) + +## πŸ§ͺ How to Test the Function + +You can test the function locally or on Azure using **Postman**. + +### Testing `UpdateData` Locally + +1. **Run the Azure Function Locally**: + - Open a terminal or command prompt and run: + + ```bash + func start + ``` + + - The function will start at `http://localhost:7071`. + +2. **Open Postman**: + - Create a new **POST** request. + +3. **Configure the Postman Request**: + - **Method**: `POST` + - **URL**: `http://localhost:7071/api/update` + - **Headers**: + - **Content-Type**: `application/json` + - **Body**: + - Select **Raw** and choose **JSON** format. + - Example JSON data: + + ```json + { + "PartitionKey": "SamplePartition", + "RowKey": "SampleRow", + "Name": "John Doe", + "Email": "john.doe@example.com" + } + ``` + +4. **Send the Request**: + - Click **Send** to execute the request. + +5. **Check the Response**: + - You should receive a response with `200 OK` and a message indicating that the data was updated and indexed successfully. + +### Testing `SearchData` Locally + +1. **Open Postman**: + - Create a new **GET** request. + +2. **Configure the Postman Request**: + - **Method**: `GET` + - **URL**: `http://localhost:7071/api/search?search=John Doe` (replace `John Doe` with the desired search query) + +3. **Send the Request**: + - Click **Send** to execute the request. + +4. **Check the Response**: + - You should receive a response with `200 OK` and the search results in JSON format. + +### Testing on Azure + +1. **Deploy the Azure Function**: + - Deploy the function to Azure using the provided GitHub Actions workflow. + +2. **Open Postman**: + - Create a **POST** or **GET** request, depending on the function you are testing. + +3. **Configure the Postman Request**: + - **URL for `UpdateData`**: `https://.azurewebsites.net/api/update` + - **URL for `SearchData`**: `https://.azurewebsites.net/api/search?search=John Doe` + +4. **Include Function Key (If Required)**: + - If your function requires a function key: + - Add it as a query parameter or header as described earlier. + +5. **Send the Request** and **Check the Response**. + +## πŸ” Key Points to Remember + +- **Ensure Environment Variables**: Make sure all required environment variables are properly set. +- **Check Azure Resources**: Confirm that Azure Table Storage and Azure Cognitive Search are correctly configured. +- **Monitor Logs**: Use Azure Monitor or local logs to debug and trace function execution. + +## πŸ“„ License + +This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details. + +## πŸ™Œ Contributions + +Contributions are welcome! Please open an issue or submit a pull request for any improvements or suggestions. diff --git a/azure-functions/index-table-function/.gitignore b/azure-functions/index-table-function/.gitignore new file mode 100644 index 0000000..ff5b00c --- /dev/null +++ b/azure-functions/index-table-function/.gitignore @@ -0,0 +1,264 @@ +## Ignore Visual Studio temporary files, build results, and +## files generated by popular Visual Studio add-ons. + +# Azure Functions localsettings file +local.settings.json + +# User-specific files +*.suo +*.user +*.userosscache +*.sln.docstates + +# User-specific files (MonoDevelop/Xamarin Studio) +*.userprefs + +# Build results +[Dd]ebug/ +[Dd]ebugPublic/ +[Rr]elease/ +[Rr]eleases/ +x64/ +x86/ +bld/ +[Bb]in/ +[Oo]bj/ +[Ll]og/ + +# Visual Studio 2015 cache/options directory +.vs/ +# Uncomment if you have tasks that create the project's static files in wwwroot +#wwwroot/ + +# MSTest test Results +[Tt]est[Rr]esult*/ +[Bb]uild[Ll]og.* + +# NUNIT +*.VisualState.xml +TestResult.xml + +# Build Results of an ATL Project +[Dd]ebugPS/ +[Rr]eleasePS/ +dlldata.c + +# DNX +project.lock.json +project.fragment.lock.json +artifacts/ + +*_i.c +*_p.c +*_i.h +*.ilk +*.meta +*.obj +*.pch +*.pdb +*.pgc +*.pgd +*.rsp +*.sbr +*.tlb +*.tli +*.tlh +*.tmp +*.tmp_proj +*.log +*.vspscc +*.vssscc +.builds +*.pidb +*.svclog +*.scc + +# Chutzpah Test files +_Chutzpah* + +# Visual C++ cache files +ipch/ +*.aps +*.ncb +*.opendb +*.opensdf +*.sdf +*.cachefile +*.VC.db +*.VC.VC.opendb + +# Visual Studio profiler +*.psess +*.vsp +*.vspx +*.sap + +# TFS 2012 Local Workspace +$tf/ + +# Guidance Automation Toolkit +*.gpState + +# ReSharper is a .NET coding add-in +_ReSharper*/ +*.[Rr]e[Ss]harper +*.DotSettings.user + +# JustCode is a .NET coding add-in +.JustCode + +# TeamCity is a build add-in +_TeamCity* + +# DotCover is a Code Coverage Tool +*.dotCover + +# NCrunch +_NCrunch_* +.*crunch*.local.xml +nCrunchTemp_* + +# MightyMoose +*.mm.* +AutoTest.Net/ + +# Web workbench (sass) +.sass-cache/ + +# Installshield output folder +[Ee]xpress/ + +# DocProject is a documentation generator add-in +DocProject/buildhelp/ +DocProject/Help/*.HxT +DocProject/Help/*.HxC +DocProject/Help/*.hhc +DocProject/Help/*.hhk +DocProject/Help/*.hhp +DocProject/Help/Html2 +DocProject/Help/html + +# Click-Once directory +publish/ + +# Publish Web Output +*.[Pp]ublish.xml +*.azurePubxml +# TODO: Comment the next line if you want to checkin your web deploy settings +# but database connection strings (with potential passwords) will be unencrypted +#*.pubxml +*.publishproj + +# Microsoft Azure Web App publish settings. Comment the next line if you want to +# checkin your Azure Web App publish settings, but sensitive information contained +# in these scripts will be unencrypted +PublishScripts/ + +# NuGet Packages +*.nupkg +# The packages folder can be ignored because of Package Restore +**/packages/* +# except build/, which is used as an MSBuild target. +!**/packages/build/ +# Uncomment if necessary however generally it will be regenerated when needed +#!**/packages/repositories.config +# NuGet v3's project.json files produces more ignoreable files +*.nuget.props +*.nuget.targets + +# Microsoft Azure Build Output +csx/ +*.build.csdef + +# Microsoft Azure Emulator +ecf/ +rcf/ + +# Windows Store app package directories and files +AppPackages/ +BundleArtifacts/ +Package.StoreAssociation.xml +_pkginfo.txt + +# Visual Studio cache files +# files ending in .cache can be ignored +*.[Cc]ache +# but keep track of directories ending in .cache +!*.[Cc]ache/ + +# Others +ClientBin/ +~$* +*~ +*.dbmdl +*.dbproj.schemaview +*.jfm +*.pfx +*.publishsettings +node_modules/ +orleans.codegen.cs + +# Since there are multiple workflows, uncomment next line to ignore bower_components +# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) +#bower_components/ + +# RIA/Silverlight projects +Generated_Code/ + +# Backup & report files from converting an old project file +# to a newer Visual Studio version. Backup files are not needed, +# because we have git ;-) +_UpgradeReport_Files/ +Backup*/ +UpgradeLog*.XML +UpgradeLog*.htm + +# SQL Server files +*.mdf +*.ldf + +# Business Intelligence projects +*.rdl.data +*.bim.layout +*.bim_*.settings + +# Microsoft Fakes +FakesAssemblies/ + +# GhostDoc plugin setting file +*.GhostDoc.xml + +# Node.js Tools for Visual Studio +.ntvs_analysis.dat + +# Visual Studio 6 build log +*.plg + +# Visual Studio 6 workspace options file +*.opt + +# Visual Studio LightSwitch build output +**/*.HTMLClient/GeneratedArtifacts +**/*.DesktopClient/GeneratedArtifacts +**/*.DesktopClient/ModelManifest.xml +**/*.Server/GeneratedArtifacts +**/*.Server/ModelManifest.xml +_Pvt_Extensions + +# Paket dependency manager +.paket/paket.exe +paket-files/ + +# FAKE - F# Make +.fake/ + +# JetBrains Rider +.idea/ +*.sln.iml + +# CodeRush +.cr/ + +# Python Tools for Visual Studio (PTVS) +__pycache__/ +*.pyc \ No newline at end of file diff --git a/azure-functions/index-table-function/.vscode/extensions.json b/azure-functions/index-table-function/.vscode/extensions.json new file mode 100644 index 0000000..dde673d --- /dev/null +++ b/azure-functions/index-table-function/.vscode/extensions.json @@ -0,0 +1,5 @@ +{ + "recommendations": [ + "ms-azuretools.vscode-azurefunctions" + ] +} \ No newline at end of file diff --git a/azure-functions/index-table-function/Index Table Pattern on Azure.sln b/azure-functions/index-table-function/Index Table Pattern on Azure.sln new file mode 100644 index 0000000..fc35877 --- /dev/null +++ b/azure-functions/index-table-function/Index Table Pattern on Azure.sln @@ -0,0 +1,30 @@ +ο»Ώ +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio Version 17 +VisualStudioVersion = 17.5.002.0 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "azure-functions", "azure-functions", "{C78388A3-D2C5-47F4-A3B8-215EB08D5E98}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "IndexTableFunction", "azure-functions\index-table-function\IndexTableFunction.csproj", "{89799645-1FCE-4683-8B69-6F410931CE76}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Release|Any CPU = Release|Any CPU + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {89799645-1FCE-4683-8B69-6F410931CE76}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {89799645-1FCE-4683-8B69-6F410931CE76}.Debug|Any CPU.Build.0 = Debug|Any CPU + {89799645-1FCE-4683-8B69-6F410931CE76}.Release|Any CPU.ActiveCfg = Release|Any CPU + {89799645-1FCE-4683-8B69-6F410931CE76}.Release|Any CPU.Build.0 = Release|Any CPU + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection + GlobalSection(NestedProjects) = preSolution + {89799645-1FCE-4683-8B69-6F410931CE76} = {C78388A3-D2C5-47F4-A3B8-215EB08D5E98} + EndGlobalSection + GlobalSection(ExtensibilityGlobals) = postSolution + SolutionGuid = {384F91F3-7F6D-4AC0-9867-02FB0C2A9BB1} + EndGlobalSection +EndGlobal diff --git a/azure-functions/index-table-function/IndexTable.cs b/azure-functions/index-table-function/IndexTable.cs new file mode 100644 index 0000000..d766313 --- /dev/null +++ b/azure-functions/index-table-function/IndexTable.cs @@ -0,0 +1,119 @@ +using Microsoft.Azure.Functions.Worker; +using Microsoft.Azure.Functions.Worker.Http; +using Microsoft.Extensions.Logging; +using Microsoft.Azure.Cosmos.Table; +using Azure.Search.Documents; +using Azure; +using Newtonsoft.Json; +using Azure.Search.Documents.Models; + +namespace index_table_function +{ + public class IndexTableFunction + { + private readonly ILogger _logger; + private readonly CloudTableClient _tableClient; + private readonly SearchClient _searchClient; + private readonly string _tableName = "dataIndexTable"; + + public IndexTableFunction(ILogger logger) + { + _logger = logger; + + string storageConnectionString = Environment.GetEnvironmentVariable("AzureWebJobsStorage") + ?? throw new ArgumentNullException(nameof(storageConnectionString), "Environment variable 'AzureWebJobsStorage' is not set."); + string searchEndpoint = Environment.GetEnvironmentVariable("AzureAISearchEndpoint") + ?? throw new ArgumentNullException(nameof(searchEndpoint), "Environment variable 'AzureAISearchEndpoint' is not set."); + string searchApiKey = Environment.GetEnvironmentVariable("AzureAISearchApiKey") + ?? throw new ArgumentNullException(nameof(searchApiKey), "Environment variable 'AzureAISearchApiKey' is not set."); + CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString); + _tableClient = storageAccount.CreateCloudTableClient(new TableClientConfiguration()); + + _searchClient = new SearchClient(new Uri(searchEndpoint), "my-index", new AzureKeyCredential(searchApiKey)); + } + + [Function("UpdateData")] + public async Task UpdateData([HttpTrigger(AuthorizationLevel.Function, "post", Route = "update")] HttpRequestData req) + { + _logger.LogInformation("Received data update request."); + + var requestBody = await new StreamReader(req.Body).ReadToEndAsync(); + + dynamic? data = JsonConvert.DeserializeObject(requestBody); + if (data == null) + { + var badResponse = req.CreateResponse(System.Net.HttpStatusCode.BadRequest); + await badResponse.WriteStringAsync("Invalid or empty data provided."); + return badResponse; + } + + try + { + // Update Azure Table Storage + var table = _tableClient.GetTableReference(_tableName); + var insertOrMergeOperation = TableOperation.InsertOrMerge(new DynamicTableEntity("PartitionKey", "RowKey", "*", data)); + await table.ExecuteAsync(insertOrMergeOperation); + + _logger.LogInformation("Data updated in Azure Table Storage."); + + // Index data in Azure AI Search + var searchIndexData = new IndexDocumentsBatch(); + searchIndexData.Actions.Add(IndexDocumentsAction.Upload(new SearchDocument(data))); + + await _searchClient.IndexDocumentsAsync(searchIndexData); + + _logger.LogInformation("Data indexed in Azure AI Search."); + + var response = req.CreateResponse(System.Net.HttpStatusCode.OK); + await response.WriteStringAsync("Data updated and indexed successfully."); + return response; + } + catch (Exception ex) + { + _logger.LogError($"An error occurred: {ex.Message}"); + var errorResponse = req.CreateResponse(System.Net.HttpStatusCode.InternalServerError); + await errorResponse.WriteStringAsync("An error occurred while processing your request."); + return errorResponse; + } + } + + [Function("SearchData")] + public async Task SearchData([HttpTrigger(AuthorizationLevel.Function, "get", Route = "search")] HttpRequestData req) + { + _logger.LogInformation("Received search request."); + + // Get the search query from query parameters + string query = req.Url.Query.TrimStart('?'); + if (string.IsNullOrEmpty(query)) + { + var badResponse = req.CreateResponse(System.Net.HttpStatusCode.BadRequest); + await badResponse.WriteStringAsync("Query parameter is missing."); + return badResponse; + } + + try + { + // Perform the search using Azure Cognitive Search + var options = new SearchOptions + { + Size = 10, // Limit the results to 10 items for this example + IncludeTotalCount = true + }; + + var searchResults = await _searchClient.SearchAsync(query, options); + + // Prepare the response + var searchResponse = req.CreateResponse(System.Net.HttpStatusCode.OK); + await searchResponse.WriteStringAsync(JsonConvert.SerializeObject(searchResults.Value.GetResults())); + return searchResponse; + } + catch (Exception ex) + { + _logger.LogError($"Error performing search: {ex.Message}"); + var errorResponse = req.CreateResponse(System.Net.HttpStatusCode.InternalServerError); + await errorResponse.WriteStringAsync("An error occurred while performing the search."); + return errorResponse; + } + } + } +} diff --git a/azure-functions/index-table-function/Program.cs b/azure-functions/index-table-function/Program.cs new file mode 100644 index 0000000..9389455 --- /dev/null +++ b/azure-functions/index-table-function/Program.cs @@ -0,0 +1,13 @@ +using Microsoft.Azure.Functions.Worker; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.DependencyInjection; + +var host = new HostBuilder() + .ConfigureFunctionsWebApplication() + .ConfigureServices(services => { + services.AddApplicationInsightsTelemetryWorkerService(); + services.ConfigureFunctionsApplicationInsights(); + }) + .Build(); + +host.Run(); diff --git a/azure-functions/index-table-function/Properties/launchSettings.json b/azure-functions/index-table-function/Properties/launchSettings.json new file mode 100644 index 0000000..694ec12 --- /dev/null +++ b/azure-functions/index-table-function/Properties/launchSettings.json @@ -0,0 +1,9 @@ +{ + "profiles": { + "index_table_function": { + "commandName": "Project", + "commandLineArgs": "--port 7292", + "launchBrowser": false + } + } +} \ No newline at end of file diff --git a/azure-functions/index-table-function/host.json b/azure-functions/index-table-function/host.json new file mode 100644 index 0000000..ee5cf5f --- /dev/null +++ b/azure-functions/index-table-function/host.json @@ -0,0 +1,12 @@ +{ + "version": "2.0", + "logging": { + "applicationInsights": { + "samplingSettings": { + "isEnabled": true, + "excludedTypes": "Request" + }, + "enableLiveMetricsFilters": true + } + } +} \ No newline at end of file diff --git a/azure-functions/index-table-function/index-table-function.csproj b/azure-functions/index-table-function/index-table-function.csproj new file mode 100644 index 0000000..b8cfaf6 --- /dev/null +++ b/azure-functions/index-table-function/index-table-function.csproj @@ -0,0 +1,35 @@ +ο»Ώ + + net8.0 + v4 + Exe + enable + enable + index_table_function + + + + + + + + + + + + + + + + + PreserveNewest + + + PreserveNewest + Never + + + + + + \ No newline at end of file diff --git a/infrastructure/README.md b/infrastructure/README.md new file mode 100644 index 0000000..185dbd4 --- /dev/null +++ b/infrastructure/README.md @@ -0,0 +1,111 @@ +# Infrastructure Deployment for Index Table Pattern + +This folder contains the **Bicep template** and GitHub Actions workflow for deploying the necessary Azure resources for the **Index Table Pattern** on Azure. This architecture pattern uses Azure Table Storage for storing data and Azure AI Search for indexing and searching the data efficiently. + +## πŸ“‘ Overview of Services + +The following Azure services are deployed by the Bicep template: + +1. **Azure Storage Account**: Provides scalable, low-cost storage for Table Storage. +2. **Azure Table Storage**: A NoSQL storage service for storing structured data. +3. **Azure Function App**: Hosts the Azure Function that handles data updates and triggers indexing in Azure AI Search. +4. **Azure AI Search**: A fully managed search-as-a-service solution to build rich search experiences over your data. +5. **Application Insights**: A monitoring service to collect telemetry data from the Azure Function. + +### πŸš€ Deployed Resources + +1. **Azure Storage Account**: + - **Purpose**: Provides the underlying storage for Azure Table Storage. + - **Configuration**: + - **Type**: Standard Locally Redundant Storage (LRS) + - **Access Tier**: Hot + - **Minimum TLS Version**: TLS 1.2 + +2. **Azure Table Storage**: + - **Purpose**: Stores structured data in a NoSQL format. + - **Configuration**: + - **Table Name**: `dataIndexTable` + - **Partitioning**: Uses `PartitionKey` and `RowKey` for data organization. + +3. **Azure Function App**: + - **Purpose**: Hosts the function that updates Azure Table Storage and triggers indexing in Azure AI Search. + - **Configuration**: + - **Runtime**: .NET 8 (Isolated Mode) + - **Plan**: Consumption Plan (Y1 Dynamic) + - **Application Settings**: + - **`FUNCTIONS_WORKER_RUNTIME`**: `dotnet` + - **`APPINSIGHTS_INSTRUMENTATIONKEY`**: Instrumentation Key from Application Insights + - **`AzureWebJobsStorage`**: Connection string for Azure Storage Account + - **`AzureAISearchEndpoint`**: Endpoint for Azure AI Search + - **`AzureAISearchApiKey`**: API key for Azure AI Search + +4. **Azure AI Search**: + - **Purpose**: Provides search and indexing capabilities for data stored in Azure Table Storage. + - **Configuration**: + - **SKU**: Standard (can be adjusted as needed) + - **Partition Count**: 1 + - **Replica Count**: 1 + +5. **Application Insights**: + - **Purpose**: Collects telemetry data from the Azure Function for monitoring and diagnostics. + - **Configuration**: + - **Retention**: 30 days + - **Application Type**: Web + +## πŸ“‚ Files in This Folder + +- **`azure-resources.bicep`**: Bicep template file defining all the necessary Azure resources. +- **`.github/workflows/deploy-bicep.yml`**: GitHub Actions workflow to automate the deployment of the Azure infrastructure. + +## πŸ› οΈ How to Deploy the Infrastructure + +### Prerequisites + +1. **Azure Subscription**: An active Azure account with appropriate permissions. +2. **GitHub Repository**: Fork or create a repository to store your code and workflows. +3. **Azure CLI**: Installed and configured to manage Azure resources. + +### Steps to Deploy + +1. **Add Required Secrets to GitHub**: + - Go to your repository’s **Settings > Secrets and variables > Actions > New repository secret**. + - Add the following secrets: + - **`AZURE_CLIENT_ID`**: Your Azure service principal client ID. + - **`AZURE_CLIENT_SECRET`**: Your Azure service principal client secret. + - **`AZURE_TENANT_ID`**: Your Azure tenant ID. + +2. **Run the GitHub Action**: + - The GitHub Actions workflow will automatically trigger on a push to the `main` branch or can be manually triggered. + - To manually trigger, go to the **Actions** tab in your GitHub repository and select **Deploy Azure Infrastructure with Bicep**. + +3. **Monitor the Deployment**: + - Go to the **Actions** tab in your GitHub repository. + - Select the **Deploy Azure Infrastructure with Bicep** workflow to monitor the deployment progress. + - The workflow will use the Bicep template to deploy the resources to your Azure subscription. + +## πŸ” Verification After Deployment + +1. **Azure Portal**: + - Go to the [Azure Portal](https://portal.azure.com/) and verify that the following resources are deployed: + - **Storage Account** with Table Storage named `dataIndexTable`. + - **Function App** named `indexTableFunctionApp`. + - **Azure AI Search** service. + - **Application Insights** for monitoring. + +2. **Application Insights**: + - Check the Application Insights resource for telemetry data to ensure that it’s properly configured and collecting data from your Azure Function. + +## πŸ“Š Post-Deployment Configuration + +- **Verify Environment Variables**: + - Ensure that all required environment variables (`AzureWebJobsStorage`, `AzureAISearchEndpoint`, `AzureAISearchApiKey`) are set correctly in your Azure Function App. +- **Run Tests**: + - Follow the instructions in the [Function README](../azure-functions/README.md) to test the deployed function using Postman or any other HTTP client. + +## πŸ“„ License + +This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details. + +## πŸ™Œ Contributions + +Contributions are welcome! Please open an issue or submit a pull request for any improvements or suggestions. diff --git a/infrastructure/azure-resources.bicep b/infrastructure/azure-resources.bicep new file mode 100644 index 0000000..8c0e8b1 --- /dev/null +++ b/infrastructure/azure-resources.bicep @@ -0,0 +1,109 @@ +// Parameters +param location string = resourceGroup().location +param storageAccountName string = 'indextablestorage' +param functionAppName string = 'indexTableFunctionApp' +param appInsightsName string = 'indexTableAppInsights' +param searchServiceName string = 'indexTableSearchService' +param searchServiceSku string = 'standard' + +// Azure Storage Account +resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = { + name: storageAccountName + location: location + sku: { + name: 'Standard_LRS' + } + kind: 'StorageV2' + properties: { + minimumTlsVersion: 'TLS1_2' + } +} + +resource tableService 'Microsoft.Storage/storageAccounts/tableServices@2023-01-01' = { + name: 'default' + parent: storageAccount +} + +resource table 'Microsoft.Storage/storageAccounts/tableServices/tables@2023-01-01' = { + name: 'dataIndexTable' + parent: tableService + properties: { + signedIdentifiers: [ + { + id: 'unique-id' + accessPolicy: { + startTime: '2024-09-12T00:00:00Z' + expiryTime: '2024-09-13T00:00:00Z' + permission: 'r' + } + } + ] + } +} +// Azure Function App Plan +resource functionAppPlan 'Microsoft.Web/serverfarms@2022-03-01' = { + name: 'indexTableFunctionAppPlan' + location: location + sku: { + name: 'Y1' + tier: 'Dynamic' + } +} + + +// Application Insights +resource appInsights 'Microsoft.Insights/components@2020-02-02-preview' = { + name: appInsightsName + location: location + kind: 'web' + properties: { + Application_Type: 'web' + } +} + +// Azure Function App +resource functionApp 'Microsoft.Web/sites@2022-03-01' = { + name: functionAppName + location: location + kind: 'functionapp' + properties: { + serverFarmId: functionAppPlan.id + siteConfig: { + appSettings: [ + { + name: 'FUNCTIONS_WORKER_RUNTIME' + value: 'dotnet' + } + { + name: 'APPINSIGHTS_INSTRUMENTATIONKEY' + value: appInsights.properties.InstrumentationKey + } + { + name: 'AzureWebJobsStorage' + value: storageAccount.properties.primaryEndpoints.table + } + { + name: 'AzureCognitiveSearchEndpoint' + value: 'https://{searchServiceName}.search.windows.net' + } + { + name: 'AzureCognitiveSearchApiKey' + value: searchService.listKeys().primaryKey + } + ] + } + } +} + +// Azure Cognitive Search Service +resource searchService 'Microsoft.Search/searchServices@2024-06-01-preview' = { + name: searchServiceName + location: location + sku: { + name: searchServiceSku + } + properties: { + partitionCount: 1 + replicaCount: 1 + } +}