Azure Function Event Hub Trigger and Reliability

In this post I am going to talk about an EventHubTrigger and data reliability in azure function. Also how to handle data loss in case of issues in azure function.

Generic scenario people use for EventHubTrigger is receiving device telemetry data from IoTHub and massaging that data and pass it to down system like database, rest api call for storing the data. Which can be used later for display or analytics purpose.

The current implementation of EventHubTrigger doesn’t provide any Fault handling mechanism. So once data is received in azure function the checkpoint maintained in azure function storage account against each partition is increased regardless your function completes successfully or not. Even if you throw an exception from the function as output the azure treats the host completed and increments the checkpoint.

This is critical issue considering reliability of data. Consider a scenario if database or rest api is down but your azure function is running. The data is ultimately lost unless you go back and manually change the checkpoint files to previous successful checkpoint.

The content of the blob file look like below. See the pointers for offset and sequence number for a partition. The file is automatically updated once function instance completes the execution regardless of error.

{“PartitionId”:”1″,”Owner”:”asfasfasf”,”Token”:”asafsa”,”Epoch”:259,”Offset”:”266318360104“,”SequenceNumber”:2432227}

How to handle issues during Data Processing

  • If it is a random failure from database or rest api you can add retry logic with some delay like 5 sec with each retry.
  • If your database is down or rest api is down due to network issues or load issues or any other issue then the other option currently available is killing the function instance using standard process libraries.

Killing Function execution : You can use Process.GetCurrentProcess().Kill() from exception handling block which will basically indicate to azure that instance has stopped and it will not increment the pointer.

Process.GetCurrentProcess().Kill();

Drawbacks of this Approach:

  • You will not see anything in logs in azure portal or app-insight
  • You will still pay up the cost of function execution as your function gets triggered again with old and new data again n again.

The issue for logs can be easily handled using alert functionality like sending an email to team that something is went wrong and data is not getting processed. So that team can take decision to stop the azure function until the down systems like database or rest api are restored back to normal position.

To avoid the cost you can disable the azure function until the down systems comes back. I have another blog post which talks about programetically disabling function. -> https://amarplayground.home.blog/2019/01/03/enable-disable-azure-function-using-rest-api/    

If down system has hearbeat kind of thing implemented then enable/disable functionality can be hooked with that to automatically start/stop the function to avoid the manual intervention and data loss.

Until Microsoft azure provides inbuilt support for handling the issues in data processing in azure function and avoid auto incremental checkpoints this can be used as a solution to avoid the data loss.

Another Approach: Another approach can be managing the checkpoint files. The last successfull checkpoint can be maintained in seperate database from azure function. So whenever such situation occurs we can restore the checkpoint in the storage account so the we receive the data back again. But yes provided that you restore the checkpoint files before retention period for data at IoTHub which is default 7 days. I will talk about this in another blog post.

Thank you for reading. Happy coding. If you have any question or suggestion please do not hesitate to post a comment on this blog. I would be happy to discuss with you.

Ref :https://hackernoon.com/reliable-event-processing-in-azure-functions-37054dc2d0fc

Enable Disable Azure Function using Power shell / Rest API

In this post I am going to talk about how we can enable/disable azure function using rest api. I am going to take example of compiled azure function but the logic applies to in portal function too. The logic can be used to update any app setting value.

Basics setting for Azure function

For compiled function the enable or disable can be controlled using attribute called Disable and the value is stored in appsetting file. Default value is false. See below code the attribute Disable added to the function method. The parameter ‘is_disabled’ controls the enable/disable of the function. We can choose any name for this param. You need to have the is_disabled key in appsetting with value true/false.

public static class TestFunction
{
        [Disable("is_disabled")]
        [FunctionName("TestFunction")]
        public static async Task Run([IoTHubTrigger("messages/events", Connection = "IoTEventHub")]EventData[] myIoTHubMessage, ILogger log)
        {
//your logic
  }
}

Appsetting on azure portal. : true means function is disabled false means function is enabled.

Enabling/Disabling Azure function Using code

The need for disabling the azure function can be use case basis but most prominent use case is the down processing is down and you want stop processing the data and reduce your bills and in case of Event Hub Trigger to avoid data loss by stopping the function and avoiding checkpoint increment.

To disable/enable basically we need to change the value in app.setting . It can be done with 3 lines of powershell code. But if you want to automate that and give it to other process so everything runs smooth then rest api is best way. I have tried with logic apps but its complex and lengthy to setup. We are going to use azure function rest api written in powershell to enable/disable intended function.

Powershell Script Standalone 

For using power-shell script you need to install azure cli on your local box which can be downloaded from https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest

Script for updating app setting is_disabled value: The below script can be used to add/update any app-setting key value. You need to have service principle created if you want to run the power-shell script in non interactive mode. Link for creating service principle -> https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

$isdisabled = true # change true or false as per your need.
az login --service-principal --username 'your-app-id' --password 'you-app-key' --tenant 'your-tenant-id'
$keyvalue='is_disabled'+'='+$isdisabled
az functionapp config appsettings set --resource-group 'your-resource-group' --name 'your-site-name' --settings $keyvalue

The above powershell can be run from local box but what if you want to automate this and give this to some other process like down process in azure function controlling the azure function depending on load. In case of high load it disables the function when load in low it enables the function back. For this kind of scenario we can use rest api written in azure function using powershell. I am going with azure function as it is easy and fast to create and test. And if you are wondering if azure function can be written in powershell then answer is yes. See below link -> https://www.brianbunke.com/blog/2018/02/27/powershell-in-azure-functions/ Imp: The function app needs to have run-time version 1 to use powershell scripting.

Azure function Code for enabling/Disabling the other Az Function

# POST method: $req
$requestBody = Get-Content $req -Raw | ConvertFrom-Json
$name = $requestBody.name
$isdisabled = $requestBody.isdisabled
$resourceGroup = $requestBody.resourceGroup

# GET method: each querystring parameter is its own variable
if ($req_query_name) 
{
    $name = $req_query_name  # function name you want to disable
}
if ($req_query_isdisabled) 
{
    $isdisabled = $req_query_isdisabled #flag indicating you want to enable or disable.
}
if ($req_query_resourcegroup) 
{
    $resourceGroup = $req_query_resourcegroup #resource group of function app
}

# login to the azure using service principal
$azureAplicationId ="Your-ServicePrinceple-Id"
$azureTenantId= "Your-Tenant-Id"
$azurePassword = ConvertTo-SecureString "Your-App-Key" -AsPlainText -Force
$psCred = New-Object System.Management.Automation.PSCredential($azureAplicationId , $azurePassword)
Add-AzureRmAccount -Credential $psCred -TenantId $azureTenantId  -ServicePrincipal

# Retrieve the web app setting 
$webApp = Get-AzureRMWebApp -ResourceGroupName $resourceGroup -Name $name
$appSettingList = $webApp.SiteConfig.AppSettings

$hash = @{}
ForEach ($kvp in $appSettingList) {
    $hash[$kvp.Name] = $kvp.Value
}

#update the value required
$hash['is_disabled'] = $isdisabled

#set the application setting.
Set-AzureRMWebApp -ResourceGroupName $resourceGroup -Name $name -AppSettings $hash 

Out-File -Encoding Ascii -FilePath $res -inputObject "App Setting Updated."

You can quickly test using azure portal using Test pane. That’s it from my side happy coding. Thanks for reading and would be happy to answer your questions.