Key Azure Serverless Services
As a follow up to my previous blog on the Era of Serverless Computing, here I share some key Azure services commonly utilised in serverless computing. Within this blog I will also describe our involvement in an interesting use case for one of our utility customers. As mentioned in my previous blog, a serverless computing solution means the onus to design, manage and maintain the underlying infrastructure is effectively removed for you. In the case of Azure, this is undertaken by Microsoft.
Serverless computing components in Azure
Within Microsoft Azure, there are a plethora of serverless computing services at our disposal. I will just provide an overview of seven key services. To learn more about the entire Azure serverless platform, see azure.com/Serverless
Azure Functions (Serverless Compute)
Logic Apps (Serverless Workflow Orchestration)
Azure Logic Apps allows developers to visually create automated scalable workflows that integrate with apps and data sources, both in the cloud and on-premises. There are over 200 pre-built connectors available from within a connection gallery that can massively assist in development of process workflows. Every logic app is started with a trigger and then the logic app instance runs the actions within a workflow. You may not even need to write code but if it is required, code can be run as an Azure function on demand within a Logic App.
Event Grid (Serverless Events)
Azure Event Grid is a message routing service. It provides a single place to manage and listen for events from various sources (subscribers) and send (publish) or multicast information to various destinations. Event Grid is built into the fabric of Azure. This means events can be detected from many sources. Examples of events that can be detected are:
- data being written into Azure storage
- new subscriptions being created
- new record being added to a database.
Azure Event Hubs (Serverless Ingestion)
Azure Event Hubs is an ingestion service capable of receiving and processing millions of events per second. It is often used as an “on ramp” for Big Data, where the data is typically coming from event publishers such as IoT devices, user data, traffic from applications or web services such as Twitter. Events are captured into Blob storage or an Azure Data Lake and held or buffered for a retention period of up to 7 days. Typically, a retention period of 24 hours is used.
Azure Storage (Serverless Storage)
Azure Storage provides a fully managed service that is highly available, scalable and redundant. In relation to serverless solutions Azure Storage offers four key services (there are a few others):
- Blob – Used for unstructured data
- File – Used for distributed File system
- Queue – Used for durable queues could use Event Hubs though
- Table – No SQL and used for key-value storage
Azure Cosmos DB (Database)
Cosmos DB is Microsoft’s globally distributed, massively scalable, multi-model database service. It is being touted by many, as the database for the serverless computing era. Azure Cosmos DB’s database engine natively supports DocumentDB’s SQL dialect, MongoDB API, Gremlin (graph) API, and Azure Table Storage (key value) APIs to name a few. Importantly from a serverless architecture perspective, it has native integration with Azure Functions. Silversands sees Azure Cosmos DB as being the go to database for serverless applications. You can try Cosmos DB for free and get hands on with it at Try Azure Cosmos DB for Free.
Even though Cosmos DB is marketed as globally distributed, multi-node, highly scalable, highly available service, don’t let this put you off using it within your potentially non-global serverless solution.
Azure Stream Analytics (Event Processing)
Azure Stream Analytics is a service that enables action on and analysis of in motion data. It is used to examine high volumes of data to look for patterns, extract data, trends, etc. These patterns or data can be used to trigger other processes or workflows.
Stream Analytics is often integrated with Event hubs and IoT hubs. If interested, you can see how we have used Azure Stream Analytics in an internal PoC displaying light and temperature levels at the Silversands office.
Use Case – Ingestion of Raw Sensor Data (IoT) and Data Visualisation
Silversands has been working with a large utility company which has thousands of Internet of Things (IoT) devices that make up their supervisory control and data acquisition (SCADA) systems. We were asked to look at how the data is processed, transformed into semi structure data and then stored in a suitable repository.
This was a PoC to demonstrate a potential architecture for the ingestion of data and provide ideas on how the data could be interrogated by their line of business application (asset management system) and data scientists. It also provided a vision on how the data could be used to bring about cost savings, improvements to quality of service and potential ways data could be visualised.
As access to the live SCADA system, and therefore the IoT data, was restricted, the initial task was to create a process by which we could emulate the IoT data in a newly defined JSON format. For the PoC it was requested that we write the emulated data to Azure blog storage to emulate a potential production scenario (not directly into Event Hub or IoT Hub as you would typically do). We then developed a series of Azure functions to pick up this data and send it to the Event Hub. The JSON documents were then manipulated into a required format for storage. Table storage was used to look up key value information to append to the JSON files. Finally, the manipulated IoT data in JSON format was written into a desired partition in Azure Cosmos DB (Document DB).
Once the data was in Cosmos DB it could be utilised for various activities. The Cosmos DB change feed was leveraged to create a trigger which wrote information into the asset management system.
Getting the data into a structure format was another requirement of the PoC. Azure Data Factory was utilised to extract data on a scheduled basis into an Azure SQL (could potentially use Azure Data Warehouse).
Finally, the customer wanted to visualise the data. Power BI was used to visualise the structured and unstructured data.
As a further extension to this PoC we worked directly with Microsoft to link up R notebooks on Azure Databricks (in preview at time of writing this blog) to demonstrate a potential mechanism by which the data scientists could undertake detailed and very complex analysis of the IoT data.
How can Silversands help?
Hopefully this blog has provided you with a high-level overview of some of the key serverless Azure services and potentially where some of these services could be utilised within serverless computing solutions. If you need any assistance with Azure development, designing your next serverless application or want to find out more about Azure in general then please complete the enquiry form below. We would love to speak to you and help you at any point along your Azure journey.