QA Innovative Solutions

IoT Application Landscape

There are many verticals which will be impacted by the emerging IoT. These domains can be classified based on the type of network availability, coverage, scale, heterogeneity, repeatability, user involvement, and impact.

IoT Key Components

IoT building blocks consist of three major components –

THINGS

Embedded objects residing within products, which collect data from Sensors and Actuators and send them to the backend systems.

COMPUTING INFRASTRUCTURE

This component receives data from different things and applications and provides different business views of KPIs and performance parameters with the help of analytical tools, thus helping the consumer/ enterprise to take business-critical decisions.

IoT Testing Coverage

Each IoT scope comes with a list of challenges that the testers have to deal with while testing. These issues have their own levels of severity, based on which the testers generally prioritize the test cases.

The testing team has to focus on core components of the IoT system to treat functional and connectivity testing as important elements of overall IoT product testing. This testing also has to be prioritized based on the nature of the IoT product and the dependency. The following type of testing is usually performed across the IoT products before their release to market.

• Functional Testing: To check the way the consumer wants the output, based on specific inputs given to the IoT app
• Compatibility Testing: Categories verify and validate the possible combination of device versions, communication protocol versions, mobile devices, resolutions, and mobile OS version
• Usability Testing: Verify user experience of IoT app, with respect to its usage, visibility of text, appeal, and usefulness of the content to the end user
• Network Testing: Verify the IoT app with different network connections and ensure app to sync with all different backend combination protocol
• Security Testing: Validate privacy of data, reliability of IoT app, verification, availability, and authorization are the factors that need to be considered. Also verifies if the IoT app uses any
• weak password or missing data encryption. Verify apps followed network security standards and authentication mechanism to authenticate of the required app.
• Performance Testing: To check the overall performance of the IoT app and validate the response time based on different user loads, optimize the code to improve the performance, verify with different scenarios like low battery, less memory, switch between different networks.
• Services Testing: Check the IoT app enabled with different service requests with backend system with online and offline mode. Also verifies request and response of system.
• Operational Testing: To verify behavior of IoT app when battery fully discharged when updated version app get installed or for any interruption of message or call received.

Performance Testing for IoT Applications

IoT Key differentiators in Performance Testing – Some of the key differences between traditional PT and IoT PT:

Because of these differences, IoT poses a lot of Challenges to Performance Engineers.

IoT does not have standard protocol set to establish the connectivity between IoT application and devices. These protocols are continuously evolving with IoT applications. Since these are new technologies, protocols, and current performance testing tools may or may not support them.

IoT Performance Testing Tools

Malaria	MQTTLens	MQTT Stress	MQTT SPY	JMeter CoAP
JMeter MQTT	Gatling MQTT Plug-In	Neoload	SmartBear LoadUI NG	HPE StormRunner MQTT
HPE LoadRunner 12.55	Performance Center	Copper (CoAP Firefox Add-In)	Californium JAVA CoAP library	Locust

measure performance, it is necessary to simulate real-world workload models which can be designed using:

• Business requirements
• Historic data and future requirements
• Type of Devices
• Network Conditions
• Usage pattern
• Geographic spread

Test your MQTT (Message Queue Telemetry Transport) protocol-based IoT system with LoadRunner

MQTT is a simple, lightweight, publish/subscribe messaging protocol designed for constrained devices and low-bandwidth, high-latency or unreliable networks. MQTT is ideal for machine-to-machine (M2M) and Internet of Things (IoT) communications, as well as for mobile applications where bandwidth and battery power are at a premium. MQTT focuses on minimizing network bandwidth and device resource requirements, while attempting to ensure reliability and some degree of assurance of delivery.

MQTT (MQ Telemetry Transport) as publish/subscribe messaging protocol. In 2014, the MQTT protocol became an open standard maintained by the OASIS group. Nowadays it is considered to be one of the most commonly used protocols in the IoT industry.

The network of an IoT system can be quite complex and the variety of client devices is very large. The more complex an IoT system is, the more important it is to perform load tests.

By performing these tests, we can make sure the system can handle all the messages—without losing any data during data processing and/or execution. A system with a various collection of devices (clients) might be hard to load test manually. 

So, Microfocus decided to add support for load testing IoT devices in the recent Microfocus LoadRunner 12.55 release, by including the MQTT protocol.

The benefits of adding MQTT protocol:

• Detect service-capability limits for particular configurations of subscribers/publishers
• Monitor internal broker statistics (through $SYS-type topic)
• Check how the broker handles connectivity problems
• Test different topics

Although the MQTT protocol is not secure on its own, it can be wrapped with SSL/TLS which is supported in LoadRunner. The creation of a multiple-protocol script is also supported. MQTT can be integrated into a recorded Web – HTTP/HTML script.

Test Script Development Approach – Supported MQTT features in LoadRunner 12.55

The MQTT protocol script supports both subscriber and publisher mode, allowing you to control the Quality of Service and configure the automated connection restoring and keep-alive features (if needed). The protocol supports both Windows and Linux Load Generators.

How does it actually work using LoadRunner 12.55?

By creating a load test with LoadRunner, you can monitor how your MQTT broker handles the situation of a heavy load.

• First, you’ll need to create a test script in VuGen (Virtual User Generator). This script implements clients which connect to the broker—either as a publisher or a subscriber.
• Then, creating a test scenario in Controller simulates multiple users based on the test script, running either locally or on remote machines, for a pre-defined time period.

Creating an MQTT test script in VuGen is quite straightforward. Select “New Script and Solution” > “Mobile and IoT” > “MQTT (Internet of Things)”.

Creating an MQTT test script in VuGen is quite straightforward. Select “New Script and Solution” > “Mobile and IoT” > “MQTT (Internet of Things)”.

Script Generation- Ready-to-use MQTT script template
The default, single-protocol MQTT script includes actions that are pre-populated with commented out steps (shown below).

Working with multi-protocol scripts

You can test scenarios in which your script uses both of the following:

• MQTT to communicate between the client and the broker
• Web – HTTP/HTML to communicate between the broker and the web application

You prepare your script by recording the steps on your web application. Then you manually add MQTT steps as needed. The pre-populated MQTT script is not included when working with a multiple protocol script, but you can copy/paste from the ready-to-use MQTT script template.

Enable support for multi protocol

• Create a multiple-protocol script that includes MQTT and Web – HTTP/HTML.
• Record a web script.
  • Recording a web script overwrites globals.h, removing the default support for MQTT steps.
• Restore support for MQTT steps:
  • Open globals.h (located in the Solution Explorer under Extra Files).
  • Add #include “MqttApi.h” to the Include Files section.
• Add MQTT steps to the script.

Execution and Analysis

Execute your test script in a load test scenario using Controller which is created by using MQTT. MQTT scripts will report the following measurements by default:

An example of a load test result in Controller