This post provides guidelines for configuration, system sizing and capacity planning when deploying Caché 2015 and later on a VMware ESXi 5.5 and later environment.

I jump right in with recommendations assuming you already have an understanding of VMware vSphere virtualization platform. The recommendations in this guide are not specific to any particular hardware or site specific implementation, and are not intended as a fully comprehensive guide to planning and configuring a vSphere deployment -- rather this is a check list of best practice configuration choices you can make. I expect that the recommendations will be evaluated for a specific site by your expert VMware implementation team.

A list of other posts in the InterSystems Data Platforms and performance series is here.

Note: This post was updated on 3 Jan 2017 to highlight that VM memory reservations must be set for production database instances to guarantee memory is available for Caché and there will be no swapping or ballooning which will negatively impact database performance. See the section below Memory for more details.

References

The information here is based on experience and reviewing publicly available VMware knowledge base articles and VMware documents for example Performance Best Practices for VMware vSphere and mapping to requirements of Caché database deployments.

Are InterSystems' products supported on ESXi?

It is InterSystems policy and procedure to verify and release InterSystems’ products against processor types and operating systems including when operating systems are virtualised. For specifics see InterSystems support policy and Release Information.

For example: Caché 2016.1 running on Red Hat 7.2 operating system on ESXi on x86 hosts is supported.

Note: If you do not write your own applications you must also check your application vendors support policy.

Supported Hardware

VMware virtualization works well for Caché when used with current server and storage components. Caché using VMware virtualization has been deployed succesfully at customer sites and has been proven in benchmarks for performance and scalability. There is no significant performance impact using VMware virtualization on properly configured storage, network and servers with later model Intel Xeon processors, specifically: Intel Xeon 5500, 5600, 7500, E7-series and E5-series (including the latest E5 v4).

Generally Caché and applications are installed and configured on the guest operating system in the same way as for the same operating system on bare-metal installations.

It is the customers responsibility to check the VMware compatibility guide for the specific servers and storage being used.

Virtualised architecture

I see VMware commonly used in two standard configurations with Caché applications:

• Where primary production database operating system instances are on a ‘bare-metal’ cluster, and VMware is only used for additional production and non-production instances such as web servers, printing, test, training and so on.
• Where ALL operating system instances, including primary production instances are virtualized.

This post can be used as a guide for either scenario, however the focus is on the second scenario where all operating system instances including production are virtualised. The following diagram shows a typical physical server set up for that configuration.

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_201.png">

Figure 1. Simple virtualised Caché architecture

Figure 1 shows a common deployment with a minimum of three physical host servers to provide N+1 capacity and availability with host servers in a VMware HA cluster. Additional physical servers may be added to the cluster to scale resources. Additional physical servers may also be required for backup/restore media management and disaster recovery.

For recommendations specific to VMware vSAN, VMware's Hyper-Converged Infrastructure solution, see the following post: Part 8 Hyper-Converged Infrastructure Capacity and Performance Planning. Most of the recommendations in this post can be applied to vSAN -- with the exception of some of the obvious differences in the Storage section below.

VMWare versions

The following table shows key recommendations for Caché 2015 and later:

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_202.png">

vSphere is a suite of products including vCenter Server that allows centralised system management of hosts and virtual machines via the vSphere client.

This post assumes that vSphere will be used, not the "free" ESXi Hypervisor only version.

VMware has several licensing models; ultimately choice of version is based on what best suits your current and future infrastructure planning.

I generally recommend the "Enterprise" edition for its added features such as Dynamic Resource Scheduling (DRS) for more efficient hardware utilization and Storage APIs for storage array integration (snapshot backups). The VMware web site shows edition comparisons.

There are also Advanced Kits that allow bundling of vCenter Server and CPU licenses for vSphere. Kits have limitations for upgrades so are usually only recommended for smaller sites that do not expect growth.

ESXi Host BIOS settings

The ESXi host is the physical server. Before configuring BIOS you should:

• Check with the hardware vendor that the server is running the latest BIOS
• Check whether there are any server/CPU model specific BIOS settings for VMware.

Default settings for server BIOS may not be optimal for VMware. The following settings can be used to optimize the physical host servers to get best performance. Not all settings in the following table are available on all vendors’ servers.

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_203.png">

Memory

The following key rules should be considered for memory allocation:

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_210.png">

When running multiple Caché instances or other applications on a single physical host VMware has several technologies for efficient memory management such as transparent page sharing (TPS), ballooning, swap, and memory compression. For example when multiple OS instances are running on the same host TPS allows overcommitment of memory without performance degradation by eliminating redundant copies of pages in memory, which allows virtual machines to run with less memory than on a physical machine.

Note: VMware Tools must be installed in the operating system to take advantage of these and many other features of VMware.

Although these features exist to allow for overcommitting memory, the recommendation is to always start by sizing vRAM of all VMs to fit within the physical memory available. Especially important in production environments is to carefully consider the impact of overcommitting memory and overcommit only after collecting data to determine the amount of overcommitment possible. To determine the effectiveness of memory sharing and the degree of acceptable overcommitment for a given Caché instance, run the workload and use Vmware commands resxtop or esxtop to observe the actual savings.

A good reference is to go back and look at the fourth post in this series on memory when planning your Caché instance memory requirements. Especially the section "VMware Virtualisation considerations" where I point out:

Set VMware memory reservation on production systems.

You want tomust avoid any swapping for shared memory so set your production database VMs memory reservation to at least the size of Caché shared memory plus memory for Caché processes and operating system and kernel services. If in doubtReserve the full production database VMs memory (100% reservation) to guarantee memory is available for your Caché instance so there will be no swapping or ballooning which will negatively impact database performance.

Notes: Large memory reservations will impact vMotion operations so it is important to take this into consideration when designing the vMotion/management network. A virtual machine can only be live migrated, or started on another host with Vmware HA if the target host has free physical memory greater than or equal to the size of the reservation. This is especially important for production Caché VMs. For example pay particular attention to HA Admission Control policies.

Ensure capacity planning allows for distribution of VMs in event of HA failover.

For non-production environments (test, train, etc) more aggressive memory overcommitment is possible, however do not over commit Caché shared memory, instead limit shared memory in the Caché instance by having less global buffers. 

Current Intel processor architecture has a NUMA topology. Processors have their own local memory and can access memory on other processors in the same host. Not surprisingly accessing local memory has lower latency than remote. For a discussion of CPU check out the third post in this series including a discussion about NUMA in the comments section.

As noted in the BIOS section above a strategy for optimal performance is to ideally size VMs only up to maximum of number of cores and memory on a single processor. For example if your capacity planning shows your biggest production Caché database VM will be 14 vCPUs and 112 GB memory then consider whether a a cluster of servers with 2x E5-2680 v4 (14-core processor) and 256 GB memory is a good fit.

Ideally size VMs to keep memory local to a NUMA node. But dont get too hung up on this.

If you need a "Monster VM" bigger than a NUMA node that is OK, VMware will manage NUMA for optimal performance. It also important to right-size your VMs and not allocate more resources than are needed (see below).

CPU

The following key rules should be considered for virtual CPU allocation:

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/vmware_best_practice_cpu_20171017.png">

Production Caché systems should be sized based on benchmarks and measurements at live customer sites. For production systems use a strategy of initially sizing the system the same as bare-metal CPU cores and as per best practice monitoring to see if virtual CPUS (vCPUs) can be reduced.

Hyperthreading and capacity planning

A good starting point for sizing production database VMs based on your rules for physical servers is to calculate physical server CPU requirements for the target processor with hyper-threading enabled then simply make the transaltaion:

One physical CPU (includes hyperthreading) = One vCPU (includes hyperthreading).

A common misconception is that hyper-threading somehow doubles vCPU capacity. This is NOT true for physical servers or for logical vCPUs. Hyperthreading on a bare-metal server may give a 30% uplift in performance over the same server without hyperthreading, but this can also be variable depending on the application.

For initial sizing assume is that the vCPU has full core dedication. For example; if you have a 32-core (2x 16-core) E5-2683 V4 server – size for a total of up to 32 vCPU capacity knowing there may be available headroom. This configuration assumes hyper-threading is enabled at the host level. VMware will manage the scheduling between all the applications and VMs on the host. Once you have spent time monitoring the appliaction, operating system and VMware performance during peak processing times you can decide if higher consolidation is possible.

Licencing

In vSphere you can configure a VM with a certain number of sockets or cores. For example, if you have a dual-processor VM (2 vCPUs), it can be configured as two CPU sockets, or as a single socket with two CPU cores. From an execution standpoint it does not make much of a difference because the hypervisor will ultimately decide whether the VM executes on one or two physical sockets. However, specifying that the dual-CPU VM really has two cores instead of two sockets could make a difference for software licenses. Note: Caché license counts the cores (not threads).

Storage

This section applies to the more traditional storage model using a shared storage array. For vSAN recommendations also see the following post: Part 8 Hyper-Converged Infrastructure Capacity and Performance Planning

The following key rules should be considered for storage:

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_205_1.png">

Size storage for performance

Bottlenecks in storage is one of the most common problems affecting Caché system performance, the same is true for VMware vSphere configurations. The most common problem is sizing storage simply for GB capacity, rather than allocating a high enough number of spindles to support expected IOPS. Storage problems can be even more severe in VMware because more hosts can be accessing the same storage over the same physical connections.

VMware Storage overview

VMware storage virtualization can be categorized into three layers, for example:

• The storage array is the bottom layer, consisting of physical disks presented as logical disks (storage array volumes or LUNs) to the layer above.
• The next layer is the virtual environment occupied by vSphere. Storage array LUNs are presented to ESXi hosts as datastores and are formatted as VMFS volumes.
• Virtual machines are made up of files in the datastore and include virtual disks are presented to the guest operating system as disks that can be partitioned and used in file systems.

VMware offers two choices for managing disk access in a virtual machine—VMware Virtual Machine File System (VMFS) and raw device mapping (RDM), both offer similar performance. For simple management VMware generally recommends VMFS, but there may be situations where RDMs are required. As a general recommendation – unless there is a particular reason to use RDM choose VMFS, new development by VMware is directed to VMFS and not RDM.

###Virtual Machine File System (VMFS)

VMFS is a file system developed by VMware that is dedicated and optimized for clustered virtual environments (allows read/write access from several hosts) and the storage of large files. The structure of VMFS makes it possible to store VM files in a single folder, simplifying VM administration. VMFS also enables VMware infrastructure services such as vMotion, DRS and VMware HA.

Operating Systems, applications, and data are stored in virtual disk files (.vmdk files). vmdk files are stored in the Datastore. A single VM can be made up of multiple vmdk files spread over several datastores. As the production VM in the diagram below shows a VM can include storage spread over several data stores. For production systems best performance is achieved with one vmdk file per LUN, for non-production systems (test, training etc) multiple VMs vmdk files can share a datastore and a LUN.

While vSphere 5.5 has a maximum VMFS volume size of 64TB and VMDK size of 62TB when deploying Caché typically multiple VMFS volumes mapped to LUNs on separate disk groups are used to separate IO patterns and improve performance. For example random or sequential IO disk groups or to separate production IO from IO from other environments.

The following diagram shows an overview of an example VMware VMFS storage used with Caché:

Figure 2. Example Caché storage on VMFS

RDM

RDM allows management and access of raw SCSI disks or LUNs as VMFS files. An RDM is a special file on a VMFS volume that acts as a proxy for a raw device. VMFS is recommended for most virtual disk storage, but raw disks might be desirable in some cases. RDM is only available for Fibre Channel or iSCSI storage.

VMware vStorage APIs for Array Integration (VAAI)

For the best storage performance, customers should consider using VAAI-capable storage hardware. VAAI can improve the performance in several areas including virtual machine provisioning and of thin-provisioned virtual disks. VAAI may be available as a firmware update from the array vendor for older arrays.

Virtual Disk Types

ESXi supports multiple virtual disk types:

Thick Provisioned – where space is allocated at creation. There are further types:

• Eager Zeroed – writes 0’s to the entire drive. This increases the time it takes to create the disk, but results in the best performance, even on the first write to each block.
• Lazy Zeroed – writes 0’s as each block is first written to. Lazy zero results in a shorter creation time, but reduced performance the first time a block is written to. Subsequent writes, however, have the same performance as on eager-zeroed thick disks.

Thin Provisioned – where space is allocated and zeroed upon write. There is a higher I/O cost (similar to that of lazy-zeroed thick disks) during the first write to an unwritten file block, but on subsequent writes thin-provisioned disks have the same performance as eager-zeroed thick disks

In all disk types VAAI can improve performance by offloading operations to the storage array. Some arrays also support thin provisioning at the array level, do not thin provision ESXi disks on thin provisioned array storage as there can be conflicts in provisioning and management.

Other Notes

As noted above for best practice use the same strategies as bare-metal configurations; production storage may be separated at the array level into several disk groups:

• Random access for Caché production databases
• Sequential access for backups and journals, but also a place for other non-production storage such as test, train, and so on

Remember that a datastore is an abstraction of the storage tier and, therefore, it is a logical representation not a physical representation of the storage. Creating a dedicated datastore to isolate a particular I/O workload (whether journal or database files), without isolating the physical storage layer as well, does not have the desired effect on performance.

Although performance is key, choice of shared storage depends more on existing or planned infrastructure at site than impact of VMware. As with bare-metal implementations FC SAN is the best performing and is recommended. For FC 8Gbps adapters are the recommended minimum. iSCSI storage is only supported if appropriate network infrastructure is in place, including; minimum 10Gb Ethernet and jumbo frames (MTU 9000) must be supported on all components in the network between server and storage with separation from other traffic.

Use multiple VMware Paravirtual SCSI (PVSCSI) controllers for the database virtual machines or virtual machines with high I/O load. PVSCSI can provide some significant benefits by increasing overall storage throughput while reducing CPU utilization. The use of multiple PVSCSI controllers allows the execution of several parallel I/O operations inside the guest operating system. It is also recommended to separate journal I/O traffic from the database I/O traffic through separate virtual SCSI controllers. As a best practice, you can use one controller for the operating system and swap, another controller for journals, and one or more additional controllers for database data files (depending on the number and size of the database data files).

Aligning file system partitions is a well-known storage best practice for database workloads. Partition alignment on both physical machines and VMware VMFS partitions prevents performance I/O degradation caused by I/O crossing track boundaries. VMware test results show that aligning VMFS partitions to 64KB track boundaries results in reduced latency and increased throughput. VMFS partitions created using vCenter are aligned on 64KB boundaries as recommended by storage and operating system vendors.

Networking

The following key rules should be considered for networking:

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_211_1.png">

As noted above VMXNET adapaters have better capabilities than the default E1000 adapter. VMXNET3 allows 10Gb and uses less CPU where as E1000 is only 1Gb. If there is only 1 gigabit network connections between hosts there is not a lot of difference for client to VM communication. However with VMXNET3 it will allow 10Gb between VMs on the same host, which does make a difference especially in multi-tier deployments or where there is high network IO requirements between instances. This feature should also be taken into consideration when planning affinity and antiaffinity DRS rules to keep VMs on the same or separate virtual switches.

The E1000 use universal drivers that can be used in Windows or Linux. Once VMware Tools is installed on the guest operating system VMXNET virtual adapters can be installed.

The following diagram shows a typical small server configuration with four physical NIC ports, two ports have been configured within VMware for infrastructure traffic: dvSwitch0 for Management and vMotion, and two ports for application use by VMs. NIC teaming and load balancing is used for best throughput and HA.

Figure 3. A typical small server configuration with four physical NIC ports.

Guest Operating Systems

The following are recommended:

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_208.png">

It is very important to load VMware tools in to all VM operating systems and keep the tools current.

VMware Tools is a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine. Without VMware Tools installed in your guest operating system, guest performance lacks important functionality.

Its vital that the time is set correctly on all ESXi hosts - it ends up affecting the Guest VMs. The default setting for the VMs is not to sync the guest time with the host - but at certain times the guest still do sync their time with the host and if the time is out has been known to cause major issues. VMware recommends using NTP instead of VMware Tools periodic time synchronization. NTP is an industry standard and ensures accurate timekeeping in your guest. It may be necessary to open the firewall (UDP 123) to allow NTP traffic.

DNS Configuration

If your DNS server is hosted on virtualized infrastructure and becomes unavailable, it prevents vCenter from resolving host names, making the virtual environment unmanageable -- however the virtual machines themselves keep operating without problem.

<img alt=""src="https://community.intersystems.com/sites/default/files/inline/images/cachebestpractice2016_209.png">

High Availability

High availability is provided by features such as VMware vMotion, VMware Distributed Resource Scheduler (DRS) and VMware High Availability (HA). Caché Database mirroring can also be used to increase uptime.

It is important that Caché production systems are designed with n+1 physical hosts. There must be enough resources (e.g. CPU and Memory) for all the VMs to run on remaining hosts in the event of a single host failure. In the event of server failure if VMware cannot allocate enough CPU and memory resources on the remaining server VMware HA will not restart VMs on the remaining servers.

vMotion

vMotion can be used with Caché. vMotion allows migration of a functioning VM from one ESXi host server to another in a fully transparent manner. The OS and applications such as Caché running in the VM have no service interruption.

When migrating using vMotion, only the status and memory of the VM—with its configuration—moves. The virtual disk does not need to move; it stays in the same shared-storage location. Once the VM has migrated, it is operating on the new physical host.

vMotion can function only with a shared storage architecture (such as Shared SAS array, FC SAN or iSCSI). As Caché is usually configured to use a large amount of shared memory it is important to have adequare network capacity available to vMotion, a 1Gb nework may be OK, however higher bandwidth may be required or multi-NIC vMotion can be configured.

DRS

Distributed Resource Scheduler (DRS) is a method of automating the use of vMotion in a production environment by sharing the workload among different host servers in a cluster. DRS also presents the ability to implement QoS for a VM instance to protect resources for Production VMs by stopping non-production VMs over using resources. DRS collects information about the use of the cluster’s host servers and optimize resources by distributing the VMs’ workload among the cluster’s different servers. This migration can be performed automatically or manually.

Caché Database Mirror

For mission critical tier-1 Caché database application instances requiring the highest availability consider also using InterSystems synchronous database mirroring. Additional advantages of also using mirroring include:

• Separate copies of up-to-date data.
• Failover in seconds (faster than restarting a VM then operating System then recovering Caché).
• Failover in case of application/Caché failure (not detected by VMware).

vCenter Appliance

The vCenter Server Appliance is a preconfigured Linux-based virtual machine optimized for running vCenter Server and associated services. I have been recommending sites with small clusters to use the VMware vCenter Server Appliance as an alternative to installing vCenter Server on a Windows VM. In vSphere 6.5 the appliance is recommended for all deployments.

Summary

This post is a rundown of key best practices you should consider when deploying Caché on VMware. Most of these best practices are not unique to Caché but can be applied to other tier-1 business critical deployments on VMware.

If you have any questions please let me know via the comments below.

Hi Folks,

I created a persistent class as below 

Class myclass.DataBase Extends %Persistent
{

Property ID As %String;

Property Skill As list Of myclass.SerialTablelist;
}

and Created another Serial class as 

Class myclass.SerialTablelist Extends %SerialObject
{

Property PSkill As %String;

Property OSkill As %String;
}

Now I will save the id as below

do rs.Prepare("Insert into myclass.DataBase(ID)VALUES(?)")
do rs.Execute(ID)

I am facing issue with the AS LIST OF 

Could any one guide me how to save the Skill (PSkill ,OSkill ) using resultset (I am able to )

For those that, at some point, need to test what means that of ECP for horizontal escalability (computing power and/or users and processes concurrency), but they're lazy o have no much time to build the environment, configure the server nodes, etc..., I've just published in Open Exchange the app/sample OPNEx-ECP Deployment .

PDF version: https://github.com/yurimarx/iris-periodic-table/raw/master/periodic%20table%20iris.pdf

GIT sources: https://github.com/yurimarx/iris-periodic-table

InterSystems IRIS is a Data Platform with a lot of features. These features and relevant topics about IRIS is represented in the periodic table elements.

The TOGAF is the The Open Group Architecture Framework. It provides an approach for planning, design, implement, deploy and govern your EA (Enterprise Architecture) projects.

The TOGAF has a concept called Building Block. It is any element that can be used, reused a executed to deliver value and new functions to the business.

In the picture above, I present to you the main IRIS building blocks to create fantastic apps.

To learn more about TOGAF and building blocks, see https://www.opengroup.org/togaf.

The Zachman Framework™ is an ontology - a theory of the existence of a structured set of essential components of an object for which explicit expressions is necessary and perhaps even mandatory for creating, operating, and changing the object (the object being an Enterprise, a department, a value chain, a “sliver,” a solution, a project, an airplane, a building, a product, a profession or whatever or whatever). Source: https://www.zachman.com/about-the-zachman-framework.

This post provides useful links and an overview of best practice configuration for low latency storage IO by creating LVM Physical Extent (PE) stripes for database disks on InterSystems Data Platforms; InterSystems IRIS, Caché, and Ensemble.

Consistent low latency storage is key to getting the best database application performance. For applications running on Linux, Logical Volume Manager (LVM) is often used for database disks, for example, because of the ability to grow volumes and filesystems or create snapshots for online backups. For database applications, the parallelism of writes using LVM PE striped logical volumes can also help increase performance for large sequential reads and writes by improving the efficiency of the data I/O.

This post has a focus on using LVM PE stripes with HCI and was also prompted by publication of the white paper Software Defined Data Centers (SDDC) and Hyper-Converged Infrastructure (HCI) – Important Considerations for InterSystems Clients here on the community. The white paper recommended “use of LVM PE striping with Linux virtual machines, which spreads IO across multiple disk groups” and to “Use of the Async IO with the rtkaio library for all databases and write image journal (WIJ) files with Linux virtual machines”. This post provides some context to those requirements and examples.

NOTE:

Currently there are multiple Hyper-Converged, Converged and Software Defined vendor platforms, rather than provide detailed instructions for each I have used the configuration of InterSystems IRIS or Caché on Red Hat Enterprise Linux (RHEL) 7.4 running on VMware ESXi and vSAN as the example in this post. However, the basic process is similar for other solutions, especially at the InterSystems IRIS or Caché and operating system level. If you are unsure how to translate these instructions to other platforms, please contact the respective vendor’s support for their best practice. InterSystems technology experts can also advise directly with customers and vendors or through the community.

It’s also worth noting that the guidance on LVM PE striping in this post can be applied to both HCI and “traditional” storage.

Do you have to use LVM striping?

For traditional storage such as disk arrays, the short answer is no. It is not mandatory especially with modern all-flash storage arrays to run LVM striped volumes for your database disks if your performance is ok now and you have no requirement for LVM then you do not have to change.

However, as stated above; LVM stripes are recommended for database disks on Hyper-Converged and storage solutions like Nutanix and VMware vSAN to allow for more host nodes and disk groups to be used in IO operations.

Why use LVM Stripes for Data Platforms?

LVM stripes are especially recommended for the database disks on HCI to mitigate the performance overhead of some features of the architecture, such as to lessen the impact of the Write Daemon (WD) on database writes and journal writes. Using an LVM stripe spreads the database write burst across more disk devices and multiple disk groups.  In addition, this post also shows how to increase the parallelism of the large IO size Write Image Journal (WIJ) which lessens the impact on latency for other IOs.

Note: In this post when I say “disk” I mean NVMe, Optane, SATA or SAS SSD, or any other flash storage devices.

vSAN storage architecture Overview

HCI storage, for example, when running ESXi on vSAN, uses two disk tiers; a cache tier and a capacity tier. For an all-flash architecture (you must use all flash - do not use spinning disks!) all writes go to the cache tier with data then ultimately destaged to the capacity tier. Reads come from the capacity tier (or possibly from cache on the Cache tier). Each host in the HCI cluster can have one or more disk groups. Where there are disk groups, for example with vSAN, each disk group is made up of a cache disk and multiple capacity disks. For example, the cache disk is a single NVMe disk and the capacity disks are three or more write intensive SAS SSD disks.

For further details on HCI, including vSAN disk groups, see the community post Hyper-Converged Infrastructure (HCI) on the community or contact your HCI vendor.

LVM Striped logical volumes overview

A good overview of Linux LVM is available on the Red Hat Support web site and also other places, for example, this tutorial for system administrators is very good.

Data Platforms storage IO

It is important you understand the types of IO generated by InterSystems Data Platforms. An overview of storage IO patterns is available on the community.

Process to create LVM PE stripe

Prerequisites and procedure

Before we dive into the process you should also remember that other variables can come into play that impact storage performance. Simply creating an LVM stripe is not a guarantee of optimal performance, you will also have to consider the storage type, and the whole IO path, including IO queues and queue depth.

This example is for VMware, the InterSystems IRIS VMware best practice guide should also be read and recommendations applied. Especially considerations for storage such as separation of storage IO types across PVSCSI controllers.

Overview

The following example is for best practice using InterSystems IRIS or Caché on Red Hat Enterprise Linux (RHEL) 7.4 running on VMware ESXi and vSAN 6.7.

The following steps are outlined below;

1. ESXi configuration
2. RHEL configuration
3. Caché / InterSystems IRIS configuration

1. ESXi Configuration

a) Create VMDK disks

You must create disks as per the InterSystems IRIS VMware best practice guide; Databases, journal and WIJ are on separate PVSCI devices.

Create the number of VMDKs depending on your sizing requirements. In this example, the database file system will be made up of four 255GB VMDK disks that will be striped together to create a 900GB logical disk for the database filesystem.

Steps:

1. Power off the VM before adding the VMDKs,
2. In the vCenter console create multiple disks (VMDKs) at 255GB each, all disks in a single LVM stripe must be associated with the same PVSCSI controller.
3. Power on the VM. During power on the new disks will be created at the operating system, for example /dev/sdi etc.

Why create multiple 255 GB VMDKs? in vSAN storage components are created in chunks of 256GB, keeping the VMDK size just under 256 GB we are trying force the components to be on separate disk groups. Enforcing another level of striping (This has been the case in my testing, but I cannot guarantee that vSAN will actually do this).

Note: During creation vSAN spreads the disk components across all hosts and across disk groups for availability. For example in the case of Failures To Tolerate (FTT) set to 2 there are three copies of each disk component plus two small witness components all on separate hosts. In the event of a disk group, host or network failure the application continues without data loss using the remaining disk component. It is possible to overthink this process! With HCI solutions like vSAN, there is no control over what physical disk the components that make up the VMDKs will reside on at any point in time. In fact, due to maintenance, resynchronisation, or rebuilds, the VMDKs could move to different disk groups or hosts over time. This is OK.

2. RHEL Configuration

a) Confirm RHEL IO scheduler is NOOP for each of the disk devices.

The best practice is to use the ESXi kernel’s scheduler. For more information on setting the scheduler see the Red Hat knowledge base article. We recommend using the option to set for all devices at boot time. To validate you have set the scheduler correctly you can display the current setting for a disk device, for example in this case /dev/sdi as follows;

[root@db1 ~]# cat /sys/block/sdi/queue/scheduler
[noop] deadline cfq

You can see noop is enabled because it is highlighted between the square brackets.

b) Create striped LVM and XFS file system

We are now ready to create the LVM stripe and database filesystem in RHEL. Following is an example of the steps involved, note the made-up names vgmydb, lvmydb01, and path /mydb/db would be substituted for your environment.

Steps

1. Create the Volume Group with new disk devices using the vgcreate command.

vgcreate -s 4M <vg name> <list of all disks just created>

For example, if disks /dev/sdh, /dev/sdi, /dev/sdj and /dev/sdk were created:

vgcreate -s 4M vgmydb /dev/sd[h-k]

2. Create the striped Logical Volume using the lvcreate command. A minimum of four disks is recommended. Start with a 4MB stripe, however with very large logical volumes you may be prompted for a larger size such as 16M.

lvcreate -n <lv name> -L <size of LV> -i <number of disks in volume group> -I 4MB <vg name>

For example to create the 900GB disk with 4 stripes and stripe size of 4 MB :

lvcreate -n lvmydb01 -L 900G -i 4 -I 4M vgmydb

3. Create the database file system using the mkfs command.

mkfs.xfs -K <logical volume device>

For example:

mkfs.xfs -K /dev/vgmydb/lvmydb01

4. Create the file system mount point, for example:

mkdir /mydb/db

5. Edit /etc/fstab with following mount entries and mount the file system. For example:

/dev/mapper/vgmydb-lvmydb01 /mydb/db xfs defaults 0 0

6. Mount the new filesystem.

mount /mydb/db

3. Caché/InterSystems IRIS configuration

In this section we will configure:

• Asynchronous and direct IO for optimal write performance on the database and WIJ. This also enables direct IO for database read operations.

NOTE: Because direct IO bypasses filesystem cache, OS file copy operations including Caché Online Backup will be VERY slow when direct IO is configured.

For added performance and lowest latency for the WIJ on RHEL (this is not supported on SUSE since SUSE 9), and to lessen the impact on other IO we will also configure:

• Use of the rtkaio library for RHEL systems using Caché. Note: IRIS does not need this library.

NOTE: For Caché, Ensemble, and HealthShare distributions beginning with version 2017.1.0. on Linux (only if a backup or async mirror member is configured to use the rtkaio library) you must apply RJF264, available via Ad Hoc distribution from InterSystems Worldwide Response Center (WRC).  

Steps

The procedure is to:

1. Shutdown Caché
2. edit the <install_directory>/cache.cpf file
3. Restart Caché.

In the cache.cpf file add the following three lines to the top of [config] stanza, leaving other lines unchanged, as shown in the example below;

[config]
wduseasyncio=1
asyncwij=8

For RHEL Caché (not IRIS) also add the following to the [config] section:

LibPath=/lib64/rtkaio/

Note: When Caché restarts the lines will be sorted in alphabetical order in the [config] stanza.

Summary

This post gave an example of creating a 900GB LVM PE stripe and creating a file system for a database disk on vSAN. To get the best performance from the LVM stripe you also learned how to configure Caché/InterSystems IRIS for asynchronous IO for database writes and the WIJ.

I have attached a document that describes the product I have developed called NiPaRobotica Pharmacy. This is an interface I developed that accepts Pharmacy Dispense Requests and converts the line items on the order into dispense dialogues which it sends to pharmacy robots. I deployed the interface into 3 Hospital pharmacies two of which had 6 robots that were arranged in such a way that the dispense chutes channelled medications to desks by the pharmacists sitting in windows serving 1200 patients a day. The robots cut the average waiting time from 2 hours down to one hour. I then deployed the

Hi developers!

Do you know a CRM that is built with InterSystems IRIS or Caché or Ensemble on a backend?

Hi Community and InterSystems Partners!

We are glad to share great news for Intersystems Partner Directory Members:
here is a list of services you can use to become more visible within our InterSystems Community.

As a partner, you may order one of the services every six months free of charge:

$1,000 Google AdWords Campaign Voucher
We will set up and launch the campaign for you

Promotion within the Developer Ecosystem
We put a banner on the website with 50K+ monthly InterSystems related audience

Hey Developers,

We are excited to announce the launch of the InterSystems Partner Directory!

This is the place to go to find commercial services and solutions built on InterSystems products.

Why InterSystems Partner Directory?

I'm glad to announce we have recently released our second Starter Pack. This one is for the Mining Industry, and the first was for Industrial IoT (OEE).

But what does it exactly mean?

InterSystems IRIS Starter Packs (thanks to Joe Lichtenberg that helped with this text)

It's a challenge when you need, as a software architect, design a corporate architecture to meet the current business requirements, you need achieve level 5. With InterSystems IRIS.
it's possible. With 1 product you get SQL + NoSQL + ESB + BI + Open Analytics + Real time virtual cubes + NLP + AutoML + ML (with Python) and Advanced cloud + Sharding support.

What is Data Fabric?

“It is a set of hardware infrastructure, software, tools and resources to implement, administer, manage and execute data operations in the company, including, acquisition, transformation, storage, distribution, integration, replication, availability, security, protection , disaster recovery, presentation, analysis, preservation, retention, backup, recovery, archiving, recall, deletion, monitoring and capacity planning, across all data storage platforms and enabling application use to meet the company's data needs company". (Alan McSweeney)

If you need write your organization Data Architecture and map to the InterSystems IRIS, consider following Data Architecture Diagram and references to the intersystems iris documentation, see:

Architecture mapping:

Introduction

InterSystems has recently completed a performance and scalability benchmark of IRIS for Health 2020.1, focusing on HL7 version 2 interoperability. This article describes the observed throughput for various workloads, and also provides general configuration and sizing guidelines for systems where IRIS for Health is used as an interoperability engine for HL7v2 messaging.

Hi 

we are implementing HIE and wan to get expert opinion on facility registry setup. Below are some example and which method we should adapt.

3 EMRs sending u patient data (ADT, ORU etc) for 3 facility groups

Facility Group A (Has 20 branches locations, same MRN across all branches, each branch is registered with health department)

Facility Group B (Has 30 branches locations, same MRN across all branches, each branch is registered with health department)

Facility Group C (Has only one main branch single location and is registered with health department)

Facility Registry Setup Option - 1

Hi Developers!

Thank you for using ZPM Package Manager and contributing more and more useful packages to the public registry!

But as you already aware ZPM packages are always deployed with source code.

Do you think we need to add the option to deploy without source code - e.g. if you want to deploy a commercial package?

Will you develop commercial modules if there will be an option and deploy it with ZPM?

How do you deploy commercial applications today?

Organizations around the world lose an estimated five percent of their annual revenues to fraud, according to a survey of Certified Fraud Examiners (CFEs) who investigated cases between January 2010 and December 2011. Applied to the estimated 2011 Gross World Product, this figure translates to a potential total fraud loss of more than $3.5 trillion (source: https://www.acfe.com/press-release.aspx?id=4294973129).

Good day, developers!

The "apptools-admin" project is not only a set of tools for the administrator, but a platform for the rapid creation of a prototype of any solution. For example, Photo Album, Music Player, and Personal Library all come together in a user-friendly treasure chest design.

Load http:// your-host:your-port/apptoolsrest/a/infochest

Change the path to your treasure files and start viewing, listening or reading from any convenient device: desktop, tablet or smartphone.

The example will be located at ${CSPdir}/apptools/files

After saving and updating the panel, you can select a music track.

The built-in HTML5 player is used to play music.

The Lightbox Component is used to view pictures and videos: UiKit.

All "chest" code is located in the class apptools.lte.infochest and can be easily modified to suit your needs. Vote for this project apptools-admin.

Happy coding!

Hi community,

When InterSystems Health stack are working toghether the hospital ecosystem get this:

Hi community, I used website-analyzer - an app that uses InterSystems NLP and Crawler4J to extract all website content and do NLP on it. I limited to 200 pages and discovered this:

Top 10 Concepts - business and content topics in the InterSystems site:

Other frequency concepts, see the focus in the iris speed, scale and data value:

Top 10 Concept Dominance - business relations in the InterSystems site. See the InterSystems slogans ("first data platform", "intersystems data platform" and "healthcare data platform")

I have already asked a question about migrating an Oracle database to Caché, and I was promptly answered.Now, I have one more question, however it is more conceptual.In general, any tool defends its benefits, and looking at Caché's, among many, there is a lot of talk about speed.The question is, if I already have a database, and at the moment, its performance is not a problem, I would like to know what else could influence the change of this database?Remembering that it is functional and without any apparent problem.

IRIS is an excellent option for machine learning projects with massive data operation scenarios, because the following reasons:

The InterSystems IRIS has an integration engine with these core elements:

1. Adapters: are inbound (data/message input/request) and outbound (data/message output/response)  integration logic specialized in the type of data or message protocol (file, http, etc.). They are the interface to allows connect with source or target data repositories or systems.

2. DTL: is a component to mapping and transform data between two components in the flow orchestration.

3. BPL: is a component executing BPEL language to execute integration/orchestration flow.

Some clients ask me about migrate from Cache to IRIS. Why migrate to IRIS? Cache is excellent, stable, has good performance. These clients are right, but in the last years, the digital transformation imposed more complete solutions for the new requirements and InterSystems was a visionary to perceive it and launch IRIS.
It is a data platform ready to this digital transformation challenge. To show this, I created a Value Canvas.

The InterSystems IRIS has functions that allows create DIWK digital services. A few products have the ability to transform data into wisdom, according to the following pyramid.

Using SOAP Web Services or REST API Resources, if you want to deliver strategic digital assets for your organization, SOA aproach is an excellent option. The InterSystems IRIS supports like a charm the SOA principles with Contract First technique to model services aligned with the business, and create the services from the service contracts (Open API or WSDL).

Inter

Intersystems IRIS is a complete platform to get insights from SQL and NoSQL data. It is possible get data from Interoperability adapters or using a set of IRIS tables as data sources and model BI or NLP Cubes, covering all type of data (other tools are limited to SQL). There are the option to enable intensive analytics processing using Spark too. So you can model your analysis using IRIS web analyzers (many tools use desktop tools) and than visualize and produce insights using IRIS Dashboards and IRIS User Portal or your third party options, using open options like MDX and REST.