#Distributed Data Management

Target Practice for IrisClusters with KWOK

KWOK, Kubernetes WithOut Kubelet, is a lightweight tool that simulates nodes and pods—without running real workloads—so you can quickly test and scale IrisCluster behavior, scheduling, and zone assignment.  For those of you wondering what value is in this without the IRIS workload, you will quickly realize it when you play with your Desk Toys awaiting nodes and pods to come up or get the bill for provisioning expensive disk behind the pvc's for no other reason than just to validate your topology.

Here we will use it to simulate an IrisCluster and target a topology across 4 zones, implementing high availability mirroring across zones, disaster recovery to an alternate zone, and horizontal ephemeral compute (ecp) to a zone of its own.  All of this done locally, suitable for repeatable testing, and a valuable validation check mark on the road to production.

I'm playing with %Net.DB.Iris and stumbled over a mystery

set con=##class(%Net.DB.DataSource).CreateConnection(host,...)
set srv=con.CreateIris()
write srv.ClassMethodValue("%SYSTEM.Util","InstallDirectory")

Entering the above lines (in a terminal session) on my local instance yields the correct answer for:

host = "localhost"
host = the real IP of localhost (i.e. host="192.168...")
host = "10.x.y.dev" customers development system (over a VPN tunnel)

but for

What is Distributed Artificial Intelligence (DAI)?

Attempts to find a “bullet-proof” definition have not produced result: it seems like the term is slightly “ahead of time”. Still, we can analyze semantically the term itself – deriving that distributed artificial intelligence is the same AI (see our effort to suggest an “applied” definition) though partitioned across several computers that are not clustered together (neither data-wise, nor via applications, not by providing access to particular computers in principle). I.e., ideally, distributed artificial intelligence should be arranged in such a way that none of the computers participating in that “distribution” have direct access to data nor applications of another computer: the only alternative becomes transmission of data samples and executable scripts via “transparent” messaging. Any deviations from that ideal should lead to an advent of “partially distributed artificial intelligence” – an example being distributed data with a central application server. Or its inverse. One way or the other, we obtain as a result a set of “federated” models (i.e., either models trained each on their own data sources, or each trained by their own algorithms, or “both at once”).

With the release of InterSystems IRIS, we're also making available a nifty bit of software that allows you to get the best out of your InterSystems IRIS cluster when working with Apache Spark for data processing, machine learning and other data-heavy fun. Let's take a closer look at how we're making your life as a Data Scientist easier, as you're probably already facing tough big data challenges already, just from the influx of job offers in your inbox!

Last week, we announced the InterSystems IRIS Data Platform, our new and comprehensive platform for all your data endeavours, whether transactional, analytics or both. We've included many of the features our customers know and loved from Caché and Ensemble, but in this article we'll shed a little more light on one of the new capabilities of the platform: SQL Sharding, a powerful new feature in our scalability story.

Learning Services Live Webinars are back! 

At this year’s Global Summit, InterSystems debuted InterSystems IRIS Data Platform™, a single, comprehensive product that provides capabilities spanning data management, interoperability, transaction processing, and analytics. InterSystems IRIS sets a new level of performance for the rapid development and deployment of data-rich and mission-critical applications. Now is your chance to learn more! 

While creating namespaces, there is a provision for keeping the data, and code separately in database.
What I am looking for,

What are the benefit we can achieve by doing this?

Is there any documentation of article which might help understanding?

Thanks You!

Several years ago everyone got mad about BigData – nobody knew when smallish data will become BIGDATA, but all knows that it’s trendy and the way to go. Time passed, BigData is not a buzz anymore (most of us missed the moment when Gartner has removed BigData term from their 2016 buzzword 2016 curve http://www.kdnuggets.com/2015/08/gartner-2015-hype-cycle-big-data-is-out-machine-learning-is-in.html), so it’s probably a good time to look back and realize what it is (what it was)…

When it becomes “BigData”?

In part I of this series we have introduced MapReduce as a generic concept, and in part II we started to approach Caché ObjectScript implementation via introducing abstract interfaces. Now we will try to provide more concrete examples of applications using MapReduce.

Good day

Additional tool for MUMPS servers MiniM InterConnect:

http://www.minimdb.com/tools/interconnect.html

to connect job on one MUMPS server to other job on other MUMPS server in client-server mode. This tool can connect different MUMPS systems on different OS and with different processors - MiniM, Cache and GT.M in any pairs.

Regards,

Eugene Karataev

The technology of load balancing between several servers with relatively low capacity has been a standard feature of Caché for quite a while. It is based on the distributed cache technology called ECP (Enterprise Cache Protocol). ECP provides a host of possibilities for horizontal scaling of an application, and yet keeping the project budget fairly low. Another apparent advantage of ECP network is the possibility to conceal its architecture in the depths of Caché configuration so that applications developed for the traditional (vertical) architecture can be fairly easily migrated to a horizontal ECP environment. The ease of this process is so mesmerizing, that you start wishing it was always this way. For instance, everybody is used to having a possibility to control Caché processes: the $Job system variable and associated classes/functions work magic in skilful hands. Stop, but now processes can end up being on different Caché servers…

This article is about how to gain as much transparency in controlling processes in ECP environment as in traditional (non ECP) one.

We are finally planning to migrate some ancient Caché applications that are run on Caché 5.0.21 to a new server with Caché 2016.2.0 or so.

I wonder if we could use Shadowing between those to keep the data on the new server up to date?

We would copy the Caché backup from the old environment to the new and do a RESTORE there and then start shadowing.

I know than 5.0.21 is no more officially supported by ISC.

Consider a design where there could be three or four or more servers and there is a need to have these eventually consistent between them all (and not considering database mirroring here).  

The current Caché documentation here demonstrates this well using object synchronization between two servers, however it doesn't indicate whether more than two servers can participate to create a "mesh type" deployment.  Below is a diagram of what I'm curious to know is possible to implement with Object Synchronization.