I recently helped a site investigate a problem that appeared after they upgraded their Windows instance of Caché from 2015.1 to 2017.1. A terminal session launched from the server's desktop cube was unable to run OS-level commands using the $ZF(-1) function. For instance, using the no-op command "REM" as follows:

write $zf(-1,"rem")

was returning -1, indicating that the Windows command could not be issued.

What is the recommended library for converting documents to pdf. Any suggestions welcome has anyone out there had a use case where they had to convert documents to pdf .What worked for them .I have asked WRC for the best practice and they have recommended that may be  a sale engineer might have an answer to this .

Question:can a business process with pool size > 1 invoke external programs in parallel via $ZF(-100)?

We have a production that takes HL7 (ORU_R01) messages in and produces a PDF from each one for onward transmission to a document store.

Hello,

I am trying to grab a specific zip file from a webpage but once I grab the zip file I need to grab a specific file within that zip file and I was hoping someone could explain how to do this or provide an example

Some time ago I got a WRC case transferred where a customer asks for the availability of a raw DEFLATE compression/decompression function built-in Caché.

When we talk about DEFLATE we need to talk about Zlib as well, since Zlib is the de-facto standard free compression/decompression library developed in the mid-90s.

Zlib works on particular DEFLATE compression/decompression algorithm and the idea of encapsulation within a wrapper (gzip, zlib, etc.).
https://en.wikipedia.org/wiki/Zlib

I have a C string and I need to build a $lb from it.

This code works fine for strings shorter than 254 characters:

char *str = "some string";
int len = strlen(str);
int add = 2;
char *list = malloc(len + add + 1);
char lenChar = len + add;
sprintf(list, "%c\x01%s", lenChar, str);

Thought maybe someone can share the code for longer strings?

Our company is in the process of converting our software for use in Intersystems IRIS and one of the major sections of the software makes use of a custom statically-linked C library using the $ZF("function-name") functionality. During this, I found out that the process for setting up the C library to be used within the database platform has changed significantly between Cache and IRIS.

I am working on a BPL to take data from a MS SQL database and create an HL7 Materials Message for our EMR.  I have done this plenty of times in the past however I am running into an error.

"Remote Gateway Error: JDBC Gateway SP execute(0) error 0: Access to the remote server is denied because no login-mapping exists."

What is confusing is that this BPL doesn't differ from any of my other BPLs in connecting to MS SQL Server. I know I am missing something..

This BPL will execute the 1st Stored Procedure without any issues, the issue is when it comes to executing the second stored procedure.

Hello Community,

We have two live servers running DeepSee dashboards for users.  One of the servers can print a widget to a pdf file, and the other can't.  

I learned that 1) a Java JRE needed to be installed on the second server, and 2) it's trying to run an OS command to render a pdf file (details below).

An audit log of the event shows this:

Hi,

Is there any way to set environment variables in Linux from Cache?

I see a way to get an environment variable with: $system.Util.GetEnviron()

Essentially I am converting from VMS (DCL) to Linux.

In VMS we used $ZF(-1,"SETSYM") in Cache to a value and then interpreted that value in a DCL procedure.

Actually, I think I should just explained the need. We are writing linux scripts that call cache routines and we would like to pass back a value indicating failure or success to be handle it in the linux script. Right now we are writing out a log file, opening it and searching for a string. Any ideas of how to pass back an exit status to the calling script is helpful.

Thanks.

I'm using callin to get global values.

Here's a simple function to get string value from global and return it:

I managed to use CacheGlobalOrder to iterate over global and return the next subscript:

I am having an intermittent issue that when I make a call to MSSQL from a BPL that the response does not come back in the amount of time required. Since the call from the BPL is synchronous I tried changing the timeout to 60 but it has not helped (see below). Is there anyway to guarantee that the call waits long enough for a response before continuing on?

Thanks

Scott Roth

Intro

Recently I reread this article by @Bernd.Mueller. It's about calling DELFATE function from zlib library. In this article I'll demonstrate several different approaches to callout libraries, we'll build the same functionality (compress function) in several different languages and compare them.

Let's start with NodeJS. I'm taking the code almost directly from Bernd's article, except it does not use files, but rather direct http connection to pass data. For production use, it would be better to pass request as a body and encode both request and response as base64. Still, here's the code:

//zlibserver.js
const express = require('express');
const zlib = require('zlib');
 
var app = express();
 
 app.get('/zlibapi/:text', function(req, res) {
    res.type('application/json');
    
    var text=req.params.text;
    
    try {        
		zlib.deflate(text, (err, buffer) => {
		   if (!err) {
				res.status(200).send(buffer.toString('binary'));
			} else {
				res.status(500).json( { "error" : err.message});
			// handle error
			}
		});
     }
    catch(err) {
      res.status(500).json({ "error" : err.message});
      return;
    }
    
});
app.listen(3000, function(){
    console.log("zlibserver started");
});

To start it execute in OS bash (assuming node and npm installed):

cd <repo>\node
npm install
node  ./zlibserver.js

We're running on port 3000, reading input string from request and returning compressed data in response as is. On a Caché side http request is used to interact with this api:

/// NodeJS implementation
/// do ##class(isc.zlib.Test).node()
ClassMethod node(text As %String = "Hello World", Output response As %String) As %Status
{
    kill response
    set req = ##class(%Net.HttpRequest).%New()
    set req.Server = "localhost"
    set req.Port = 3000
    set req.Location = "/zlibapi/" _ text
    set sc = req.Get(,,$$$NO)
    quit:$$$ISERR(sc) sc
    set response = req.HttpResponse.Data.Read($$$MaxStringLength)
    quit sc
}

Note, that I'm setting the third argument set sc = req.Get(,,$$$NO) - reset to zero. If you're writing interface to the outside http(s) server it's best to reuse one request object and just modify it as needed to perform new requests.

Java

Java Gateway allows calling arbitrary Java code. Coincidently Java has Deflater class which does exactly what we need:

package isc.zlib;

import java.util.Arrays;
import java.util.zip.Deflater;

public abstract class Java {

    public static byte[] compress(String inputString) {
        byte[] output = new byte[inputString.length()*3];
        try {
            // Encode a String into bytes
            byte[] input = inputString.getBytes("UTF-8");

            // Compress the bytes

            Deflater compresser = new Deflater();
            compresser.setInput(input);
            compresser.finish();
            int compressedDataLength = compresser.deflate(output);
            compresser.end();
            output = Arrays.copyOfRange(output, 0, compressedDataLength);

        } catch (java.io.UnsupportedEncodingException ex) {
            // handle
        }


        return output;
    }
}

The problem with this implementation is that it returns byte[] which becomes a Stream on Caché side. I have tried to return a string, but hadn't been able to found how to form proper binary string from byte[]. If you have any ideas please leave a comment. To run it place jar from releases page into <instance>/bin folder, load ObjectScript code into your instance and execute:

write $System.Status.GetErrorText(##class(isc.zlib.Utils).createGateway())
write $System.Status.GetErrorText(##class(isc.zlib.Utils).updateJar())

Check createGateway method before running the command. Second argument javaHome assumes that JAVA_HOME environment variable is set. If it does not, specify home of Java 1.8 JRE as a second argument. After installing run this code to get compressed text:

set gateway = ##class(isc.zlib.Utils).connect()
set response = ##class(isc.zlib.Java).compress(gateway, text)

C

An InterSystems Callout library is a shared library that contains your custom Callout functions and the enabling code that allows Caché to use them.

Here's our Callout library:

#define ZF_DLL

// Ugly Windows hack
#ifndef ulong
   typedef unsigned long ulong;
#endif

#include "string.h"
#include "stdio.h"
#include "stdlib.h"
#include "zlib.h"
#include <cdzf.h>

int Compress(char* istream, CACHE_EXSTRP retval)
{
	ulong srcLen = strlen(istream)+1;      // +1 for the trailing `\0`
	ulong destLen = compressBound(srcLen); //  estimate size needed for the buffer
	char* ostream = malloc(destLen);
	int res = compress(ostream, &destLen, istream, srcLen);
	CACHEEXSTRKILL(retval);
	if (!CACHEEXSTRNEW(retval,destLen)) {return ZF_FAILURE;}
	memcpy(retval->str.ch,ostream,destLen);   // copy to retval->str.ch
	return ZF_SUCCESS;
}

ZFBEGIN
	ZFENTRY("Compress","cJ",Compress)
ZFEND

To run it place dll or so files from releases page into <instance>/bin folder. Repository also contains build scripts for Windows and Linux, execute them to build your own version. Linux prerequisites:

apt install build-essential zlib1g zlib1g-devel

Windows prerequisites: WinBuilds - comes with zlib.

To interact with callout library execute:

set path =  ##class(isc.zlib.Test).getLibPath() //get path to library file
set response = $ZF(-3, path, "Compress", text)       // execute function
do $ZF(-3, "")                                  //unload library

System

A little unexpected in an article about callout mechanisms, but Caché also has built-in Compress (and Decompress function). Call it with:

set response = $extract($SYSTEM.Util.Compress(text), 2, *-1)

Remember that searching the docs or asking the questions here on the community may save you some time.

Comparison

I have run simple tests (1Kb text, 1 000 000 iterations) on Linux and Windows and got these results.

Windows:

Linux:

To run tests load code and call:

do ##class(isc.zlib.Test).test(textLength, iterations)

Conclusion

With InterSystems products, you can easily leverage existing code in other languages. However, choosing correct implementation is not always easy, you need to take several metrics into account, such as development speed, performance, and maintainability. Do you need to run on different operating systems? Finding answers to these questions can help you decide on the best implementation plan.

Links

• Repo
• Binaries
• Http requests
• Java Gateway
• Callout library
• Compress function

I have a simple callout library:

#define ZF_DLL
#include 
#include 
#undef ERROR

int GetRandom(double* random) {
   // Py_Initialize();
   // Py_Finalize();
   *random = 1.0;
   return ZF_SUCCESS;
}

int main(int argc, char **argv)
{
   printf("Random: ");
   double random=0;
   GetRandom(&random);
   printf("%lf", random);
   return 0;
}

ZFBEGIN
    ZFENTRY("GetRandom","D",GetRandom)
ZFEND

I compile this code as a shared library and it works fine with:

set path = "library.dll"
write $ZF(-3, path, "GetRandom")

It also compiles and works as an executable.

There are numerous ways to interact with InterSystems Caché: We can start with ODBC/JDBC that are available via SQL gateway. There are API for .NET and Java too. But if we need to work with native binary libraries, such interaction is  possible through Caché Callout Gateway, which can be tricky. You can read more about the ways of facilitating the work with native libraries directly from Caché in the article below.

Caché Callout Gateway

Caché uses Caché Callout Gateway for working with native code. This name applies to a few functions united under a single name, $ZF(). These functions are divided into two groups:

• $ZF(-1), $ZF(-2). The first group of functions allows you to work with system commands and shell scripts. It’s an efficient tool, but its shortcoming is evident – the entire functionality of the library cannot be implemented in one or several programs.

  An example of using $ZF(-1)

  Creation of a new directory called “newdir” in the working directory:

  set name = "newdir"
  set status = $ZF(-1, "mkdir " _ name) 
  

• $ZF(-3), $ZF(-5), $ZF(). The second group of functions provides access to dynamic and static libraries. It looks more like what we need. But it’s not so easy: $ZF() functions do not work with any libraries, but only with libraries of a particular type — Callout Libraries. A Callout Library differs from a regular library in that its code has a special character table called ZFEntry, which contains a certain version of prototypes of exported functions. Moreover, the type of arguments of the exported functions is strictly limited — only int and a few other pointer types are supported. To make a Callout Library from an arbitrary one, you will most probably need to write a wrapper for the entire library, which is far from convenient.

  An example of creating a Callout Library and calling a function from it

  Callout Library, test.c

  #define ZF_DLL
  #include <cdzf.h> // the cdzf.h file is located in Cache/dev/cpp/include  
  int  
  square(int input, int *output)  
  {  
    *output = input * input;  
    return ZF_SUCCESS;  
  }  
  
  ZFBEGIN // character table  
  ZFENTRY("square", "iP", square) // "iP” means that square has two arguments - int and int*  
  ZFEND 
  

  Compilation (mingw):

  gcc -mdll -fpic test.c -o test.dll
  

  For linux use -shared instead of -mdll.

  Calling square() from Caché:

  USER> do $ZF(-3, "test.dll", "square", 9)
  81
  

Caché Native Access

To remove the limitations of a Callout Gateway and make the work with native libraries comfortable, the CNA project was created. The name is a copy of a similar project for a Java machine, JNA.

CNA capabilities:

• You can call functions from any dynamic (shared) library that is binary compatible with C
• To call functions, you only need code in ObjectScript – you don’t need to write anything in C or any language compiled into the machine code
• Support of all simple types of the C language, size_t and pointers
• Support of structures (and nested structures)
• Support of Caché threads
• Supported platforms: Linux (x86-32/64), Windows (x86-32/64)

Installation

First, let’s compile the C part, which is done with a single command —

make libffi && make

In Windows, you can use mingw or download pre-compiled binary files. After that, import the cna.xml file to any convenient namespace:

do $system.OBJ.Load("path to cna.xml", "c") 

An example of working with CNA

The simplest native library that exists on every system is the C standard library. In Windows, it’s usually located at C:\Windows\System32\msvcrt.dll, in Linux — /usr/lib/libc.so. Let’s try calling a function from it, for example, strlen. It has the following prototype:

size_t strlen(const char *); 

Class CNA.Strlen Extends %RegisteredObject 
{ 
  ClassMethod Call(libcnaPath As %String, libcPath As %String, string As %String) As %Integer 
  { 
    set cna = ##class(CNA.CNA).%New(libcnaPath)      // creates an object of CNA.CNA 
    do cna.LoadLibrary(libcPath)                     // uploads libc to CNA 
 
    set pString = cna.ConvertStringToPointer(string) // converts the string into the C format an save a pointer to its beginning 
 
    // Calling strlen: pass the function name, type of returned value,  
    // list of argument type and a comma-delimited list of arguments 
    set result = cna.CallFunction("strlen", cna.#SIZET, $lb(cna.#POINTER), pString) 
 
    do cna.FreeLibrary() 
    return result 
  } 
}

In the shell:

USER>w ##class(CNA.Strlen).Call("libcna.dll", "C:\Windows\system32\msvcrt.dll", "hello") 
5 

Implementation details

CNA is a link between a C library and a Caché class. CNA heavily relies on libffi. libffi is a library that lets developers organize a “low-level” interface for external functions (FFI). It helps forget about the existence of various call conventions and call functions at runtime, without providing their specifications during compilations. However, in order to call functions from libffi, you need addresses, and we’d like to do it using names only. To get a function’s address by its name, we’ll have to use some platform-dependent interfaces: POSIX and WinAPI. POSIX has the dlopen() / dlsym() mechanism for loading a library and searching for a function’s address; WinAPI has the LoadLibrary() and GetProcAddress() functions. This is one of the obstacles for porting CNA to other platforms, although virtually all modern system at least partially support the POSIX standard (except for Windows, of course).

libffi is written C and assembler. This makes libffi a native library and you need to use the Callout Gateway to access it from Caché. That is, you need to write a middleware layer that will connect libffi and Caché and be a Callout Library so that you can call it from ObjectScript. Schematically, CAN works like this:

At this stage, we face a data conversion issue. When we call a function from ObjectScript, we pass the parameters in the internal Caché format. We need to pass them to Сallout Gateway, then to libffi, but we need to convert them to the C format at some point. However, Callout Gateway supports very few data types and if we converted data on the C side, we’d have to pass everything as strings, then parse them, which is obviously inconvenient. Therefore, we made a decision to convert all data on the Cache side and pass all arguments as strings with binary data already in the C format.

Since all types of C data, except composite ones, are numbers, this task of data conversion effectively boils down to converting numbers into binary strings using ObjectScript. For this purpose, Caché has some great functions saving you the trouble of accessing data directly: $CHAR and $ASCII. They convert an 8-bit number into a character and back. There are corresponding functions for all the necessary numbers: 16-, 32- and 64-bit numbers and for double-precision floating-point numbers. There is one “but”, however – all these functions work only for signed or unsigned numbers (apparently, we are talking about integers). In C, however, a number of any size can be both signed and unsigned. Therefore, we’ll need to manually customize these functions to fulfill their purpose.

In C the two’s complement is used for representing signed numbers:

• The first bit is responsible for the sign of a number: 0 — plus, 1 — minus
• Positive numbers are coded as unsigned ones
• The maximum positive number is 2^k-1-1, where k is the number of bits
• The code of a negative number x is the same as that of an unsigned number 2^k+x

This method allows you to use the same addition operation that you use for unsigned numbers. This is achieved with the help of integer overflow.

Let’s consider an example of converting unsigned 32-bit numbers. If the number is positive, we need to use the $ZLCHAR function, if it’s negative, we need to find such an unsigned number that their binary representations are identical. The method of searching for this number becomes evident from the very definition of the extra code – we need to add the initial number to the minimal one that doesn’t fit into 32 bits – 2³² or FFFFFFFF₁₆ + 1. As a result, we have the following piece of code:

if (x < 0) { 
    set x = $ZLCHAR($ZHEX("FFFFFFFF") + x + 1) 
} else 
    set x = $ZLCHAR(x) 
} 

The next problem is the transformation of the structures (composite type of C language). Things would be so much easier if the structures in the memory were represented in the same way they were written to it — in a sequence, field after field. However, every structure in the memory is located so that the address of every field is a product of a special field alignment number. Alignment is necessary because most platforms either do not support unaligned data or do it rather slowly. As a rule, the alignment value on the x86 platform is equal to the size of the field, but there are exceptions like the 32-bit Linux, where all fields over 4 bytes equal exactly 4 bytes. More information about data alignment can be found in this article.

Let’s take this structure, for example:

struct X { 
    char a, b; // sizeof(char) == 1 
    double c;  // sizeof(double) == 8 
    char d;     
}; 

On the x86-32 platform, it will be located in the memory differently in different operating systems:

In practice, such representation of the structure is formed quite easily. You just need to sequentially write the fields to the memory but add padding – an empty space before each record – every time you perform a write operation. Padding is calculated using this formula:

set padding = (alignment - (offset # alignment)) # alignment //offset – the address of the end of the last record 

What’s not working yet

1. In Caché, integers are represented in a way that accurate work with them is only guaranteed for as long as the number doesn’t exceed the boundaries of a 64-bit signed number. However, C also has a 64-bit unsigned type (unsigned long long). That is, you won’t be able to pass a value exceeding the size of a 64-bit signed number, 2⁶³-1(~9 * 10¹⁸), to an external function.

2. Caché has two variable types for working with real numbers: its own decimal and double-precision floating-point numbers compliant with the IEEE 754 standard. That is Caché has no equivalents of the float and long double types found in C. You can work with these types in CNA, but they will be automatically converted to double when passed to Caché.

3. If you work on Windows, the use of the long double type will most probably cause problems. This is caused by the fact that Microsoft and the mingw development team have completely different opinions about the size of the long double type. Microsoft believes that its size should be exactly 8 bytes both on 32- and 64-bit systems. Mingw thinks that it should be 12 bytes long on 32-bit systems and 16 bytes long on 64-bit systems. And since CNA is compiled using mingw, forget about using the long double type.

4. Unions and bitfields in structures are not supported. This is caused by the fact that libffi doesn’t support them.

Any comments or suggestions will be highly appreciated.

The entire source code is available on github under the MIT license. https://github.com/intersystems-community/cna

Hi!

It is often necessary to run some external command such as a python program or a shell script from inside Caché/Ensemble. There are three ways of doing this:

• $ZF(-1) - Runs the command and waits for it to finish.  
• $ZF(-2) - Runs the command and don't wait for it to finish.
• Using CPIPE device - Runs the command and opens a device for you to read its output or (exclusive or here!) write to its input.

I use zf(-2) to spawn a external a Java application in a *nix instance.

I would like to kill this process after some conditions met.

I would like to leverage $zf("kill ... ") but this requires its the pid of this child process.

So is there a way to acquire the pid for the child process when I create it ?

If not, how is the suggested way to kill this process?

Thanks.

I could launch a external process via $zf(-2).

However, this child process need to refer some shared libraries in RUNTIME,so I have to set up the proper paths to the environment variable LD_LIBRARY_PATH, is anyone know how to do this?

Thank for your help.

Is there a way to get one of:

• Struct
• $lb
• Subscripted local variable
• List/array

As a return type from callout call?

And related question: is there a way to return several values (which may contain commas) from one callout call.  Or rather what's the best approach when doing that?

Hi I've created a word macro in order to convert doc to txt via the command line, this works fine via the command line by myself or another user but when I try as an the intersystems user which runs under  LocalSystem it doesn't work. 

So can I change the user, or set the $ZF to run as a different user?

Or do I have to try another way to convert doc to txt - it's looking like libreOffice?

I just wanted to stick with word because I could be guaranteed on the result being accurate.

Thanks

Regards

Richard

Hi guys,

   I am trying to run a command line code using $zf(-1) in cache terminal. it is returning access denied error.

I have tried to run the code in cmd itself it is also throwing Access denied error. But if opened cmd as administrator and run the same code it is working perfectly. I am using windows system. 

Hence i need to know how can i run the cmd line code as administrator using our terminal or studio. Please help me out.

Thanks,

Mani

Hello everybody,

We have a piece of Caché software which calls an external utility using $zf(-1,command). It works fine under Linux, but under Windows an external process occasionally hangs (due to some internal problems out of the scope here) and need to be killed programmatically. Having PID, it's easy to kill a process. If a Caché process is called with JOB command, the caller can easily get its PID from $zchild, but alas $zf(-1) does not seem to return the similar info. Is it possible to get it somehow?

Hi everyone,

I have a project which requires the sending of JSON messages to an external service provider using REST. The service provider requires the message contents to be signed.

Their instructions:

I would like  to invoke some unix/windows commands from COS, any API could do that?

Also, I would like to set up  some Environment Variables for this process which would execute  the above external command.

Anyway to achieve that?

Thanks for your help.

I have done Python - Cache binding setup following the guide from http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY…. I have also run test.py from sample3 folder and it able to run and complete successfully.

However, when I try to run the same test.py code via $zf, it gives error with exit code 1.

I've tried running help("intersys.pythonbind3") via $zf and also running from Cache terminal as follows:

1. $zf(-1,"C:\Python36\python <path>/script.py")
2. ! C:\Python36\python <path>/script.py

which gives me the following output:

Hello,

I'd like to know if there is an easy way to get result from a $ZF(-1) into a varaible to use it in HS code.

Thanks

This is the second part of my long post about package managers in operating systems and language distributions. Now, hopefully, we have managed to convince you that convenient package manager and rich 3^rd party code repository is one key factor in establishing of a vibrant and fast growing ecosystem. (Another possible reason for ecosystem success is the consistent language design, but it will be topic for another day.)

In this second part we plan to discuss the practical aspects of creating a package manager in general and their projection to the Caché database environment.

Per the information at http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY… I am trying to use a call to $ZF("GETFILE") to obtain information about an OpenVMS file. But I get an <ILLEGAL VALUE> error.

For example:

w $zf("GETFILE",filename,"UIC")

reports:

<ILLEGAL VALUE>

My filename variable contains the full path and name of a file that I own. I hold the %All role in Cache.

This is 2012.1.5 on OpenVMS/IA64  V8.4

Any ideas what's going wrong?

Does anyone have a good example of using %Net.SSH.Session  to do a scp copy of a file from one server to another? I tried calling do $ZF(-1,"scp /ensemble/Scott/sr1.dat egate@inteng3:/home/egate/Scott/") but did not have any success in getting the system to copy the file over to what I am calling inteng3.

Thanks

Scott