新しい投稿

検索

記事
· 2024年6月1日 3m read

How to set up columnar storage in classes

Columnar storage is one of the newer offers provided by InterSystems IRIS. Unlike traditional row-based storage, it optimizes query processing by storing data in columns rather than rows, enabling faster access and retrieval of relevant information.

A couple of articles have been written on when it should be used to give a system the biggest boost, how to create tables like that using SQL

CREATE TABLE table (column1 type1, column2 type2, column3 type3) WITH STORAGETYPE = COLUMNAR  -- ex 1
CREATE TABLE table (column1 type1, column2 type2, column3 type3 WITH STORAGETYPE = COLUMNAR)  -- ex 2

and even the performance tests.

As we all know, InterSystems IRIS is a multi-model DBMS and it gives seamless access to the same data using relational and object access. So the former is covered in other articles, but what about the latter?

It is good to know that the columnar storage layout can also be defined in classes, and there are several ways to do it.

1. If you wish to define the storage for all properties as columnar (example 1 at the top) just add a following parameter to your class:

Parameter STORAGEDEFAULT = "columnar" 

And if we take an example above, we'll get the following class:

Class Post.Address Extends %Persistent [Final]
{
Parameter STORAGEDEFAULT = "columnar";
Parameter USEEXTENTSET=1;
Property City As %String(MAXLEN = 12);
Property ZIP As %String(MAXLEN = 9);
Property Country As %String(MAXLEN = 12);
}

Parameter STORAGEDEFAULT = "columnar" tells the system that all the data has to be stored in the columnar form (meaning each column will have its own global).

Parameter USEEXTENTSET = 1 tells the system to generate more efficient, hashed globals with shorter names.

You can declare any table as columnar. However, tables that use columnar as the default storage layout must specify either the Final class keyword or the NoExtent class keyword, with any immediate subclasses defined explicitly as Final. Otherwise you will get an error during compilation.

2. You can also define only some properties as stored by columns (example 2 at the top). To do this you need to specify the parameter STORAGEDEFAULT = "columnar" for a property.

Class Post.Address Extends %Persistent
{
Parameter STORAGEDEFAULT = "row";
Parameter USEEXTENTSET=1;
Property City As %String(MAXLEN = 12);
Property ZIP As %String(MAXLEN = 9);
Property Country As %String(MAXLEN = 12, STORAGEDEFAULT = "columnar");
}

This way Cities and ZIPs will be stored as usual in a global ^Post.AddressD and Countries will be stored in separate global as a column.

In the case of this example, this would be a much better approach, because if we have a database of different cities in different countries, the number of countries is limited while the number of cities not so much. Besides, there aren't many cases where you would need to do analytics queries on cities unlike on coutries.

I hope this helps you understand how to work with columnar storage in classes. There are quite a few limitations which you need to keep in mind so please read more on the subject here.

ディスカッション (0)1
続けるにはログインするか新規登録を行ってください
記事
· 2024年6月1日 3m read

Cómo configurar el almacenamiento en columnas en clases

​El almacenamiento en columnas es una de las nuevas ofertas de InterSystems IRIS. A diferencia del almacenamiento tradicional basado en filas, optimiza el procesamiento de consultas al almacenar datos en columnas en lugar de filas, lo que permite un acceso y una recuperación más rápidos de información relevante.

Se han escrito un par de artículos sobre cuándo se debe utilizar para darle el mayor impulso a un sistema, y cómo crear tablas así usando SQL.

CREATE TABLE tabla (columna1 tipo1, columna2 tipo2, columna3 tipo3) WITH STORAGETYPE = COLUMNAR -- ex 1
CREATE TABLE tabla (columna1 tipo1, columna2 tipo2, columna3 tipo3 WITH STORAGETYPE = COLUMNAR) -- ex 2

e incluso las pruebas de rendimiento.

Como todos sabemos, InterSystems IRIS es un DBMS multimodelo y brinda acceso perfecto a los mismos datos mediante acceso relacional y de objetos. Lo primero se trata en otros artículos, pero ¿qué pasa con lo segundo?

Es bueno saber que el diseño de almacenamiento en columnas también se puede definir en clases y hay varias formas de hacerlo.

1. Si desea definir el almacenamiento de todas las propiedades como en columnas (ejemplo 1 en la parte superior), simplemente agregue el siguiente parámetro a su clase:

Parameter STORAGEDEFAULT = "columnar" 

Y si tomamos un ejemplo anterior, obtendremos la siguiente clase:

Class Post.Address Extends %Persistent [Final]
{
Parameter STORAGEDEFAULT = "columnar";
Parameter USEEXTENTSET=1;
Property City As %String(MAXLEN = 12);
Property ZIP As %String(MAXLEN = 9);
Property Country As %String(MAXLEN = 12);
}

El parámetro STORAGEDEFAULT = "columnar" le dice al sistema que todos los datos deben almacenarse en forma de columnas (lo que significa que cada columna tendrá su propio global).

El parámetro USEEXTENTSET = 1 le indica al sistema que genere globales hash más eficientes con nombres más cortos.

Puede declarar cualquier tabla como de columnas. Sin embargo, las tablas que utilizan columnas como diseño de almacenamiento predeterminado deben especificar la palabra clave de clase Final o la palabra clave de clase NoExtent , con cualquier subclase inmediata definida explícitamente como Final. De lo contrario, obtendrá un error durante la compilación.

2. También puede definir solo algunas propiedades almacenadas por columnas (ejemplo 2 en la parte superior). Para hacer esto, debe especificar el parámetro STORAGEDEFAULT = "columnar" para una propiedad.

Class Post.Address Extends %Persistent
{
Parameter STORAGEDEFAULT = "row";
Parameter USEEXTENTSET=1;
Property City As %String(MAXLEN = 12);
Property ZIP As %String(MAXLEN = 9);
Property Country As %String(MAXLEN = 12, STORAGEDEFAULT = "columnar");
}

De esta manera, las ciudades y los códigos postales se almacenarán como de costumbre en una ^Post.AddressD global y los países se almacenarán en una columna global separada.

En el caso de este ejemplo, este sería un enfoque mucho mejor, porque si tenemos una base de datos de diferentes ciudades en diferentes países, la cantidad de países es limitada, mientras que la cantidad de ciudades no lo es tanto. Además, no hay muchos casos en los que sea necesario realizar consultas analíticas en ciudades, a diferencia de los países.

Espero que esto te ayude a comprender cómo trabajar con el almacenamiento en columnas en las clases. Hay bastantes limitaciones que debes tener en cuenta, así que lee más sobre el tema aquí.

ディスカッション (0)1
続けるにはログインするか新規登録を行ってください
記事
· 2024年6月1日 2m read

Searching the limits of new datatype VECTOR

Translated from the Spanish Community Article Contest.

Following the latest programming contest on OEX I had some surprising observation.
There were almost exclusive applications based on AI in combination with pre-cooked Py modules.
But digging deeper, all examples used the same technical pieces of IRIS.

Seen from point of view of IRIS it was pretty much the same whether searching for text
or searching for images or other pattern.  It ended in almost exchangeable methods.

This reminds me my private situation at home. My wife and my daughter maintain an
(to me) incredible huge collection of skirts, shirts and all other clothes. But in the end
of day it's my wife and my daughter I talk to and I live with - 
no matter what wrap around is applied.

Back to contest:
A lot of fancy wrapping for more or less the same technical IRIS content.
Everyone was running down the same highway. No one ever touched any limit.

So I tried to dig deeper and find the limits of datatype VECTOR.
All vectors have 2 base parameters
- static DATATYPE : "integer" (or "int"), "double", "decimal", "string", and "timestamp".
- semi dynamic LEN(gth): > 0  often also referred as POSITION; a pure Integer.

This LEN/POSITION parameter is the equivalent what you know as mathematical dimensions of a vector.
Of course with in Einstein's universe you may just need 4 dimensions or less
based on his Theory of Relativity.
Even Cosmologic String Theory that came up in the 60ies doesn't pass 11..12 dimensions.
But all the nice pre-coocked text analysis solution packages use 238, 364, >1200, ....
dimensions and probably more.

So: What is the limit set by IRIS to the possible positions?
Official documentation has no answer.
So I took my terminal window and tried

for i=1:1 set $vector(test,I,"int")=i
;; very fast
<VECTOR>
zwrite i
i=65537

I tried with all data types: The limit is 65536

OK. The length of numeric types * 65536  is clear under the magic <MAXSTRING> limit bigger than 3 Mb

BUT: What is happening with type "string" if its size has a significant dimension ?

The impressive result: 
I succeeded with 65536 positions and a STRING of 3.600.000 bytes
The test_string is a few kB under <MAXSTRING>
Though!  This are  225.000 MB in total in a single VECTOR !
I fail to imagine how this could be done.

No doubt handling this unusual giant takes time and you have to wait long enough for any access.
But it demonstrates that DataType VECTOR is able to serve all practical  requirements
without being limited by design..

I wish you much success working with VECTORs.

ディスカッション (0)1
続けるにはログインするか新規登録を行ってください
お知らせ
· 2024年6月1日

Developer Community Recap, May 2024

Hello and welcome to the May 2024 Developer Community Recap.
General Stats
150 new posts published in May:
 35 new articles
 44 new announcements
 68 new questions
 3 new discussions
332 new members joined in May
13,330 posts published all time
12,397 members joined all time
Top posts
Top authors of the month
Articles
#InterSystems IRIS
After upgrading, when I try to open the Management Portal, I get ERROR #5001 and cannot open it
By Tomoko Furuzono
Tasks flow with InterSystems IRIS Workflow Engine - Configuration
By Luis Angel Pérez Ramos
Demo: Connecting Locally to an S3 Bucket without an AWS Account
By Macey Minor
Using VECTORs in ObjectScript
By Robert Cemper
Notes on changing the system date for testing purposes
By Hiroshi Sato
Microsoft 365 - Events and Tasks
By David Hockenbroch
IRIS AI Studio: A playground to explore the Generative AI & Vector Embedding capabilities
By Ikram Shah
Vector Search : Application to demonstrate Vector Search and Generative AI capabilities
By Muhammad Waseem
Preventive Health with ChatIRIS: Integrating InterSystems IRIS for Enhanced Patient Engagement
By Crystal Cheong
Q&A Chatbot with IRIS and langchain
By José Pereira
Retrieve images using vector search (1)
By shan yue
Vector search + GenAI + InterSystems technologies with Banksia Global
By Maria Gladkova
IRIS AI Studio: Connectors to Transform your Files into Vector Embeddings for GenAI Capabilities
By Ikram Shah
DNA Similarity and Classification: An Analysis
By Nicole Raimundo
How to run SQL in terminal
By Hiroshi Sato
Search for companies using smart search engines
By Lucas Fernandes
IRIS SIEM System Integration with Crowdstrike Logscale
By Ron Sweeney
IRIS AI Studio: Playground to explore RAG capabilities on top of IRIS DB vector embeddings
By Ikram Shah
Enhancing Preventive Health Engagement: The Backend Powering ChatIRIS with InterSystems IRIS
By Crystal Cheong
Wall-M : Perform semantic queries on your email inbox and get accurate responses along with source citations
By Arvind Menon
Imagine the Future of Medical Triage: MediCoPilot
By Henry Pereira
Open AI integration with IRIS - His master voice’s – Are you angry?
By Kurro Lopez
InterSystems® IRIS Conky
By Ron Sweeney
TIMESTAMP type format
By Megumi Kakechi
VIP in GCP
By Eduard Lebedyuk
How to avoid ODBC query timeouts
By Hiroshi Sato
 
#Open Exchange
 
#InterSystems Ideas Portal
 
#InterSystems IRIS for Health
 
#HealthShare
 
#Caché
 
Announcements
#InterSystems IRIS
InterSystems Certification has changed exam delivery platforms!
By Shane Nowack
[Video] Container Lifecycle - When is Your App Ready to Accept Work
By Anastasia Dyubaylo
InterSystems IRIS & IRIS for Health 2024.2 Developer preview update
By Bob Kuszewski
[Webinar in Hebrew] Maximize Your Success with InterSystems Support
By Ronnie Hershkovitz
Documentation New Look
By Ashok Kumar
InterSystems announces InterSystems IRIS 2024.1.0.267.2, includes support for Ubuntu 24.04
By Bob Kuszewski
iris-medicopilot video announcement
By Henrique Dias
[Video] IRIS AI Studio Walkthrough: Unleash the Power of Vector Embeddings in IRIS DB
By Ikram Shah
[Video] IRIS AI Studio: A detailed Functionality & Code Walkthrough
By Ikram Shah
Just launched: Deltanji source control (version 8)
By Laurel James (GJS)
MUMPS / CACHE developer
By Carlos Cordovil
May 30, 2024 – Advisory: License Enforcement Changes – REST and SOAP
By Fabiano Sanches
Quiz time on the Developer Community
By Olga Zavrazhnova
 
#Developer Community Official
 
#VSCode
 
#InterSystems IRIS for Health
 
#Learning Portal
 
#IRIS contest
 
#Global Masters
 
#InterSystems Official
 
#HealthShare
 
#InterSystems Ideas Portal
InterSystems Ideas News #13
By Vadim Aniskin
 
#Job Opportunity
 
Questions
#InterSystems IRIS
Cache and IRIS: How to identify system namespaces?
By Anna Golitsyna
What's the most recent non-preview Community Edition Container?
By Rob Tweed
IRIS on AWS EC2 and Hugepages
By Cristiano Silva
angular application with healthshare authentication
By Sylvie Greverend
docker run iris ,the container log always show Starting InterSystems IRIS instance IRIS... and can not open web page
By bold t
IIS webserver much slower than PWS
By Marcel den Ouden
Migrating to IRIS changes
By Rochdi Badis
Samples-FHIRStarter Questions
By Scott Roth
Using %JSON.Adaptor with %Integer value using the DISPLAYLIST VALUELIST parameters
By André-Claude Gendron
Is there a VSCode equivalent to the Studio SOAP Wizard?
By Victoria Castillo
SQL LOAD DATA ISSUES
By Ashok Kumar
Difference between %KillExtent and %DeleteExtent
By Meenakshi Muthu
X12 832 (4010) SEF or XSD file
By Jonathan Lent
Setting up the Quartz Scheduler with IRIS (Spring-boot)
By Ugljesa Pupavac
Mongodb integration
By Davide Cecchetto
Embeded python : Can I create a dict from COS?
By Stephane Devin
FHIR Profile validation Java issues
By Ashok Kumar
RSA Encryption Decryption
By David Marinkovic
InterSystems Class Dictionary
By lai pham
How are Stream saved?
By Stephane Devin
Pooling exception: Unable to find original pool for connection and IRIS Security Error
By Harikumar V
ObjectScript: Cannot Read OpenId, Result Doesn't Display
By Steven Henry Suhendra
Join JSON responses
By Yone Moreno Jimenez
DEBUG HTTPS REST API WITH WIRESHARK
By David Marinkovic
SYS.VSSWriter is generating alerts in Messages.log
By Andy Stobirski
Connecting to IRIS Database Using py.odbc
By A J
OPENSSL_Uplink(....): no OPENSSL_Applink
By Alin Soare
FSLog not populating when trying to test HS.FHIRServer.Interop.HTTPOperation
By Scott Roth
How to Remove Search Text Box in Filter Dropdown
By Virat Sharma
Is there a way to do an override of a %ALL Global Mapping?
By Ben Spead
python library "irisnative" for Windows 10
By Alin Soare
Javascript library "Irisnative" does not work for 8 bits IRIS servers
By Alin Soare
Irisnative library that works on Windows 10 with 8-bit IRIS
By Alin Soare
 
#Caché
 
#InterSystems IRIS for Health
Parsing a JSON message with
By Phil Burkhalter
what's the format of the file to import lookup table ?
By Yuhong Snyder
help with HL7 Transformation
By Will
Incorporating OAuth into HS.FHIRServer.Interop.HTTPOperation
By Scott Roth
HL7 DTL Code action question
By Will
How to access Management Portal of lockeddown iris health server
By Jagadiswara Bandaru
Intersystems local DB size monitoring using SolarWinds
By Vwnu B
End of File handling in Cache and IRIS
By Nisha Thomas
Implementing a counter for OBX
By Santosh K
Iris Client on MacOS - how to run it after installation
By Will
Customizing retry frequency of transaction
By Santosh Kannan
Need to evaluate rule based upon the content of "Z" segment, and only process if it exists on a table.
By John Steffen
Cannot parse valid HL7 message
By Stefano Mandelli
Benefit of custom FHIR profile usage
By Sebastian Thiele
Obtaining a payload from a FHIR response of type HS.FHIRServer.Interop.Response
By Stella Ticker
IRIS Weblink - Basic capabilities
By Michael Young
 
#Ensemble
 
#VSCode
Annoying Bulb in VS Code
By Alexey Maslov
 
#HealthShare
 
#TrakCare
 
#InterSystems IRIS BI (DeepSee)
 
Discussions
#InterSystems IRIS
Code Golf: Smiley Faces
By Eduard Lebedyuk
 
#Other
 
May, 2024Month at a GlanceInterSystems Developer Community
ディスカッション (0)1
続けるにはログインするか新規登録を行ってください
ダイジェスト
· 2024年6月1日

Resumo do InterSystems Developer Community, Maio 2024

Olá e bem-vindo ao boletim informativo da comunidade de desenvolvedores Maio 2024.
Estatísticas gerais
27 novas postages publicadas em Maio:
 11 novos artigos
 14 novos anúncios
 1 new question
 1 new discussion
4 novos membros ingressaram em Maio
1,076 postagens publicadas ao todo
580 membros ingressaram ao todo
Principais publicações
Principais autores do mês
Artigos
#InterSystems IRIS
Usando VECTORs em ObjectScript
Por Heloisa Paiva
Python BPL em pré-visualização
Por Heloisa Paiva
Monitorando ambientes InterSystems IRIS com Red Hat Insights
Por Heloisa Paiva
Similaridade e Classificação de DNA por Vector Search e Machine Learning
Por Nicole Raimundo
Busca de empresas usando Vector Search.
Por Lucas Fernandes
Feedback : Usando Python embutido diariamente por mais de dois anos
Por Heloisa Paiva
Studio IRIS AI Studio: Um playground para explorar as capacidades de Generative AI & Vector Embedding
Por Heloisa Paiva
Usando o Prometheus para coletar métricas no Intersystems IRIS
Por Julio Esquerdo
IRIS AI Studio: Conectores para transformar seus arquivos em vetores incorporados para capacidades do GenAI
Por Heloisa Paiva
Studio IRIS AI Studio: Playground para explorar as capacidades RAG sobre as incorporações de vetores da IRIS DB
Por Heloisa Paiva
Wall-M : Performe consultas semânticas na caixa de entrada do seu email e tenha respostas acuradas com citações de fontes
Por Heloisa Paiva
Anúncios
#InterSystems IRIS
#VSCode
#InterSystems IRIS for Health
#Developer Community Oficial
#Outro
#InterSystems Oficial
#Portal de Aprendizagem
Perguntas
Discussões
#InterSystems IRIS
Code Golf: Sorrisos
Por Heloisa Paiva
Maio, 2024Month at a GlanceInterSystems Developer Community