Helical Insight with Apache Drill

In this document we are covering how you can connect to Mongo and another RDBMS (like mysql, oracle, sqlserver etc) and do a crossdb join and create reports.

Pre-requisite :

– Helical Insight (version 5.0) installed

– Apache Drill (version 1.20) installed

Useful links :

https://www.helicalinsight.com/connecting-mongodb-using-apache-drill/

https://www.helicalinsight.com/drill-installation-helical-insight/

NOTE : Above documents might be using different version of apache drill as reference in their explanation, but you must install apache drill version 1.20

Below we have covered details step by step :

  1. In you drill version 1.20 storage create connection to mongo

    URL to access storages : http://localhost:8047/storage

    Sample mongo connection as below :

    {

"type": "mongo",

"connection": "mongodb+srv://USERNAME:PASSWORD@HOSTURL/DBNAME?retryWrites=true&w=majority",

"pluginOptimizations": {

"supportsProjectPushdown": true,

"supportsFilterPushdown": true,

"supportsAggregatePushdown": true,

"supportsSortPushdown": true,

"supportsUnionPushdown": true,

"supportsLimitPushdown": true

},

"batchSize": 100,

"enabled": true

}
  1. In order to make connection with any Relational Database like oracle, mysql, postgres, MS SQL Server. You need to download its relevant JDBC Jar file and need to upload it in the location : 
    “…..\apache-drill-1.20.0\jars\3rdparty”

    NOTE : 

    “…..\apache-drill-1.20.0\jars\3rdparty”

    is drill installation directory of your server / machine.

    After adding respective JDBC JAR into mentioned folder, you need to restart service of Apache drill.

  1. Once drill is up again, you need to create new storage with your RDBMS details. For example, we have given reference of RDBMS mysql db server. We have added storage with name as RDBMS (Name can be anything) and added connection details as below : 
    {

"type": "jdbc",

"driver": "com.mysql.jdbc.Driver",

"url": "jdbc:mysql://localhost:3306/DBNAME?enabledTLSProtocols=TLSv1.2",

"username": "dbusername",

"password": "dbpassword",

"writerBatchSize": 10000,

"enabled": true

}
  1. Till step 3, we have created 2 connection in our apache drill. 1 with mongo (NOSQL) and 1 with RDBMS (SQL). So all work is done from apache drill side.
    Helical Insight With Apache Drill
  2. Now we will make connection of apache drill with our Helical Insight application. In order to do that login into HI application and go to page : http://YOURIP_ADDRESS_OR_HOST:8085/hi-ee/#/datasource/all
  3. On this page you will see icon of Apache drill. Make connection of your drill server with HI application. Sample screenshot as below :
    Helical Insight With Apache Drill

 

  1. On top of this drill connection, create a metadata.
  2. In this metadata you will see list of all databases available in your mongo as well as RDBMS (in our case mysql)
  3. Expand any of the database either from mongo or from RDBMS
  4. You will see respective collections or tables of that database.
  5. Select tables / collections which are required to create report and add those into metadata.
    NOTE: Based on the kind of version of Helical Insight you are using you might need to follow some additional steps. If you are using any version above Helical Insight version 4.1, then follow below.
    Go to defaults.properties located at …/hi/hi-repository/System/Admin/. And can change default from strict-metadata on line no 68 and 70 so that your custom columns and other things work as is. Then restart the server.
  1. Now if you have selected tables from Mongo then create view of RDBMS and vice versa. I.e. if RDBMS tables are selected as part of metadata then create view using mongo.
    NOTE : In metadata view which you will create, you can simply select everything I.e select * from TABLE
  1. Join your normal selected tables with views within metadata based on common criteria I.e. primary key and foreign key
  2. Save the metadata which is created from normal table and views. View and normal tables are joined.
  3. Create report on top of this metadata, the report can fetch some columns from RDBMS and some columns from mongo as both are joined at metadata level.

You May Also Like

Helical Insight’s self-service capabilities is one to reckon with. It allows you to simply drag and drop columns, add filters, apply aggregate functions if required, and create reports and dashboards on the fly. For advanced users, the self-service component has ability to add javascript, HTML, HTML5, CSS, CSS3 and AJAX. These customizations allow you to create dynamic reports and dashboards. You can also add new charts inside the self-service component, add new kind of aggregate functions and customize it using our APIs.
Helical Insight’s self-service capabilities is one to reckon with. It allows you to simply drag and drop columns, add filters, apply aggregate functions if required, and create reports and dashboards on the fly. For advanced users, the self-service component has ability to add javascript, HTML, HTML5, CSS, CSS3 and AJAX. These customizations allow you to create dynamic reports and dashboards. You can also add new charts inside the self-service component, add new kind of aggregate functions and customize it using our APIs.
Helical Insight, via simple browser based interface of Canned Reporting module, also allows to create pixel perfect printer friendly document kind of reports also like Invoice, P&L Statement, Balance sheet etc.
Helical Insight, via simple browser based interface of Canned Reporting module, also allows to create pixel perfect printer friendly document kind of reports also like Invoice, P&L Statement, Balance sheet etc.
If you have a product, built on any platform like Dot Net or Java or PHP or Ruby, you can easily embed Helical Insight within it using iFrames or webservices, for quick value add through instant visualization of data.
If you have a product, built on any platform like Dot Net or Java or PHP or Ruby, you can easily embed Helical Insight within it using iFrames or webservices, for quick value add through instant visualization of data.
Being a 100% browser-based BI tool, you can connect with your database and analyse across any location and device. There is no need to download or install heavy memory-consuming developer tools – All you need is a Browser application! We are battle-tested on most of the commonly used browsers.
Being a 100% browser-based BI tool, you can connect with your database and analyse across any location and device. There is no need to download or install heavy memory-consuming developer tools – All you need is a Browser application! We are battle-tested on most of the commonly used browsers.
We have organization level security where the Superadmin can create, delete and modify roles. Dashboards and reports can be added to that organization. This ensures multitenancy.
We have organization level security where the Superadmin can create, delete and modify roles. Dashboards and reports can be added to that organization. This ensures multitenancy.
We have organization level security where the Superadmin can create, delete and modify roles. Dashboards and reports can be added to that organization. This ensures multitenancy.
We have organization level security where the Superadmin can create, delete and modify roles. Dashboards and reports can be added to that organization. This ensures multitenancy.
A first-of-its-kind Open-Source BI framework, Helical Insight is completely API-driven. This allows you to add functionalities, including but not limited to adding a new exporting type, new datasource type, core functionality expansion, new charting in adhoc etc., at any place whenever you wish, using your own in-house developers.
A first-of-its-kind Open-Source BI framework, Helical Insight is completely API-driven. This allows you to add functionalities, including but not limited to adding a new exporting type, new datasource type, core functionality expansion, new charting in adhoc etc., at any place whenever you wish, using your own in-house developers.
It handles huge volumes of data effectively. Caching, Pagination, Load-Balancing and In-Memory not only provides you with amazing experience, but also and does not burden the database server more than required. Further effective use of computing power gives best performance and complex calculations even on the big data even with smaller machines for your personal use. Filtering, Sorting, Cube Analysis, Inter Panel Communication on the dashboards all at lightning speed. Thereby, making best open-source Business Intelligence solution in the market.
It handles huge volumes of data effectively. Caching, Pagination, Load-Balancing and In-Memory not only provides you with amazing experience, but also and does not burden the database server more than required. Further effective use of computing power gives best performance and complex calculations even on the big data even with smaller machines for your personal use. Filtering, Sorting, Cube Analysis, Inter Panel Communication on the dashboards all at lightning speed. Thereby, making best open-source Business Intelligence solution in the market.
With advance NLP algorithm, business users simply ask questions like, “show me sales of last quarter”, “average monthly sales of my products”. Let the application give the power to users without knowledge of query language or underlying data architecture
With advance NLP algorithm, business users simply ask questions like, “show me sales of last quarter”, “average monthly sales of my products”. Let the application give the power to users without knowledge of query language or underlying data architecture
Our application is compatible with almost all databases, be it RDBMS, or columnar database, or even flat files like spreadsheets or csv files. You can even connect to your own custom database via JDBC connection. Further, our database connection can be switched dynamically based on logged in users or its organization or other parameters. So, all your clients can use the same reports and dashboards without worrying about any data security breech.
Our application is compatible with almost all databases, be it RDBMS, or columnar database, or even flat files like spreadsheets or csv files. You can even connect to your own custom database via JDBC connection. Further, our database connection can be switched dynamically based on logged in users or its organization or other parameters. So, all your clients can use the same reports and dashboards without worrying about any data security breech.
Our application can be installed on an in-house server where you have full control of your data and its security. Or on cloud where it is accessible to larger audience without overheads and maintenance of the servers. One solution that works for all.
Our application can be installed on an in-house server where you have full control of your data and its security. Or on cloud where it is accessible to larger audience without overheads and maintenance of the servers. One solution that works for all.
Different companies have different business processes that the existing BI tools do not encompass. Helical Insight permits you to design your own workflows and specify what functional module of BI gets triggered
Different companies have different business processes that the existing BI tools do not encompass. Helical Insight permits you to design your own workflows and specify what functional module of BI gets triggered